试图执行r网络抓取代码，但给出了一个错误

Question

我试图执行下面的代码，并获取joblocations执行时的错误。页面加载到ulrs但位置不是从网页中提取。

library(data.table) 
library(XML) 
pages<-c(1:12) 

ulrs <- rbindlist(lapply(pages, function(x) 
    {url <- paste("http://www.r-users.com/jobs/page/",x,"/",sep = " ") 
    data.frame(url)}),fill = TRUE) 


joblocations <- rbindlist(apply(ulrs,1,function(url){ 
    doc1 <- htmlParse(url) 
    locations <- getNodeSet(doc1,'//*[@id="mainContent"]/div[2]/ol/li/dl/dd[3]/span/text()') 
data.frame(sapply(locations,function(x){xmlValue(x)}))  
}),fill = TRUE) 


Error: failed to load external entity "http://www.r-users.com/jobs/page/%201%20/" 

Answer 1

首先，改变http到https将让你过去的XML是错误，但再引出另一个问题：WARNING: XML content does not seem to be XML: 'https://www.r-users.com/jobs/page/1/，仍然不起作用。

我再次尝试，换用XML包rvest并得到它的工作。

library(data.table) 
library(rvest) 

pages<-c(1:12) 

ulrs <- rbindlist(lapply(pages, function(x) 
{url <- paste0("http://www.r-users.com/jobs/page/",x,"/") 
data.frame(url)} 
),fill = TRUE) 

joblocations <- rbindlist(apply(ulrs,1,function(url){ 
    doc1 <- read_html(url) 
    locations <- html_nodes(doc1,xpath = '//*[@id="mainContent"]/div[2]/ol/li/dl/dd[3]/span/text()') 
    data.frame(sapply(locations,function(x){html_text(x)}))  
})) 

rvest似乎工作是否HTTP或HTTPS指定，但这是什么，以前我绊倒了。