我是scrapy的新手,并试图通过scraping yellowpages.com网站了解它。Scrapy网络爬行不好
我的目标是编写一个python代码来进入yellowpages.com主页的搜索字段(业务和位置),然后刮取后续的URL。
我的代码如下所示:
import scrapy
from scrapy.spiders import Spider
from scrapy.selector import Selector
from spider.items import Website
class YellowPages(Spider):
name = "yellow"
allowed_domains = ["yellowpages.com"]
start_urls = [
"http://www.yellowpages.com/"
]
def parse(self, response):
return scrapy.FormRequest.from_response(
response,
formxpath="//form[@id='search-form']",
formdata={
"query":"business",
"location" : "78735" },
callback=self.after_results
)
def after_results(self, response):
self.logger.info("info msg")
我想搜索 “业务” 的位置, “78735”。但是,这些不是传递给网站的价值。我的日志看起来是这样的:
2016-01-28 23:55:36 [scrapy] DEBUG: Crawled (200) <GET http://www.yellowpages.com/> (referer: None)
2016-01-28 23:55:36 [scrapy] DEBUG: Crawled (200) <GET http://www.yellowpages.com/search?search_terms=&geo_location_terms=Los+Angeles%2C+CA&query=business&location=78735> (referer: http://www.yellowpages.com/)
在第二个URL,术语洛杉矶+洛杉矶以某种方式插入。当我尝试手动输入搜索字段并提交,这是URL应该怎样看这样的:
http://www.yellowpages.com/search?search_terms=business&geo_location_terms=78735
谁能告诉我什么错误,以及如何解决它?
非常感谢。
仅供参考,这里是yellowpages.com主页
<div class="search-bar"><form id="search-form" action="/search" method="GET"><div><label><span>What do you want to find?</span><input id="query" type="text" value="" placeholder="What do you want to find?" autocomplete="off" data-onempty="recent-searches" name="search_terms" tabindex="1"/></label><ul id="recent-searches" class="search-dropdown recent-searches"><li class="search-hint">Search by<b> business name,</b> or<b> keyword</b></li></ul><ul id="autosuggest-term" data-analytics='{"moi":105}' class="search-dropdown autosuggest-term"></ul></div><em>near</em><div><label><span>Where?</span> <input id="location"type="text" value="78735" placeholder="Where?" autocomplete="off" data-onempty="menu-location" name="geo_location_terms" tabindex="2"/></label>