Python的网页抓取特殊字符提取数据

Question

cityURL='https://en.wikipedia.org/wiki/Elko,_Nevada' 

def createObj(url): 
    html = urlopen(url) 
    bsObj = BeautifulSoup(html, 'lxml') 
    return bsObj 

bsObj1 = createObj(cityURL) 

table1 = bsObj1.find("table", {"class":"infobox geography vcard"}) 
incorporated = table1.find("th", text='Incorporated (city)').findNext('td').get_text() 

table1.find("th", text='. Total') # Problem here, due to the special dot, I cannot identify the "th" 

我希望得到以下数据：

总，17.6 土地，17.6 水，0.0

Answer 1

的 “。” 在页面不是“点”。这是一个unicode字符BULLET（\ u2022）。

你可以使用python的regex（re）模块来实现这一点。

更新的代码会是这个样子：

import re 
cityURL='https://en.wikipedia.org/wiki/Elko,_Nevada' 

def createObj(url): 
    html = urlopen(url) 
    bsObj = BeautifulSoup(html, 'lxml') 
    return bsObj 

bsObj1 = createObj(cityURL) 

table1 = bsObj1.find("table", {"class":"infobox geography vcard"}) 
incorporated = table1.find("th", text='Incorporated (city)').findNext('td').get_text() 

pattern = re.compile(r'Total') 
table1.find("th", text=pattern) 

或者，你可以使用lxml的模块，它是很多比beautifulsoup快。

import requests 
from lxml import html 

cityURL='https://en.wikipedia.org/wiki/Elko,_Nevada' 
r = requests.get(cityURL) 
root = html.fromstring(r.content) 

def normalize(text) : 
    return ''.join([i if ord(i) < 128 else ' ' for i in text]).strip().split()[0] 

val_list = [(normalize(root.xpath('//table[@class="infobox geography vcard"]//tr[./th/text()="Area"]/following-sibling::tr[{}]//text()'.format(str(val)))[1]), normalize(root.xpath('//table[@class="infobox geography vcard"]//tr[./th/text()="Area"]/following-sibling::tr[{}]//text()'.format(str(val)))[3])) for val in xrange(1,4)] 
print(val_list) 

上面的代码将输出：

[(u'Total', u'17.6'), (u'Land', u'17.6'), (u'Water', u'0.0')]