如何从Scrapy网站获取所有纯文本？

Question

我希望在呈现HTML之后在网站上显示所有文本。我使用Scrapy框架在Python中工作。 With xpath('//body//text()')我能够得到它，但带有HTML标签，我只想要文本。任何解决方案？谢谢 ！

Answer 1

最简单的办法是找到extract//body//text()和join一切：

''.join(sel.select("//body//text()").extract()).strip() 

其中sel是Selector实例。

另一种选择是使用nltk的clean_html()：

>>> import nltk 
>>> html = """ 
... <div class="post-text" itemprop="description"> 
... 
...   <p>I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework. 
... With <code>xpath('//body//text()')</code> I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !</p> 
... 
...  </div>""" 
>>> nltk.clean_html(html) 
"I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.\nWith xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !" 

另一种选择是使用BeautifulSoup的get_text()：

get_text()

If you only want the text part of a document or tag, you can use the get_text() method. It returns all the text in a document or beneath a tag, as a single Unicode string.

>>> from bs4 import BeautifulSoup 
>>> soup = BeautifulSoup(html) 
>>> print soup.get_text().strip() 
I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework. 
With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks ! 

另一种选择是使用lxml.html的text_content() ：

.text_content()

Returns the text content of the element, including the text content of its children, with no markup.

>>> import lxml.html 
>>> tree = lxml.html.fromstring(html) 
>>> print tree.text_content().strip() 
I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework. 
With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks ! 

Answer 2

你试过了吗？

xpath('//body//text()').re('(\w+)') 

OR

xpath('//body//text()').extract()