熊猫：从BeautifulSoup将所有研究结果写入csv

我有一个Python pandas脚本的这些开始，它可以搜索Google上的值并抓取它可以在第一页找到的任何PDF链接。熊猫：从BeautifulSoup将所有研究结果写入csv

我有两个问题，下面列出。

import pandas as pd 
from bs4 import BeautifulSoup 
import urllib2 
import re 

df = pd.DataFrame(["Shakespeare", "Beowulf"], columns=["Search"])  

print "Searching for PDFs ..." 

hdr = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11", 
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", 
    "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.3", 
    "Accept-Encoding": "none", 
    "Accept-Language": "en-US,en;q=0.8", 
    "Connection": "keep-alive"} 

def crawl(search): 
    google = "http://www.google.com/search?q=" 
    url = google + search + "+" + "PDF" 
    req = urllib2.Request(url, headers=hdr) 

    pdf_links = None 
    placeholder = None #just a column placeholder 

    try: 
     page = urllib2.urlopen(req).read() 
     soup = BeautifulSoup(page) 
     cite = soup.find_all("cite", attrs={"class":"_Rm"}) 
     for link in cite: 
      all_links = re.search(r".+", link.text).group().encode("utf-8") 
      if all_links.endswith(".pdf"): 
       pdf_links = re.search(r"(.+)pdf$", all_links).group() 
      print pdf_links 

    except urllib2.HTTPError, e: 
     print e.fp.read() 

    return pd.Series([pdf_links, placeholder]) 

df[["PDF links", "Placeholder"]] = df["Search"].apply(crawl) 

df.to_csv(FileName, index=False, delimiter=",")

从print pdf_links的结果将是：

davidlucking.com/documents/Shakespeare-Complete%20Works.pdf 
sparks.eserver.org/books/shakespeare-tempest.pdf 
www.w3.org/People/maxf/.../hamlet.pdf 
www.w3.org/People/maxf/.../hamlet.pdf 
www.w3.org/People/maxf/.../hamlet.pdf 
www.w3.org/People/maxf/.../hamlet.pdf 
www.w3.org/People/maxf/.../hamlet.pdf 
www.w3.org/People/maxf/.../hamlet.pdf 
www.w3.org/People/maxf/.../hamlet.pdf 
calhoun.k12.il.us/teachers/wdeffenbaugh/.../Shakespeare%20Sonnets.pdf 
www.yorku.ca/inpar/Beowulf_Child.pdf 
www.yorku.ca/inpar/Beowulf_Child.pdf 
https://is.muni.cz/el/1441/.../2._Beowulf.pdf 
https://is.muni.cz/el/1441/.../2._Beowulf.pdf 
https://is.muni.cz/el/1441/.../2._Beowulf.pdf 
https://is.muni.cz/el/1441/.../2._Beowulf.pdf 
www.penguin.com/static/pdf/.../beowulf.pdf 
www.neshaminy.org/cms/lib6/.../380/text.pdf 
www.neshaminy.org/cms/lib6/.../380/text.pdf 
sparks.eserver.org/books/beowulf.pdf

以及CSV输出如下：

Search   PDF Links 
Shakespeare calhoun.k12.il.us/teachers/wdeffenbaugh/.../Shakespeare%20Sonnets.pdf 
Beowulf  sparks.eserver.org/books/beowulf.pdf

问题：

有没有办法将所有结果写入csv而不是只是最下面的一行？并且如果可能的话，包含对应于"Shakespeare"或"Beowulf"的每行的值为Search？
如何写出完整的pdf链接而不用长链接自动缩写为"..."？

来源

2015-07-04 Winterflags

什么搜索词是你使用？ –

嗨@PadraicCunningham！我使用“莎士比亚”和“贝奥武夫”作为搜索词（来自DataFrame）。 – Winterflags

错误的链接http://pastebin.com/Z38X8hWU，除非你真的想要一个数据帧它也可以全部使用csv模块 –

这将使用soup.find_all("a",href=True)让你所有适当的PDF链接，并将其保存在一个数据帧和一个CSV：

hdr = { 
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11", 
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", 
    "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.3", 
    "Accept-Encoding": "none", 
    "Accept-Language": "en-US,en;q=0.8", 
    "Connection": "keep-alive"} 


def crawl(columns=None, *search): 
    df = pd.DataFrame(columns= columns) 
    for term in search: 
     google = "http://www.google.com/search?q=" 
     url = google + term + "+" + "PDF" 
     req = urllib2.Request(url, headers=hdr) 
     try: 
      page = urllib2.urlopen(req).read() 
      soup = BeautifulSoup(page) 
      pdfs = [] 
      links = soup.find_all("a",href=True) 
      for link in links: 
       lk = link["href"] 
       if lk.endswith(".pdf"): 
        pdfs.append((term, lk)) 
      df2 = pd.DataFrame(pdfs, columns=columns) 
      df = df.append(df2, ignore_index=True) 
     except urllib2.HTTPError, e: 
      print e.fp.read() 
    return df 


df = crawl(["Search", "PDF link"],"Shakespeare","Beowulf") 
df.to_csv("out.csv",index=False)

out.csv：

Search,PDF link 
Shakespeare,http://davidlucking.com/documents/Shakespeare-Complete%20Works.pdf 
Shakespeare,http://www.w3.org/People/maxf/XSLideMaker/hamlet.pdf 
Shakespeare,http://sparks.eserver.org/books/shakespeare-tempest.pdf 
Shakespeare,https://phillipkay.files.wordpress.com/2011/07/william-shakespeare-plays.pdf 
Shakespeare,http://www.artsvivants.ca/pdf/eth/activities/shakespeare_overview.pdf 
Shakespeare,http://triggs.djvu.org/djvu-editions.com/SHAKESPEARE/SONNETS/Download.pdf 
Beowulf,http://www.yorku.ca/inpar/Beowulf_Child.pdf 
Beowulf,https://is.muni.cz/el/1441/podzim2013/AJ2RC_STAL/2._Beowulf.pdf 
Beowulf,http://teacherweb.com/IL/Steinmetz/MottramM/Beowulf---Seamus-Heaney.pdf 
Beowulf,http://www.penguin.com/static/pdf/teachersguides/beowulf.pdf 
Beowulf,http://www.neshaminy.org/cms/lib6/PA01000466/Centricity/Domain/380/text.pdf 
Beowulf,http://www.sparknotes.com/free-pdfs/uscellular/download/beowulf.pdf

来源

2015-07-06 17:13:22

熊猫：从BeautifulSoup将所有研究结果写入csv

回答

相关问题