使用python获取avalibale在网站中的所有链接？

Question

是否有任何方式使用python获取网站中的所有链接，而不仅仅是在网页中？我想这个代码，但是这是给我只能在网页链接

import urllib2 
import re 

#connect to a URL 
website = urllib2.urlopen('http://www.example.com/') 

#read html code 
html = website.read() 

#use re.findall to get all the links 
links = re.findall('"((http|ftp)s?://.*?)"', html) 

print links 

Answer 1

访问递归你收集的链接，太废以下页面：

import urllib2 
import re 

stack = ['http://www.example.com/'] 
results = [] 

while len(stack) > 0: 

    url = stack.pop() 
    #connect to a URL 
    website = urllib2.urlopen(url) 

    #read html code 
    html = website.read() 

    #use re.findall to get all the links 
    # you should not only gather links with http/ftps but also relative links 
    # you could use beautiful soup for that (if you want <a> links) 
    links = re.findall('"((http|ftp)s?://.*?)"', html) 

    result.extend([link in links if is_not_relative_link(link)]) 

    for link in links: 
     if link_is_valid(link): #this function has to be written 
      stack.push(link)