合法化Web请求，以便服务器允许请求通过

我一直在尝试运行下面的代码，但它一直产生HTTP错误502.我认为错误的原因是因为网站知道程序是试图从中获取信息。因此，它不允许请求。有没有办法欺骗服务器认为这是一个合法的Web请求？我曾尝试添加标题，但它仍然无效。合法化Web请求，以便服务器允许请求通过

import urllib.request 


# Function: Convert information within html document to a text file 
# Append information to the file 
def html_to_text(source_html, target_file): 

    opener = urllib.request.build_opener() 
    opener.addheaders = [('User-agent', 'Mozilla/5.0')] 
    print(source_html) 
    r = opener.open(source_html) 
    response = r.read() 
    print(response) 
    temp_file = open(target_file, 'w+') 
    temp_file.write(response.__str__()) 


source_address = "https://sg.finance.yahoo.com/lookup/all?s=*&t=A&m=SG&r=&b=0" 
target_location = "C:\\Users\\Admin\\PycharmProjects\\TheLastPuff\\Source\\yahoo_ticker_symbols.txt" 

html_to_text(source_address, target_location)

来源

2016-01-02 Cloud

它适用于我。你向他们发送了多少请求？他们可能会将您的实验检测为bruteforce或dos攻击，并将您的请求的某些指纹列入黑名单 –

有没有办法欺骗服务器认为请求来自合法浏览器？ – Cloud

是的。捕获浏览器发送的流量，并将标题值复制到您的Python脚本中。 –

我已经对代码进行了一些修改，并达到了我所要求的。

import urllib.request 
import gzip 


# Function: Convert information within html document to a text file 
# Append information to the file 
def html_to_text(source_html, target_file): 

    opener = urllib.request.build_opener() 
    # These headers are necessary to ensure that the website thinks that a browser is retrieving information 
    # not a program. 
    opener.addheaders = [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0'), 
         ('Connection', 'keep-alive'), 
         ('Accept-encoding', 'gzip, deflate'), 
         ('Accept-language', 'en-US,en;q=0.5'), 
         ('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'), 
         ('Host', 'sg,finance.yahoo.com'), ] 
    r = opener.open(source_html) 

    # Check from the "Response Headers" in Firebug whether the content is encoded 
    # Since the content is encoded in gzip format, decompression is necessary 
    response = gzip.decompress(r.read()) 

    # The response headers would mention the "charset" from there the encoding type can be obtained 
    response = response.decode(encoding='utf-8') 
    print(response) 
    temp_file = open(target_file, 'w+') 
    temp_file.write(response) 


source_address = "https://sg.finance.yahoo.com/lookup/all?s=*&t=A&m=SG&r=&b=0" 
target_location = "C:\\Users\\Admin\\PycharmProjects\\TheLastPuff\\Source\\yahoo_ticker_symbols.txt" 

html_to_text(source_address, target_location)

来源

2016-01-05 15:52:06 Cloud

合法化Web请求，以便服务器允许请求通过

回答

相关问题