为什么这个简单的POST请求未在Python Scrapy工作，而它用简单的request.post（）

Question

的作品我有简单的POST请求的代码

import requests 

headers = { 
    'origin': 'https://jet.com', 
    'accept-encoding': 'gzip, deflate, br', 
    'x-csrf-token': 'IzaENk9W-Xzv9I5NcCJtIf9h_nT24p5fU-Tk', 
    'jet-referer': '/product/detail/87e89b3ce17f4742ab6d72aeaaa5480d?gclid=CPzS982CgdMCFcS1wAodABwIOQ', 
    'x-requested-with': 'XMLHttpRequest', 
    'accept-language': 'en-US,en;q=0.8', 
    'cookie': 'akacd_phased_release=3673158615~rv=53~id=041cdc832c1ee67c7be18df3f637ad43; jet.csrf=_JKKPyR5fKD-cPDGmGv8AJk5; jid=7292a61d-af8f-4d6f-a339-7f62afead9a0; jet-phaser=%7B%22experiments%22%3A%5B%7B%22variant%22%3A%22a%22%2C%22version%22%3A1%2C%22id%22%3A%22a_a_test16%22%7D%2C%7B%22variant%22%3A%22slp_categories%22%2C%22version%22%3A1%2C%22id%22%3A%22slp_categories%22%7D%2C%7B%22variant%22%3A%22on_cat_nav_clicked%22%2C%22version%22%3A1%2C%22id%22%3A%22catnav_load%22%7D%2C%7B%22variant%22%3A%22zipcode_table%22%2C%22version%22%3A1%2C%22id%22%3A%22zipcode_table%22%7D%5D%2C%22id%22%3A%222982c0e7-287e-42bb-8858-564332ada868%22%7D; ak_bmsc=746D16A88CE3AE7088B0CD38DB850B694F8C5E56B1650000DAA82659A1D56252~plJIR8hXtAZjTSjYEr3IIpW0tW+u0nQ9IrXdfV5GjSfmXed7+tD65YJOVp5Vg0vdSqkzseD0yUZUQkGErBjGxwmozzj5VjhJks1AYDABrb2mFO6QqZyObX99GucJA834gIYo6/8QDIhWMK1uFvgOZrFa3SogxRuT5MBtC8QBA1YPOlK37Ecu1WRsE2nh55E24F0mFDx5hXcfBAhWdMne6NrQ88JE9ZDxjW5n8qsh+QAHo=; _sdsat_landing_page=https://jet.com/product/detail/87e89b3ce17f4742ab6d72aeaaa5480d?gclid=CPzS982CgdMCFcS1wAodABwIOQ|1495705823651; _sdsat_session_count=1; AMCVS_A7EE579F557F617B7F000101%40AdobeOrg=1; AMCV_A7EE579F557F617B7F000101%40AdobeOrg=-227196251%7CMCIDTS%7C17312%7CMCMID%7C11996417004070294145733272597342763775%7CMCAID%7CNONE%7CMCAAMLH-1496310624%7C3%7CMCAAMB-1496310625%7Chmk_Lq6TPIBMW925SPhw3Q%7CMCOPTOUT-1495713041s%7CNONE; __qca=P0-949691368-1495705852397; mm_gens=Rollout%20SO123%20-%20PDP%20Grid%20Image%7Ctitle%7Chide%7Cattr%7Chide%7Cprice%7Chide~SO19712%20HP%20Rec%20View%7Clast_viewed%7Cimage-only~SO17648%20-%20PLA%20PDP%7Cdesc%7CDefault%7Cbuybox%7Cmodal%7Cexp_cart%7Chide-cart%7Ctop_caro%7CDefault; jcmp_productSku=882b1010309d48048b8f3151ddccb3cf; _sdsat_all_pages_canary_variants=a_a_test16:a|slp_categories:slp_categories|catnav_load:on_cat_nav_clicked|zipcode_table:zipcode_table; _sdsat_all_pages_native_pay_eligible=No; _uetsid=_uet6ed8c6ab; _tq_id.TV-098163-1.3372=ef52068e069c26b9.1495705843.0.1495705884..; _ga=GA1.2.789964406.1495705830; _gid=GA1.2.1682210002.1495705884; s_cc=true; __pr.NaN=6jvgorz8tb; mm-so17648=gen; __pr.11xw=xqez1m3cvl; _sdsat_all_pages_login_status=logged-out; _sdsat_jid_cookie=7292a61d-af8f-4d6f-a339-7f62afead9a0; _sdsat_phaser_id=2982c0e7-287e-42bb-8858-564332ada868; _sdsat_all_pages_jet_platform=desktop; _sdsat_all_pages_site_version=3.860.1495036770896|2017-05-16 20:35:36 UTC; _sdsat_all_pages_canary_variants_2=a_a_test16:a~slp_categories:slp_categories~catnav_load:on_cat_nav_clicked~zipcode_table:zipcode_table; jet=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6WyJmMmUwMjI1NS1iODFkLTRlOTktOGU1Yi0yZGI1MjU0ZTdjNzUiXSwiamNtcEhpc3RvcnkiOltbXV0sImlwWmlwY29kZSI6WyIyMTA2MSJdLCJjbGllbnRUaWNrZXQiOlsiZXlKMGVYQWlPaUpLVjFRaUxDSmhiR2NpT2lKSVV6STFOaUo5LmV5SmpiR2xsYm5SZmFXUWlPaUl3Tm1JMlkyTTNaVGRtTnpVME16TmhPREU0T0RjelpUWmpZMkV4WTJRelppSXNJbWx6Y3lJNkltcGxkQzVqYjIwaUxDSmhkV1FpT2lKM1pXSmpiR2xsYm5RaWZRLnlKMXdoYklDVml4TE1iblliV0xQY1RvdF9EWUo3MjFYQkdFMzBpUktpdTQiXSwicHJvbW9jb2RlIjpbIlNQUklORzE1Il0sInBsYSI6W3RydWVdLCJmcmVlU2hpcHBpbmciOltmYWxzZV0sImpjbXAiOlt7ImpjbXAiOiJwbGE6Z2dsOm5qX2R1cl9nZW5fcGF0aW9fX2dhcmRlbl9hMjpwYXRpb19fZ2FyZGVuX2dyaWxsc19fb3V0ZG9vcl9jb29raW5nX2dyaWxsX2NvdmVyc19hMjpuYTpwbGFfNzg0NzQ0NTQyXzQwNTY4Mzg3NzA2X3BsYS0yOTM2MjcyMDMzNDE6bmE6bmE6bmE6Mjo4ODJiMTAxMDMwOWQ0ODA0OGI4ZjMxNTFkZGNjYjNjZiIsImNvZGUiOiJQTEExNSIsInNrdSI6Ijg4MmIxMDEwMzA5ZDQ4MDQ4YjhmMzE1MWRkY2NiM2NmIn1dLCJpYXQiOjE0OTU3MDU4OTh9.6OEM9e9fTyUZdFGju19da4rEnFh8kPyg8wENmKyhYgc; bm_sv=360FA6B793BB42A17F395D08A2D90484~BLAlpOUET7ALPzcGziB9dbZNvjFjG3XLQPFGCRTk+2bnO/ivK7G+kOe1WXpHgIFmyZhniWIzp2MpGel1xHNmiYg0QOLNqourdIffulr2J9tzacGPmXXhD6ieNGp9PAeTqVMi+2kSccO1+JzO+CaGFw==; s_tps=30; s_pvs=173; mmapi.p.pd=%221759837076%7CDwAAAApVAgDxP2Qu1Q4AARAAAUJz0Q1JAQAmoW6kU6PUSKeaIXVTo9RIAAAAAP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FAAZEaXJlY3QB1Q4BAAAAAAAAAAAAt8wAAH0vAQC3zAAABQDZlQAAAmpAfP%2FVDgD%2F%2F%2F%2F%2FAdUO1Q7%2F%2FwYAAAEAAAAAAc9dAQCNFgIAADyXAABuDVKACdUOAP%2F%2F%2F%2F8B1Q7VDv%2F%2FCgAAAQAAAAABs2ABANweAgAAiY0AAMCzlXtx1Q4A%2F%2F%2F%2F%2FwHVDtUO%2F%2F8GAAABAAAAAAORSwEATPoBAJJLAQBO%2BgEAk0sBAFD6AQABt8wAAAYAAADYlQAAHMPK3ZbVDgD%2F%2F%2F%2F%2FAdUO1Q7%2F%2FwYAAAEAAAAAAc5dAQCJFgIAAbfMAAAGAAAAmpgAAFAf9YUU1Q4A%2F%2F%2F%2F%2FwHVDtUO%2F%2F8EAAABAAAAAAR0YwEA1R4CAHVjAQDWHgIAdmMBANgeAgB3YwEA2x4CAAG3zAAABAAAAAAAAAAAAUU%3D%22; mmapi.p.srv=%22fravwcgus04%22; mmapi.e.PLA=%22true%22; mmapi.p.uat=%7B%22PLATraffic%22%3A%22true%22%7D; _sdsat_lt_pages_viewed=6; _sdsat_pages_viewed=6; _sdsat_traffic_source=', 
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36', 
    'content-type': 'application/json', 
    'accept': 'application/json, text/javascript, */*; q=0.01', 
    'referer': 'https://jet.com/product/detail/87e89b3ce17f4742ab6d72aeaaa5480d?gclid=CPzS982CgdMCFcS1wAodABwIOQ', 
    'authority': 'jet.com', 
    'dnt': '1', 
} 

data = '{"zipcode":"21061","sku":"87e89b3ce17f4742ab6d72aeaaa5480d","origination":"PDP"}' 

r=requests.post('https://jet.com/api/product/v2', headers=headers, data=data) 
print(r) 

它返回200

而且我想这个简单的转换请求到Python请求。

body = '{"zipcode":"21061","sku":"87e89b3ce17f4742ab6d72aeaaa5480d","origination":"PDP"}' 
yield Request(url = 'https://jet.com/api/product/v2', callback=self.parse_jet_page, meta={'data':data}, method="POST", body=body, headers=self.jet_headers) 

返回400，看起来像头被过度写什么的。还是有错误？

Answer 1

我猜的错误是由饼干引起的。

默认情况下，在你的HTTP标头的“曲奇”项须由内置的下载中间件CookiesMiddleware被重写。 Scrapy希望用户使用Request.cookies传递饼干。

如果您确实需要直接在Request.headers（而不是使用Request.cookies）中传递Cookie，则需要禁用内置的CookiesMiddleware。您可以简单地在设置中设置COOKIES_ENABLED=False。