如何摆脱使用python

Question

我需要帮助制定在我的脚本新的代码读取通过文件报废，如果有文件名的任何重复（不是文件类型），然后从目录中删除重复的文件名。提前致谢！这里是我当前的代码：

from bs4 import BeautifulSoup 
import urllib.request 
import os 

url = urllib.request.urlopen("https://www.fhfa.gov/DataTools/Downloads/Pages/House-Price-Index-Datasets.aspx#mpo") 

soup = BeautifulSoup(url, from_encoding=url.info().get_param('charset')) 

FHFA = os.chdir('C:/US_Census/Directory') 

for link in soup.find_all('a', href=True): 
    href = link.get('href') 
    if not any(href.endswith(x) for x in ['.csv', '.xml', '.xls', '.xlsx', '.sql', '.txt', '.json']): 
     continue 

    filename = href.split('/')[-1] 
    url = urllib.request.urlretrieve('https://www.fhfa.gov/' + href, filename) 
    print(filename) 

print(' ') 
print("All files successfully downloaded.") 

Answer 1

你的代码检索文件名如：

HPI_master.csv 
HPI_master.xml 
HPI_master.sql 
... 

可以理解的，你只想要第一个，放弃休息。

您可以添加set保持看到文件名的轨迹：

seen = set() 
for link in soup.find_all('a', href=True): 
    href = link.get('href') 
    if not any(href.endswith(x) for x in ['.csv', '.xml', '.xls', '.xlsx', '.sql', '.txt', '.json']): 
     continue 

    file = href.split('/')[-1] 
    filename = file.rsplit('.', 1)[0] 
    if filename not in seen: # only retrieve file if it has not been seen before 
     seen.add(filename) # add the file to the set 
     url = urllib.request.urlretrieve('https://www.fhfa.gov/' + href, file) 

Answer 2

添加您已经看到

seen = set() 
for link in soup.find_all('a', href=True): 
    href = link.get('href') 
    splitted = href.replace('/','.').split('.') 
    if len(splitted)<2: 
     continue 
    filename, fileext = splitted[-2:] 
    if href.find('.')<0 or not (fileext.lower() in ['csv', 'xml', 'xls', 'xlsx', 'sql', 'txt', 'json']): 
     continue 
    if filename in seen: 
     continue 
    seen.add(filename) 
    url = urllib.request.urlretrieve('https://www.fhfa.gov/' + href, filename+'.'+fileext) 
    print(filename)