Python：将数据从大型csv导入到sqlite数据库

Question

我在尝试使用python脚本将数据从csv文件导入到sqlite数据库。

我的CSV行如下：（在从quandle下载的excel工作表）

Date  Open High  Low Last Close TotTrQt Turnover (Lacs) 
2017-05-26 2625 2626.85 2564.65 2570.05 2578.25 681275 17665.43 
2017-05-25 2577 2637.55 2568 2615.05 2624.6 2047047 53333.77 
2017-05-24 2534.8 2570 2529.65 2567.1 2559.15 1267274 32252.28 
2017-05-23 2533.2 2564.15 2514 2523.7 2521.7 1374298 34776.45 
2017-05-22 2510 2553.75 2510 2535 2531.35 831970 21054.61 
2017-05-19 2536.2 2540.55 2486 2503.85 2507.15 893022 22384.3 
2017-05-18 2450 2572 2442.25 2525 2536.2 2569297 64894.78 
2017-05-17 2433.5 2460.75 2423 2450 2455.35 1438099 35137.29 
2017-05-16 2380 2435 2373.45 2425.1 2429.15 1800513 43397.03 
2017-05-15 2375.1 2377.95 2341.6 2368 2365.1 908802 21380.43 

为了产生DB表，我已经使用下面的脚本：

import sqlite3 

try: 
    db = sqlite3.connect('NSETCS') 
    cursor=db.cursor() 
    print 'Executing: Create Table SQL' 
    cursor.execute('''CREATE TABLE NSETCS (DATE TEXT, OPEN REAL, HIGH REAL, LOW REAL, LAST REAL, CLOSE REAL,\ 
    TOTALTRADEQUANTITY REAL, TURNOVER REAL)''') 
    ##since above statment is DDL, no explicit commit is reqd 
except Exception as E: 
    print "Error=",E 
finally: 
    db.close() 

用于插入数据在表格的特定行中，我使用下面的脚本，但是由于浮点转换给出错误，所以数据插入失败，任何指导都将非常值得赞赏。

import sqlite3 

try: 
    infile = open (r'F:\mypractise_python\day11\NSE-TCS.csv','r') 
    content = infile.readlines() 
except IOError as E: 
    print "Error: ", E 
try: 
    db = sqlite3.connect('NSETCS') 
    cursor = db.cursor() 

    for line in content: 
     line =line.strip() 
     columns = line.split(',') 
     if line == '' or columns[0] == 'Date': 
      continue 
     date = columns[0].strip() 
     open_stock = float(columns[1].strip()) 
     high = float(columns[2].strip()) 
     low = float(columns[3].strip()) 
     last= float(columns[4].strip()) 
     close= float(columns[5].strip()) 
     tot_trade_qt= float(columns[6].strip()) 
     turnover= float(columns[7].strip()) 
     cursor.execute('''insert into NSETCS values (:date, :open_stock, :high, :low, :last, :close, :tot_trade_qt, :turnover)''',\ 
         {'date':date, 'open_stock':open_stock, 'high':high, 'low':low, 'last':last, 'close':close,\ 
         'tot_trade_qt':tot_trade_qt, 'turnover':turnover}) 

except Exception as E: 
    print "Error:", E 
else: 
    db.commit() 

db.close() 
infile.close() 

Answer 1

该CSV样本不匹配的代码 - 在后者你跳过一排（大概是报头），其中所述第一小区是“DATE” - 和在CSV是“日期”。

考虑使用csv.DictReader - 它会为每个源行的字典，其中一列是一个关键：

import sqlite3 
import csv 

db = sqlite3.connect('NSETCS') 

with open (r'F:\mypractise_python\day11\NSE-TCS.csv','r') as infile, db: 
    content = csv.DictReader(infile, delimiter=',') # csv generator to the file, will be read line by line 

    cursor = db.cursor() 

    for line in content: 
     # line is a dict, where each column name is the key 
     # no need to sanitize the header row, that was done automatically upon reading the file 
     date = line['Date'] 
     open_stock = float(line['Open']) 
     high = float(line['High']) 
     low = float(line['Low']) 
     last= float(line['Last']) 
     close= float(line['Close']) 
     tot_trade_qt= float(line['TotTrQt']) 
     turnover= float(line['Turnover (Lacs)']) 
     cursor.execute('''insert into NSETCS values (:date, :open_stock, :high, :low, :last, :close, :tot_trade_qt, :turnover)''',\ 
        {'date':date, 'open_stock':open_stock, 'high':high, 'low':low, 'last':last, 'close':close,\ 

      'tot_trade_qt':tot_trade_qt, 'turnover':turnover}) 

# no file closing, that is automatically done by 'with open()' 
# no db close or commit - the DB connection is in a context manger, that'll be done automatically (the commit - if there are no exceptions) 

我的主要观点是 - 不要手工解析原始CSV，但使用内置的库 - 这将为您节省很多麻烦。

例如，您尝试处理空行，跳过标题行;但在csv中还有其他一些情况 - 数据引用和转义，例如库为您处理。