为什么这个SQLite查询速度慢并且变慢？

Question

我有一个长期的数据记录服务，每个服务产生一天包含一天数据的文件。我正在将文件加载到Windows窗体应用程序中的SQLite数据库中。将文件中的数据插入到数据库中的过程包括两个查询，其结果将在后续插入中使用。

Using SQLconnect As New SQLite.SQLiteConnection("Data Source=" & fn & ";") 
SQLconnect.Open() 
Using SQLcommand As SQLite.SQLiteCommand = SQLconnect.CreateCommand 
    Dim SqlTrans As System.Data.SQLite.SQLiteTransaction = SQLconnect.BeginTransaction 
    For Each Path As String In paths 
     fs = System.IO.File.Open(Path, IO.FileMode.Open, IO.FileAccess.Read, IO.FileShare.Read) 'Open file 
     Do While ReadFromStoreFile(fs, dt, Sent) = True 'Read a Timestamp/sentence pair 
      'Create a positions table for this MMSI if one doesn't already exist 
      SQLcommand.CommandText = "CREATE TABLE IF NOT EXISTS MMSI" & msg.MMSI & " (PosID INTEGER PRIMARY KEY AUTOINCREMENT, Date NUMERIC, Lat REAL, Lon REAL, Status INTEGER, SOG REAL, COG INTEGER, HDG INTEGER, VoyageID INTEGER);" 
      SQLcommand.ExecuteNonQuery() 

      Select Case msg.Type 'Dynamic position report 

       Case AIS.MsgType.PosRptClsA 

        '###THIS QUERY TAKES 20 secs per file (day) and increases 3 seconds per day! 
        SQLcommand.CommandText = "SELECT * FROM Voyages WHERE MMSI = " & msg.MMSI & " ORDER BY VoyageID DESC LIMIT 1" 'still the same 
        SQLreader = SQLcommand.ExecuteReader() 
        SQLreader.Read() 
        VID = SQLreader.Item(0) 
        SQLreader.Close() 

        SQLcommand.CommandText = "INSERT INTO MMSI" & msg.MMSI & " (Date, Lat, Lon, Status, SOG, COG, HDG, VoyageID) VALUES (" & ts & ", " & msg.Latitude & ", " & msg.Longitude & ", " & msg.NavStatus & ", " & SOG & ", " & COG & ", " & HDG & ", " & VID & ")" 
        SQLcommand.ExecuteNonQuery() 
        SQLreader.Close() 

       Case AIS.MsgType.StatAndVge 

        'Find the latest entry for this MMSI in the Voyages table 
        '###THIS QUERY takes 3 secs for same number of queries and does NOT increase 
        SQLcommand.CommandText = "SELECT * FROM Voyages WHERE MMSI = " & msg.MMSI & " ORDER BY VoyageID DESC LIMIT 1" 
        SQLreader = SQLcommand.ExecuteReader() 
        SQLreader.Read() 

        Dim NoVoyage As Boolean = Not (SQLreader.HasRows) 
        If Not NoVoyage Then 
         'If the data has changed, add a new entry 
         If Not (SQLreader.Item(2) = msg.Length) Then Changed = True 
         If Not (SQLreader.Item(3) = msg.Breadth) Then Changed = True 
         If Not (SQLreader.Item(4) = msg.Draught) Then Changed = True 
         If Not (SQLreader.Item(5) = msg.Destination) Then Changed = True 
         If Not (SQLreader.Item(6) = msg.ETA.Ticks) Then Changed = True 
         VoyageID = SQLreader.Item(0) 
        End If 
        SQLreader.Close() 

        If Changed Or NoVoyage Then 
         Changed = False 'reset flag 
         SQLcommand.CommandText = "INSERT INTO Voyages (Date, Length, Breadth, Draught, Destination, ETA, MMSI) VALUES (" & ts & ", " & msg.Length & ", " & msg.Breadth & ", " & msg.Draught & ", '" & msg.Destination.Replace("'", "''") & "', " & msg.ETA.Ticks & ", " & msg.MMSI_int & ")" 
         SQLcommand.ExecuteNonQuery() 

         SQLcommand.CommandText = "SELECT last_insert_rowid() FROM Voyages" 
         SQLreader = SQLcommand.ExecuteReader() 
         SQLreader.Read() 
         VoyageID = SQLreader.Item(0) 
         SQLreader.Close() 

        End If 
      End Select 'message type 
     Loop 'Read next entry from file 
     fs.Close() 'Close the file 

     'Write this file into the files table, so we know it has been written to the DB 
     fileinf = New System.IO.FileInfo(Path) 
     SQLcommand.CommandText = "INSERT OR REPLACE INTO Files (Name, Size, Date) VALUES ('" & fileinf.Name & "', '" & fileinf.Length & "', '" & fileinf.LastWriteTimeUtc.Ticks & "')" 
     SQLcommand.ExecuteNonQuery() 

    Next 'The next path in the list of paths to decode 

    SqlTrans.Commit() 'End of all files reached, commit all the changes to the DB 

End Using 'SQLcommand 
End Using 'SQLconnect 

如在码所指示，第一查询花费很长的时间和（更重要的），其数据被加载到DB中的持续时间增加。当在数据库中添加21天的数据时，该查询每天的累积时间大约为20秒，并且每增加一天就会增加大约3秒。真奇怪的是，第二个查询（对我来说看起来是一样的）很快（对于相同数量的查询大约需要3秒），并且不会随着更多数据的添加而增加。

下面是创建空数据库的功能：

Public Function CreateDB(fn As String, Force As Boolean) As Boolean 

    If System.IO.File.Exists(fn) Then 
     If Force Then 
      System.IO.File.Delete(fn) 'Delete the old DB and create a new one 
     Else 
      Return True 'DB alrewady exists so just return true 
     End If 
    End If 


    Using SQLconnect As New SQLite.SQLiteConnection 
     SQLconnect.ConnectionString = "Data Source=" & fn & ";" 
     SQLconnect.Open() 

     'Create Tables 
     Using SQLcommand As SQLite.SQLiteCommand = SQLconnect.CreateCommand 

      'Set page size 
      SQLcommand.CommandText = "PRAGMA Page_size = 4096;" 
      SQLcommand.ExecuteNonQuery() 

      'Set journalling mode to off 
      SQLcommand.CommandText = "PRAGMA journal_mode = OFF;" 
      SQLcommand.ExecuteNonQuery() 

      'Set auto indexing off 
      SQLcommand.CommandText = "PRAGMA automatic_index = false;" 
      SQLcommand.ExecuteNonQuery() 

      'Create Vessels Table 
      SQLcommand.CommandText = "CREATE TABLE Vessels(MMSI TEXT PRIMARY KEY, Name TEXT, Type INTEGER, IMO TEXT, CallSign TEXT, MothershipMMSI INTEGER, LastVoyageID INTEGER);" 
      SQLcommand.ExecuteNonQuery() 

      'Create Voyages Table 
      SQLcommand.CommandText = "CREATE TABLE Voyages(VoyageID INTEGER PRIMARY KEY AUTOINCREMENT, Date NUMERIC, Length INTEGER, Breadth INTEGER, Draught INTEGER, Destination TEXT, ETA NUMERIC, MMSI INTEGER);" 
      SQLcommand.ExecuteNonQuery() 

      'Create Meta Table 
      SQLcommand.CommandText = "CREATE TABLE Files(Name TEXT PRIMARY KEY, Size NUMERIC, Date NUMERIC);" 
      SQLcommand.ExecuteNonQuery() 

     End Using 'SQLcommand 

    End Using ' SQLconnect 

    Return True 

End Function 

这可能是造成第一查询这么慢，相比于第二查询，需要更长的时间随着更多的数据添加到数据库？

SQlite和System.Data.Sqlite是最新版本。

Answer 1

假设msg由ReadFromStoreFile改变，则该查询是否有用于给定MMSI更航行

"SELECT * FROM Voyages WHERE MMSI = " & msg.MMSI & " ORDER BY VoyageID DESC LIMIT 1" 

速度会变慢。所以我假设具有AIS.MsgType.PosRptClsA的MMSI值比其他MMSI更频繁地插入。

它似乎查询获取MMSI的最大航程ID。你可以直接用这个做更多

"SELECT max(VoyageID) FROM Voyages WHERE MMSI = " & msg.MMSI 

我不知道这样会跑得更快。 或者，您可以保留一个MMSI和最大航程id的字典，并在您插入航行以消除查询时进行更新。