我怎样才能让我的文件读/写程序更快？

Question

所以我有一个包含一堆天气数据的大文件。我必须将大文件中的每一行分配到相应的状态文件中。所以总共会有50个新的状态文件和他们自己的数据。

大文件包含〜同类产品1万行的记录：

COOP:166657,'NEW IBERIA AIRPORT ACADIANA REGIONAL LA US',200001,177,553 

虽然站的名称可能有所不同，并有不同数量的单词。

目前，我正在使用正则表达式来查找模式并输出到一个文件，并且它必须按状态分组。如果我在没有任何修改的情况下读取整个文件，大约需要46秒。使用代码查找状态缩写，创建文件并输出到该文件，需要10分钟以上。

这就是我现在所拥有的：

package climate; 

import java.io.BufferedReader; 
import java.io.File; 
import java.io.FileReader; 
import java.io.FileWriter; 
import java.io.IOException; 
import java.util.Arrays; 
import java.util.Scanner; 
import java.util.regex.Matcher; 
import java.util.regex.Pattern; 

/** 
* This program will read in a large file containing many stations and states, 
* and output in order the stations to their corresponding state file. 
* 
* Note: This take a long time depending on processor. It also appends data to 
* the files so you must remove all the state files in the current directory 
* before running for accuracy. 
* 
* @author Marcus 
* 
*/ 

public class ClimateCleanStates { 

    public static void main(String[] args) throws IOException { 

     Scanner in = new Scanner(System.in); 
     System.out 
       .println("Note: This program can take a long time depending on processor."); 
     System.out 
       .println("It is also not necessary to run as state files are in this directory."); 
     System.out 
       .println("But if you would like to see how it works, you may continue."); 
     System.out.println("Please remove state files before running."); 
     System.out.println("\nIs the States directory empty?"); 
     String answer = in.nextLine(); 

     if (answer.equals("N")) { 
      System.exit(0); 
      in.close(); 
     } 
     System.out.println("Would you like to run the program?"); 
     String answer2 = in.nextLine(); 
     if (answer2.equals("N")) { 
      System.exit(0); 
      in.close(); 
     } 

     String[] statesSpaced = new String[51]; 

     File statefile, dir, infile; 

     // Create files for each states 
     dir = new File("States"); 
     dir.mkdir(); 


     infile = new File("climatedata.csv"); 
     FileReader fr = new FileReader(infile); 
     BufferedReader br = new BufferedReader(fr); 

     String line; 
     line = br.readLine(); 
     System.out.println(); 

     // Read in climatedata.csv 
     final long start = System.currentTimeMillis(); 
     while ((line = br.readLine()) != null) { 
      // Remove instances of -9999 
      if (!line.contains("-9999")) { 

         String stateFileName = null; 

         Pattern p = Pattern.compile(".* ([A-Z][A-Z]) US"); 
         Matcher m = p.matcher(line); 
         if (m.find()){ 
          stateFileName = m.group(1); 
          stateFileName = "States/" + stateFileName + ".csv"; 
          statefile = new File(stateFileName); 

          FileWriter stateWriter = new FileWriter(statefile, true); 
          stateWriter.write(line + "\n"); 
          // Progress reporting 
          //System.out.printf("Writing [%s] to file [%s]\n", line, 
          //  statefile); 

          stateWriter.flush(); 
          stateWriter.close(); 

         } 
      } 
     } 
     System.out.println("Elapsed " + (System.currentTimeMillis() - start) + " ms"); 
     br.close(); 
     fr.close(); 
     in.close(); 

    } 

} 

Answer 1

可以使用Map来跟踪状态的文件，而不是每次都关闭它们：

Map<String, FileWriter> fileMap = new HashMap<String, FileWriter>(); 

while ((line = br.readLine()) != null) { 
    if (!line.contains("-9999")) { 
     if (m.find()) { 
      stateFileName = m.group(1); 
      stateFileName = "States/" + stateFileName + ".csv"; 
      FileWriter stateFileWriter = fileMap.get(stateFileName); 
      if (stateFileWriter == null) { 
       stateFileWriter = new FileWriter(stateFileName, true); 
       fileMap.put(stateFileName, stateFileWriter); 
      } 

      stateFileWriter.write(line + "\n"); 
     } 
    } 
} 

// flush the writers and close once you have parsed the entire file 
for(Map.Entry<String, FileWriter> entry : fileMap.entrySet()) { 
    FileWriter writer = entry.getValue(); 
    writer.flush(); 
    writer.close(); 
} 

Answer 2

不要做任何事，你不必在循环中。制作键入文件名的文件地图，并将其保持打开状态。只要你需要写给他们，在你完成之前不要关闭他们中的任何一个。你也不需要冲洗它们。你击败所有好的缓冲将会做你。一旦你完成了，你可以考虑使缓冲区变大。