适用于Java的良好和有效的CSV/TSV阅读器

Question

我正在尝试阅读大CSV和TSV（Tab sepperated）具有约1000000行或更多行的文件。现在我试着读TSV含~2500000行与opencsv，但它会抛出我。它适用于较小的TSV含~250000行的文件。所以我想知道是否有任何其他Libraries，支持巨大的CSV和TSV文件的阅读。你有什么想法？

大家谁是有兴趣在我的代码（我缩短了，所以Try-Catch显然是无效的）：

InputStreamReader in = null; 
CSVReader reader = null; 
try { 
    in = this.replaceBackSlashes(); 
    reader = new CSVReader(in, this.seperator, '\"', this.offset); 
    ret = reader.readAll(); 
} finally { 
    try { 
     reader.close(); 
    } 
} 

编辑：这是我构建InputStreamReader方法：

private InputStreamReader replaceBackSlashes() throws Exception { 
     FileInputStream fis = null; 
     Scanner in = null; 
     try { 
      fis = new FileInputStream(this.csvFile); 
      in = new Scanner(fis, this.encoding); 
      ByteArrayOutputStream out = new ByteArrayOutputStream(); 

      while (in.hasNext()) { 
       String nextLine = in.nextLine().replace("\\", "/"); 
       // nextLine = nextLine.replaceAll(" ", ""); 
       nextLine = nextLine.replaceAll("'", ""); 
       out.write(nextLine.getBytes()); 
       out.write("\n".getBytes()); 
      } 

      return new InputStreamReader(new ByteArrayInputStream(out.toByteArray())); 
     } catch (Exception e) { 
      in.close(); 
      fis.close(); 
      this.logger.error("Problem at replaceBackSlashes", e); 
     } 
     throw new Exception(); 
    } 

Answer 1

我还没有尝试过，但我早些时候研究过superCSV。

http://sourceforge.net/projects/supercsv/

http://supercsv.sourceforge.net/

检查是否适合你，250万线。

Answer 2

尝试按Satish建议切换库。如果这没有帮助，则必须将整个文件分成标记并处理它们。

以为你CSV没有过的逗号

// r is the BufferedReader pointed at your file 
String line; 
StringBuilder file = new StringBuilder(); 
// load each line and append it to file. 
while ((line=r.readLine())!=null){ 
    file.append(line); 
} 
// Make them to an array 
String[] tokens = file.toString().split(","); 

然后你就可以处理任何转义字符。在使用之前不要忘记修改令牌。

Answer 3

我不知道这个问题是否仍然有效，但这里是我成功使用的问题。但是，仍然可能需要实现更多接口，例如Stream或Iterable，但是：

import java.io.Closeable; 
import java.io.File; 
import java.io.FileNotFoundException; 
import java.io.IOException; 
import java.io.InputStream; 
import java.util.Scanner; 

/** Reader for the tab separated values format (a basic table format without escapings or anything where the rows are separated by tabulators).**/ 
public class TSVReader implements Closeable 
{ 
    final Scanner in; 
    String peekLine = null; 

    public TSVReader(InputStream stream) throws FileNotFoundException 
    { 
     in = new Scanner(stream); 
    } 

    /**Constructs a new TSVReader which produces values scanned from the specified input stream.*/ 
    public TSVReader(File f) throws FileNotFoundException {in = new Scanner(f);} 

    public boolean hasNextTokens() 
    { 
     if(peekLine!=null) return true; 
     if(!in.hasNextLine()) {return false;} 
     String line = in.nextLine().trim(); 
     if(line.isEmpty()) {return hasNextTokens();} 
     this.peekLine = line;  
     return true;   
    } 

    public String[] nextTokens() 
    { 
     if(!hasNextTokens()) return null;  
     String[] tokens = peekLine.split("[\\s\t]+"); 
//  System.out.println(Arrays.toString(tokens)); 
     peekLine=null;  
     return tokens; 
    } 

    @Override public void close() throws IOException {in.close();} 
} 

Answer 4

不要使用CSV解析器来解析TSV输入。例如，如果TSV具有带引号字符的字段，它将会中断。

uniVocity-parsers附带TSV解析器。您可以解析10亿行而不会出现任何问题。

实例解析一个TSV输入：

TsvParserSettings settings = new TsvParserSettings(); 
TsvParser parser = new TsvParser(settings); 

// parses all rows in one go. 
List<String[]> allRows = parser.parseAll(new FileReader(yourFile)); 

如果输入是这么大，不能被保存在内存中，这样做：

TsvParserSettings settings = new TsvParserSettings(); 

// all rows parsed from your input will be sent to this processor 
ObjectRowProcessor rowProcessor = new ObjectRowProcessor() { 
    @Override 
    public void rowProcessed(Object[] row, ParsingContext context) { 
     //here is the row. Let's just print it. 
     System.out.println(Arrays.toString(row)); 
    } 
}; 
// the ObjectRowProcessor supports conversions from String to whatever you need: 
// converts values in columns 2 and 5 to BigDecimal 
rowProcessor.convertIndexes(Conversions.toBigDecimal()).set(2, 5); 

// converts the values in columns "Description" and "Model". Applies trim and to lowercase to the values in these columns. 
rowProcessor.convertFields(Conversions.trim(), Conversions.toLowerCase()).set("Description", "Model"); 

//configures to use the RowProcessor 
settings.setRowProcessor(rowProcessor); 

TsvParser parser = new TsvParser(settings); 
//parses everything. All rows will be pumped into your RowProcessor. 
parser.parse(new FileReader(yourFile)); 

披露：我的作者这个库。它是开放源代码和免费的（Apache V2.0许可证）。