Jsoup将内容保存到数据库中

Question

我有一个url数组，我想从URL中存储信息，我在数据库中读取它。我的问题是我的数据列表太大的URL如果读取从存储在数据库中的上面的露珠序列化每个URL将需要时间。

我知道有一种方法可以使用线程来操作，但我不知道该怎么做，请帮助我。或者任何你的方法

try { 
    String lstUrls = "http://www.java2s.com/Tutorials/Java/Scala/index.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0020__Scala_Variables.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0040__Scala_Variable_Declarations.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0060__Scala_Semicolons.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0080__Scala_Code_Blocks.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0090__Scala_Comments.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0100__Scala_Type_Hierarchy.htm\n"; 
    String[] urls = lstUrls.split("\n"); 
    for (String url : urls) { 
     Document doc = Jsoup.connect(url).userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36").get(); 
     Elements select = doc.select("div.row"); 
     String html = select.html(); 
     System.out.println(html); 
     /* 
     insert html to database 
     */ 
    } 
} catch (IOException ex) { 
    ex.printStackTrace(); 
} 

Answer 1

要使用多线程检索数据，你可以做这样的事情：

Executor ex = Executors.newFixedThreadPool(3); 
    String lstUrls = "http://www.java2s.com/Tutorials/Java/Scala/index.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0020__Scala_Variables.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0040__Scala_Variable_Declarations.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0060__Scala_Semicolons.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0080__Scala_Code_Blocks.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0090__Scala_Comments.htm\n" 
      + "http://www.java2s.com/Tutorials/Java/Scala/0100__Scala_Type_Hierarchy.htm\n"; 
    String[] urls = lstUrls.split("\n"); 
    for (final String url : urls) { 
     try { 
      ex.execute(new Runnable() { 
       @Override 
       public void run() { 
        try { 
         Document doc = Jsoup 
           .connect(url) 
           .userAgent(
             "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36") 
           .get(); 
         Elements select = doc.select("div.row"); 
         String html = select.html(); 
         System.out.println(html); 
         /* 
         * insert html to database 
         */ 
        } catch (Exception e) { 
         e.printStackTrace(); 
        } 
       } 
      }); 
     } catch (Exception e) { 
      e.printStackTrace(); 
     } 
    } 

这将使用3个线程同时处理的URL，如果你想使用更多然后3个线程更改此行Executor ex = Executors.newFixedThreadPool(3);并用您想要的任何数字替换3。

你可以找到更多关于Executors here

Answer 2

我建议在插入数据库之前压缩数据。

//PreparedStatement.setBytes(1,compress(html)); 

public static byte[] compress(String str) throws Exception { 
    if (str == null || str.length() == 0) { 
     return null; 
    } 
    ByteArrayOutputStream obj = new ByteArrayOutputStream(); 
    GZIPOutputStream gzip = new GZIPOutputStream(obj); 
    gzip.write(str.getBytes("UTF-8")); 
    gzip.close(); 
    return obj.toByteArray(); 
} 

public static String decompress(byte[] bytes) throws Exception { 
    GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(bytes)); 
    BufferedReader bf = new BufferedReader(new InputStreamReader(gis,"UTF-8")); 
    String outStr = ""; 
    String line; 
    while ((line = bf.readLine()) != null) { 
     outStr += line; 
    } 
    return outStr; 
} 

第二种方法，将html数据保存到一个文件中并且只在数据库中存储文件路径。

long ts = System.currentTimeMillis(); 
String filePath = String.valueOf(ts)+".gz"; 
saveToFile(filePath ,html); 
--------  
public static void saveToFile(String filePath, String text) { 
    try { 
     GZIPOutputStream gzos = new GZIPOutputStream(new FileOutputStream(filePath)); 
     gzos.write(text.getBytes("UTF-8")); 
     gzos.finish(); 
     gzos.close(); 

    } catch (IOException ex) { 
     ex.printStackTrace(); 
    } 
}