使用存储过程提取大数据时发生内存问题。

Question

我只需要使用Hibernate调用MySql存储过程并根据它编写一个文件。 但是从存储过程返回的行数为0.3到0.4百万行，而且它们非常大。

这里是我的代码片段

try { 
session = sessionFactory.getCurrentSession(); 
query = session.createSQLQuery("CALL MyStoredProcedure();"); 
List<Object[]> result = query.list(); 
} catch (HibernateException e) { 
    e.printStackTrace(); 
} finally { 
    if(session!= null) { 
     session.flush(); 
     session.clear(); 
    } 
} 

和错误信息

Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: GC overhead limit exceeded 
at java.io.UnixFileSystem.canonicalize0(Native Method) 
at java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:172) 
at java.io.File.getCanonicalPath(File.java:576) 
at org.apache.naming.resources.FileDirContext.file(FileDirContext.java:806) 
at org.apache.naming.resources.FileDirContext.doGetAttributes(FileDirContext.java:430) 
at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:1089) 
at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:1042) 
at org.apache.naming.resources.ProxyDirContext.getAttributes(ProxyDirContext.java:884) 
at org.apache.catalina.loader.WebappClassLoader.modified(WebappClassLoader.java:984) 
at org.apache.catalina.loader.WebappLoader.modified(WebappLoader.java:501) 
at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:421) 
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1350) 
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1537) 
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1547) 
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1547) 
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1526) 
at java.lang.Thread.run(Thread.java:636) 

调查了一下，在工作之后，我发现我不得不使用的setFetchSize（）和setMaxResults（）在我的code.Like

try { 
session = sessionFactory.getCurrentSession(); 
query = session.createSQLQuery("CALL MyStoredProcedure();"); 
query.setFetchSize(1000); 
query.setFirstResult(lowerLimit); 
query.setMaxResults(upperLimit); 
List<Object[]> result = query.list(); 
} catch (HibernateException e) { 
    e.printStackTrace(); 
} finally { 
    if(session!= null) { 
     session.flush(); 
     session.clear(); 
    } 
} 

但是这一次我得到另一个错误。以下是错误讯息

org.hibernate.exception.SQLGrammarException: could not execute query 
    at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:67) 
    at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43) 
    at org.hibernate.loader.Loader.doList(Loader.java:2216) 
    at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104) 
    at org.hibernate.loader.Loader.list(Loader.java:2099) 
    at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:289) 
    at org.hibernate.impl.SessionImpl.listCustomQuery(SessionImpl.java:1695) 
    at org.hibernate.impl.AbstractSessionImpl.list(AbstractSessionImpl.java:142) 
    at org.hibernate.impl.SQLQueryImpl.list(SQLQueryImpl.java:152) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:616) 
    at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.doInvokeMethod(HandlerMethodInvoker.java:710) 
    at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:167) 
    at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:414) 
    at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:402) 
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:771) 
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:716) 
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:647) 
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:552) 
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:621) 
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) 
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) 
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) 
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) 
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) 
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) 
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) 
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224) 
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169) 
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) 
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) 
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) 
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927) 
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) 
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) 
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987) 
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579) 
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:309) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) 
    at java.lang.Thread.run(Thread.java:636) 
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'limit 100000, 120000' at line 1 
    at sun.reflect.GeneratedConstructorAccessor50.newInstance(Unknown Source) 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
    at java.lang.reflect.Constructor.newInstance(Constructor.java:532) 
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:406) 
    at com.mysql.jdbc.Util.getInstance(Util.java:381) 
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1030) 
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956) 
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3515) 
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3447) 
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1951) 
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2101) 
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2554) 
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1761) 
    at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1912) 
    at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76) 
    at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:186) 
    at org.hibernate.loader.Loader.getResultSet(Loader.java:1787) 
    at org.hibernate.loader.Loader.doQuery(Loader.java:674) 
    at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236) 
    at org.hibernate.loader.Loader.doList(Loader.java:2213) 

那么我该如何解决这个问题呢？请帮忙。

Answer 1

而是在内存中加载的一切，做你喜欢的JDBC会做：滚动结果：

query = session.createSQLQuery("CALL MyStoredProcedure();"); 
ScrollableResults results = query.scroll(); 
while (results.next()) { 
    Object[] row = results.get(); 
    // TODO write the row into the file 
}