Bug #2813
Updated by François ARMAND over 11 years ago
Sometimes, when aggregating reports, we get end up with OOM when generating promises. It seems to happen especially when other tasks are happening at the same time. When it happens, generating report is awfully long (even tens of minutes), the JVM GC is overwhelmed, and it ends badly. An example (when aggregating report at the same time): logs looking like that: <pre> java.lang.OutOfMemoryError: Java heap space 00:08:30.536 [pool-1-thread-3] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Historize Name] scheduler task started at 2012/08/15 22:43:21, ended at 2012/08/16 00:08:30. <- Exception caught during update process. 00:08:30.536 [pool-1-thread-11] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Aggregate Report] scheduler task started at 2012/08/11 15:27:28, ended at 2012/08/15 21:59:06. <- Java heap space 00:08:30.540 [pool-1-thread-5] INFO c.n.p.a.service.AgregationService - Starting the agregation of the reports 2012-08-16 00:30:37.089:WARN::EXCEPTION java.lang.OutOfMemoryError: Java heap space 00:30:37.090 [pool-1-thread-16] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Aggregate Report] scheduler task started at 2012/08/11 15:32:28, ended at 2012/08/16 00:08:30. <- Java heap space 00:30:37.093 [pool-1-thread-5] INFO c.n.p.a.service.AgregationService - Starting the agregation of the reports 00:43:27.590 [pool-1-thread-3] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Aggregate Report] scheduler task started at 2012/08/11 15:37:28, ended at 2012/08/16 00:30:37. <- Java heap space 00:43:27.593 [pool-1-thread-5] INFO c.n.p.a.service.AgregationService - Starting the agregation of the reports Exception in thread "HashSessionScavenger-6" java.lang.OutOfMemoryError: Java heap space 01:05:17.777 [pool-1-thread-13] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Historize Name] scheduler task started at 2012/08/16 00:52:01, ended at 2012/08/16 00:57:04. <- Exception caught during update process. 01:05:17.777 [pool-1-thread-21] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Aggregate Report] scheduler task started at 2012/08/11 15:42:28, ended at 2012/08/16 00:43:27. <- Java heap space 01:05:17.777 [pool-1-thread-13] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Historize Name] scheduler task started at 2012/08/16 00:57:20, ended at 2012/08/16 01:02:28. <- Exception caught during update process. 01:05:17.782 [pool-1-thread-5] INFO c.n.p.a.service.AgregationService - Starting the agregation of the reports 01:18:15.335 [pool-1-thread-12] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Historize Name] scheduler task started at 2012/08/16 01:10:24, ended at 2012/08/16 01:17:50. <- Exception caught during update process. 01:27:24.733 [pool-1-thread-15] ERROR net.liftweb.actor.ActorLogger - Actor threw an exception java.lang.OutOfMemoryError: Java heap space Exception in thread "pool-1-thread-15" java.lang.OutOfMemoryError: Java heap space Exception in thread "pool-1-thread-22" java.lang.OutOfMemoryError: Java heap space 2012-08-16 01:41:55.924:WARN::EXCEPTION java.lang.OutOfMemoryError: Java heap space Exception in thread "qtp28290629-79" Exception in thread "qtp28290629-84" java.lang.OutOfMemoryError: Java heap space Exception in thread "HashSessionScavenger-4" java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space 01:43:25.302 [pool-1-thread-23] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Historize Name] scheduler task started at 2012/08/16 01:29:52, ended at 2012/08/16 01:41:55. <- Exception caught during update process. 2012-08-16 01:52:36.183:WARN::EXCEPTION java.lang.OutOfMemoryError: Java heap space 01:52:36.184 [pool-1-thread-20] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Historize Name] scheduler task started at 2012/08/16 01:42:35, ended at 2012/08/16 01:52:36. <- Exception caught during update process. 2012-08-16 01:57:54.626:WARN::Problem scavenging sessions java.lang.OutOfMemoryError: Java heap space 2012-08-16 01:57:54.626:WARN::EXCEPTION java.lang.OutOfMemoryError: Java heap space 01:58:19.709 [pool-1-thread-16] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Historize Name] scheduler task started at 2012/08/16 01:52:36, ended at 2012/08/16 01:57:54. <- Exception caught during update process. Exception in thread "HashSessionScavenger-8" java.lang.OutOfMemoryError: Java heap space 2012-08-16 02:03:48.771:WARN::Problem scavenging sessions java.lang.OutOfMemoryError: Java heap space 2012-08-16 02:07:55.224:WARN::/endpoint/upload/ java.lang.OutOfMemoryError: Java heap space ... 11:57:32.378 [pool-1-thread-5] INFO c.n.p.a.service.AgregationService - Starting the agregation of the reports 12:07:00.045 [pool-1-thread-5] INFO c.n.p.a.service.AgregationService - Starting the agregation of the reports Exception in thread "TransparentProxy-58" java.lang.OutOfMemoryError: Java heap space ... </pre>