Rudder

Sometimes, we get OOM when generating promises.

It seems to happen especially when other tasks are happening at the same time.
When it happens, generating report is awfully long (even tens of minutes), the JVM GC is overwhelmed, and it ends badly.

An example (when aggregating report at the same time):

java.lang.OutOfMemoryError: Java heap space
00:08:30.536 [pool-1-thread-3] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Historize Name] scheduler task started at 2012/08/15 22:43:21, ended at 2012/08/16 00:08:30. <- Exception caught during update process.
00:08:30.536 [pool-1-thread-11] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Aggregate Report] scheduler task started at 2012/08/11 15:27:28, ended at 2012/08/15 21:59:06. <- Java heap space
00:08:30.540 [pool-1-thread-5] INFO  c.n.p.a.service.AgregationService - Starting the agregation of the reports
2012-08-16 00:30:37.089:WARN::EXCEPTION
java.lang.OutOfMemoryError: Java heap space
00:30:37.090 [pool-1-thread-16] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Aggregate Report] scheduler task started at 2012/08/11 15:32:28, ended at 2012/08/16 00:08:30. <- Java heap space
00:30:37.093 [pool-1-thread-5] INFO  c.n.p.a.service.AgregationService - Starting the agregation of the reports
00:43:27.590 [pool-1-thread-3] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Aggregate Report] scheduler task started at 2012/08/11 15:37:28, ended at 2012/08/16 00:30:37. <- Java heap space
00:43:27.593 [pool-1-thread-5] INFO  c.n.p.a.service.AgregationService - Starting the agregation of the reports
Exception in thread "HashSessionScavenger-6" java.lang.OutOfMemoryError: Java heap space
01:05:17.777 [pool-1-thread-13] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Historize Name] scheduler task started at 2012/08/16 00:52:01, ended at 2012/08/16 00:57:04. <- Exception caught during update process.
01:05:17.777 [pool-1-thread-21] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Aggregate Report] scheduler task started at 2012/08/11 15:42:28, ended at 2012/08/16 00:43:27. <- Java heap space
01:05:17.777 [pool-1-thread-13] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Historize Name] scheduler task started at 2012/08/16 00:57:20, ended at 2012/08/16 01:02:28. <- Exception caught during update process.
01:05:17.782 [pool-1-thread-5] INFO  c.n.p.a.service.AgregationService - Starting the agregation of the reports
01:18:15.335 [pool-1-thread-12] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Historize Name] scheduler task started at 2012/08/16 01:10:24, ended at 2012/08/16 01:17:50. <- Exception caught during update process.
01:27:24.733 [pool-1-thread-15] ERROR net.liftweb.actor.ActorLogger - Actor threw an exception
java.lang.OutOfMemoryError: Java heap space
Exception in thread "pool-1-thread-15" java.lang.OutOfMemoryError: Java heap space
Exception in thread "pool-1-thread-22" java.lang.OutOfMemoryError: Java heap space
2012-08-16 01:41:55.924:WARN::EXCEPTION
java.lang.OutOfMemoryError: Java heap space
Exception in thread "qtp28290629-79" Exception in thread "qtp28290629-84" java.lang.OutOfMemoryError: Java heap space
Exception in thread "HashSessionScavenger-4" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
01:43:25.302 [pool-1-thread-23] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Historize Name] scheduler task started at 2012/08/16 01:29:52, ended at 2012/08/16 01:41:55. <- Exception caught during update process.
2012-08-16 01:52:36.183:WARN::EXCEPTION
java.lang.OutOfMemoryError: Java heap space
01:52:36.184 [pool-1-thread-20] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Historize Name] scheduler task started at 2012/08/16 01:42:35, ended at 2012/08/16 01:52:36. <- Exception caught during update process.
2012-08-16 01:57:54.626:WARN::Problem scavenging sessions
java.lang.OutOfMemoryError: Java heap space
2012-08-16 01:57:54.626:WARN::EXCEPTION
java.lang.OutOfMemoryError: Java heap space
01:58:19.709 [pool-1-thread-16] ERROR c.n.p.a.b.AbstractScheduler$StatusManager - Error when executing [Historize Name] scheduler task started at 2012/08/16 01:52:36, ended at 2012/08/16 01:57:54. <- Exception caught during update process.
Exception in thread "HashSessionScavenger-8" java.lang.OutOfMemoryError: Java heap space
2012-08-16 02:03:48.771:WARN::Problem scavenging sessions
java.lang.OutOfMemoryError: Java heap space
2012-08-16 02:07:55.224:WARN::/endpoint/upload/
java.lang.OutOfMemoryError: Java heap space
...
11:57:32.378 [pool-1-thread-5] INFO  c.n.p.a.service.AgregationService - Starting the agregation of the reports
12:07:00.045 [pool-1-thread-5] INFO  c.n.p.a.service.AgregationService - Starting the agregation of the reports
Exception in thread "TransparentProxy-58" java.lang.OutOfMemoryError: Java heap space
...