Rudder

Ah, the kind of things making you love java....

From an user experience point of view, I think it be better to kill the JVM on such scenario, so that at least we don't have to deal with a zombie app (and so, our agent can detect it's no more running and start it again).

Of course, it would be even better to not have such error, I'm talking about a remission scenario here.

Some more info: http://stackoverflow.com/questions/12096403/java-shutting-down-on-out-of-memory-error

We could also make the OOM detection be done in a cron, looking in the last log, and restarting Rudder when Needed.

To better explain why the -XX:OnOutOfMemoryError is not a very good solution:

On a VM with 2,5G of RAM, default Centos6, no space, where Oom is a java program that is just allocating big arrays:

At 1G heap space, everything works as expected:

root@test:~/testOOM# java6 -Xms1000m -Xmx1000m -XX:OnOutOfMemoryError="kill -9 %p && echo 'such memory, much killness, wow'" Oom
start
assigning array num. 0
assigning array num. 1000
assigning array num. 2000
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p && echo 'such memory, much killness, wow'" 
#   Executing /bin/sh -c "kill -9 3613 && echo 'such memory, much killness, wow'"...
such memory, much killness, wow
Killed

At 1,3Go heap space, the forked children get killed before having a chance to kick off, NOT doing it's job:

root@test:~/testOOM# java6 -Xms1300m -Xmx1300m -XX:OnOutOfMemoryError="kill -9 %p && echo 'such memory, much killness, wow'" Oom
start
assigning array num. 0
assigning array num. 1000
assigning array num. 2000
assigning array num. 3000
assigning array num. 4000
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p && echo 'such memory, much killness, wow'" 
#   Executing /bin/sh -c "kill -9 3601 && echo 'such memory, much killness, wow'"...
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at Oom.main(Oom.java:18)

(actually, it's even befoare 1,3Go, because heap space is just a little party of the java RAM process, and Linux has some reserved memory, and with Xms we ask for a preallocation of the whole memory segment. But you get the basic idea: around 50% of RAM allocated to the heap, -XX:OnOutOfMemoryError stop being usefull)

And at higher level of RAM, linux is killing directly the process, not letting OOM appear (this case is out of scope, it's for information):

root@test:~/testOOM# java6 -Xms2300m -Xmx2300m -XX:OnOutOfMemoryError="kill -9 %p && echo 'such memory, much killness, wow'" Oom
start
assigning array num. 0
assigning array num. 1000
assigning array num. 2000
assigning array num. 3000
assigning array num. 4000
Killed

One thing to take care of is the presence of the various heap and crash dumps the JVM will litter around, one set per OOM.
Those can be pretty troublesome as they fill up the filesystem.
In our case we got really lucky in that we had already separated the postgresql and ldap directories.
Others may not and might face OOM-related corruption on the other components.

Note: in JVM 8u92, two jvm options were introduced: http://www.oracle.com/technetwork/java/javase/8u92-relnotes-2949471.html

- ExitOnOutOfMemoryError
- CrashOnOutOfMemoryError

It seems to be the behavior that we want.

We should enable one or the other when we detect that the JVM version is sufficiently hight to support them.

Some more precision on JVM supporting these flags:

- hotspot and openjdk JDK 8u92 (1.8.0_92-b14) and up
- openjdk
- jrockit, any java 7/8
- not on IBM JVM

We have to option, I'm not sure which one we prefer. I think crash, so that we have a chance to know what part of the code OOMed.

Crash example:

% scala -J-XX:+CrashOnOutOfMemoryError

scala> val a = new Array[Array[Int]](10000)
a: Array[Array[Int]] = Array(null, null, null, null, null, ...

scala> for( i <- 0 until 10000) { a(i) = new Array(100000) }
Aborting due to java.lang.OutOfMemoryError: Java heap space
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (debug.cpp:308), pid=2125, tid=0x00007f81e7ffa700
#  fatal error: OutOfMemory encountered: Java heap space
#
# JRE version: Java(TM) SE Runtime Environment (8.0_101-b13) (build 1.8.0_101-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.101-b13 mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: /home/fanf/core or core.2125
#
# An error report file with more information is saved as:
# /home/fanf/hs_err_pid2125.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
/home/fanf/java/scala/bin/scala : ligne 21 :  2125 Abandon                 (core dumped)"$@" 

Or simply exit:

== [fanf@luhman16] ~ (master●) ==
% scala -J-XX:+ExitOnOutOfMemoryError
Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101).
Type in expressions for evaluation. Or try :help.

scala> val a = new Array[Array[Int]](10000)
a: Array[Array[Int]] = Array(null, null, null, null, null, ...

scala> for( i <- 0 until 10000) { a(i) = new Array(100000) }
Terminating due to java.lang.OutOfMemoryError: Java heap space