Rudder

In a quite loaded environement, with 2800 nodes and around 1300 report messages/seconds, we reached a point were memory contention was hight and CMS GC was spending a lot of time in full GC.
If we add more memory, it's hard to fine tune the different memory zone ratio (new, eden, etc) to avoid even longer full gc, and when they happen, it's terrible.

We tried G1GC which deals quite well with a heap a >8Go and that kind of load.

So, we should either switch or document how to use G1GC for people who need it.
The condition are that the JVM must be an openjdk, at leat 1.8.something. At least 1.8 is a requirement for rudder 5, so no problem here. But I'm not sure at all that server distro have an openjdk (suse?).

And in most cases, default configuration is sufficient (G1GC is less efficient than CMS, it need more space and has a lesser throughout - the price for better latency).

So, to change the garbage collector used for rudder, one should edit /opt/rudder/etc/rudder-jetty.conf and replace the part with JAVA_OPTIONS by

# Java VM arguments
JAVA_OPTIONS_COMMONS="-server
-Xms${JAVA_XMX}m -Xmx${JAVA_XMX}m
-Dfile.encoding=UTF-8
-Drudder.configFile=/opt/rudder/etc/rudder-web.properties
-Drudder.authFile=/opt/rudder/etc/rudder-users.xml
-Dinventoryweb.configFile=/opt/rudder/etc/inventory-web.properties
-Dlogback.configurationFile=/opt/rudder/etc/logback.xml
-Drun.mode=production" 

JAVA_OPTIONS_DEFAULT="${JAVA_OPTIONS}
${JAVA_OPTIONS_COMMONS}
-XX:+CMSClassUnloadingEnabled
-XX:+UseConcMarkSweepGC" 

JAVA_OPTIONS_G1GC="${JAVA_OPTIONS}
${JAVA_OPTIONS_COMMONS}
-XX:+UseG1GC
-XX:+UnlockExperimentalVMOptions
-XX:MaxGCPauseMillis=500" 

# you can add the following options to get more debug info about gc:
JAVA_OPTIONS_G1GC_DEBUG=" 
-XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCApplicationConcurrentTime
-verbose:gc
-XX:+PrintGCDetails
-Xloggc:/tmp/g1gc.log" 

# JAVA_OPTIONS_G1GC="${JAVA_OPTIONS_G1GC} ${JAVA_OPTIONS_G1GC_DEBUG}" 

# use default to come back to rudder default GC configuration
#JAVA_OPTIONS=${JAVA_OPTIONS_DEFAULT}
JAVA_OPTIONS=${JAVA_OPTIONS_G1GC}

You can also adapt the quantity of ram given to rudder-jetty in file /etc/default/rudder-jetty, with the JAVA_XMX parameter

After restarting rudder-jetty service, you should be able to see the parameter list with ps aux | grep jetty

[root@server]# ps aux | grep jetty
root      5486 38.3 30.2 10216788 4921580 ?    Sl   15:32  99:25 /bin/java -server -Xms9000m -Xmx9000m -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=500 
-XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -verbose:gc -XX:+PrintGCDetails -Xloggc:/tmp/g1gc.log -Dfile.encoding=UTF-8 
-Drudder.configFile=/opt/rudder/etc/rudder-web.properties -Drudder.authFile=/opt/rudder/etc/rudder-users.xml -Dinventoryweb.configFile=/opt/rudder/etc/inventory-web.properties 
-Dlogback.configurationFile=/opt/rudder/etc/logback.xml -Drun.mode=production -Djetty.home=/opt/rudder/jetty -Djetty.base=/opt/rudder/etc/rudder-jetty-base -Djava.io.tmpdir=/var/rudder
/tmp/jetty -jar /opt/rudder/jetty/start.jar OPTIONS=Server jetty.state=/opt/rudder/etc/rudder-jetty-base/rudder-jetty.state jetty-started.xml