Bug #15075
Updated by François ARMAND almost 5 years ago
In a quite loaded environement, with 2800 nodes and around 1300 report messages/seconds, we reached a point were memory contention was hight and CMS GC was spending a lot of time in full GC.
If we add more memory, it's hard to fine tune the different memory zone ratio (new, eden, etc) to avoid even longer full gc, and when they happen, it's terrible.
We tried G1GC which deals quite well with a heap a >8Go and that kind of load.
So, we should either switch or document how to use G1GC for people who need it.
The condition are that the JVM must be an openjdk, at leat 1.8.something. At least 1.8 is a requirement for rudder 5, so no problem here. But I'm not sure at all that server distro have an openjdk (suse?).
And in most cases, default configuration is sufficient (G1GC is less efficient than CMS, it need more space and has a lesser throughout - the price for better latency).
So, to change the garbage collector used for rudder, one should edit @/opt/rudder/etc/rudder-jetty.conf@ and replace the part with @JAVA_OPTIONS@ by
<pre>
# Java VM arguments
JAVA_OPTIONS_COMMONS="-server
-Xms${JAVA_XMX}m -Xmx${JAVA_XMX}m
-Dfile.encoding=UTF-8
-Drudder.configFile=/opt/rudder/etc/rudder-web.properties
-Drudder.authFile=/opt/rudder/etc/rudder-users.xml
-Dinventoryweb.configFile=/opt/rudder/etc/inventory-web.properties
-Dlogback.configurationFile=/opt/rudder/etc/logback.xml
-Drun.mode=production"
JAVA_OPTIONS_DEFAULT="${JAVA_OPTIONS}
${JAVA_OPTIONS_COMMONS}
-XX:+CMSClassUnloadingEnabled
-XX:+UseConcMarkSweepGC"
JAVA_OPTIONS_G1GC="${JAVA_OPTIONS}
${JAVA_OPTIONS_COMMONS}
-XX:+UseG1GC
-XX:+UnlockExperimentalVMOptions
-XX:MaxGCPauseMillis=500"
# you can add the following options to get more debug info about gc:
JAVA_OPTIONS_G1GC_DEBUG="
-XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCApplicationConcurrentTime
-verbose:gc
-XX:+PrintGCDetails
-Xloggc:/tmp/g1gc.log"
# JAVA_OPTIONS_G1GC="${JAVA_OPTIONS_G1GC} ${JAVA_OPTIONS_G1GC_DEBUG}"
# use default to come back to rudder default GC configuration
#JAVA_OPTIONS=${JAVA_OPTIONS_DEFAULT}
JAVA_OPTIONS=${JAVA_OPTIONS_G1GC}
</pre>
You can also adapt the quantity of ram given to @rudder-jetty@ in file @/etc/default/rudder-jetty@, with the @JAVA_XMX@ parameter
After restarting @rudder-jetty@ service, you should be able to see the parameter list with @ps aux | grep jetty@
<pre>
[root@server]# ps aux | grep jetty
root 5486 38.3 30.2 10216788 4921580 ? Sl 15:32 99:25 /bin/java -server -Xms9000m -Xmx9000m -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=500
-XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -verbose:gc -XX:+PrintGCDetails -Xloggc:/tmp/g1gc.log -Dfile.encoding=UTF-8
-Drudder.configFile=/opt/rudder/etc/rudder-web.properties -Drudder.authFile=/opt/rudder/etc/rudder-users.xml -Dinventoryweb.configFile=/opt/rudder/etc/inventory-web.properties
-Dlogback.configurationFile=/opt/rudder/etc/logback.xml -Drun.mode=production -Djetty.home=/opt/rudder/jetty -Djetty.base=/opt/rudder/etc/rudder-jetty-base -Djava.io.tmpdir=/var/rudder
/tmp/jetty -Djava.io.tmpdir=/var/rudder/tmp/jetty -jar /opt/rudder/jetty/start.jar OPTIONS=Server jetty.state=/opt/rudder/etc/rudder-jetty-base/rudder-jetty.state jetty-started.xml
</pre>