Project

General

Profile

Actions

Bug #15075

closed

Allow overriding gc options for the webapp JVM to better deal with high volumes

Added by François ARMAND almost 5 years ago. Updated almost 4 years ago.

Status:
Released
Priority:
N/A
Category:
Performance and scalability
Target version:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
Reviewed
Fix check:
Checked
Regression:

Description

In a quite loaded environement, with 2800 nodes and around 1300 report messages/seconds, we reached a point were memory contention was hight and CMS GC was spending a lot of time in full GC.
If we add more memory, it's hard to fine tune the different memory zone ratio (new, eden, etc) to avoid even longer full gc, and when they happen, it's terrible.

We tried G1GC which deals quite well with a heap a >8Go and that kind of load.

So, we should either switch or document how to use G1GC for people who need it.
The condition are that the JVM must be an openjdk, at leat 1.8.something. At least 1.8 is a requirement for rudder 5, so no problem here. But I'm not sure at all that server distro have an openjdk (suse?).

And in most cases, default configuration is sufficient (G1GC is less efficient than CMS, it need more space and has a lesser throughout - the price for better latency).

So, to change the garbage collector used for rudder, one should edit /opt/rudder/etc/rudder-jetty.conf and replace the part with JAVA_OPTIONS by


# Java VM arguments
JAVA_OPTIONS_COMMONS="-server
-Xms${JAVA_XMX}m -Xmx${JAVA_XMX}m
-Dfile.encoding=UTF-8
-Drudder.configFile=/opt/rudder/etc/rudder-web.properties
-Drudder.authFile=/opt/rudder/etc/rudder-users.xml
-Dinventoryweb.configFile=/opt/rudder/etc/inventory-web.properties
-Dlogback.configurationFile=/opt/rudder/etc/logback.xml
-Drun.mode=production" 

JAVA_OPTIONS_DEFAULT="${JAVA_OPTIONS}
${JAVA_OPTIONS_COMMONS}
-XX:+CMSClassUnloadingEnabled
-XX:+UseConcMarkSweepGC" 

JAVA_OPTIONS_G1GC="${JAVA_OPTIONS}
${JAVA_OPTIONS_COMMONS}
-XX:+UseG1GC
-XX:+UnlockExperimentalVMOptions
-XX:MaxGCPauseMillis=500" 

# you can add the following options to get more debug info about gc:
JAVA_OPTIONS_G1GC_DEBUG=" 
-XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCApplicationConcurrentTime
-verbose:gc
-XX:+PrintGCDetails
-Xloggc:/tmp/g1gc.log" 

# JAVA_OPTIONS_G1GC="${JAVA_OPTIONS_G1GC} ${JAVA_OPTIONS_G1GC_DEBUG}" 

# use default to come back to rudder default GC configuration
#JAVA_OPTIONS=${JAVA_OPTIONS_DEFAULT}
JAVA_OPTIONS=${JAVA_OPTIONS_G1GC}

You can also adapt the quantity of ram given to rudder-jetty in file /etc/default/rudder-jetty, with the JAVA_XMX parameter

After restarting rudder-jetty service, you should be able to see the parameter list with ps aux | grep jetty

[root@server]# ps aux | grep jetty
root      5486 38.3 30.2 10216788 4921580 ?    Sl   15:32  99:25 /bin/java -server -Xms9000m -Xmx9000m -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=500 
-XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -verbose:gc -XX:+PrintGCDetails -Xloggc:/tmp/g1gc.log -Dfile.encoding=UTF-8 
-Drudder.configFile=/opt/rudder/etc/rudder-web.properties -Drudder.authFile=/opt/rudder/etc/rudder-users.xml -Dinventoryweb.configFile=/opt/rudder/etc/inventory-web.properties 
-Dlogback.configurationFile=/opt/rudder/etc/logback.xml -Drun.mode=production -Djetty.home=/opt/rudder/jetty -Djetty.base=/opt/rudder/etc/rudder-jetty-base -Djava.io.tmpdir=/var/rudder
/tmp/jetty -jar /opt/rudder/jetty/start.jar OPTIONS=Server jetty.state=/opt/rudder/etc/rudder-jetty-base/rudder-jetty.state jetty-started.xml

Subtasks 1 (0 open1 closed)

Bug #15116: Document use G1GC to better deal with high volume of nodes and reportsReleasedAlexis MoussetActions
Actions

Also available in: Atom PDF