Project

General

Profile

Bug #15075

Allow overriding gc options for the webapp JVM to better deal with high volumes

Added by François ARMAND 2 months ago. Updated about 2 months ago.

Status:
Pending release
Priority:
N/A
Category:
Performance and scalability
Target version:
Severity:
User visibility:
Effort required:
Priority:
0

Description

In a quite loaded environement, with 2800 nodes and around 1300 report messages/seconds, we reached a point were memory contention was hight and CMS GC was spending a lot of time in full GC.
If we add more memory, it's hard to fine tune the different memory zone ratio (new, eden, etc) to avoid even longer full gc, and when they happen, it's terrible.

We tried G1GC which deals quite well with a heap a >8Go and that kind of load.

So, we should either switch or document how to use G1GC for people who need it.
The condition are that the JVM must be an openjdk, at leat 1.8.something. At least 1.8 is a requirement for rudder 5, so no problem here. But I'm not sure at all that server distro have an openjdk (suse?).

And in most cases, default configuration is sufficient (G1GC is less efficient than CMS, it need more space and has a lesser throughout - the price for better latency).

So, to change the garbage collector used for rudder, one should edit /opt/rudder/etc/rudder-jetty.conf and replace the part with JAVA_OPTIONS by


# Java VM arguments
JAVA_OPTIONS_COMMONS="-server
-Xms${JAVA_XMX}m -Xmx${JAVA_XMX}m
-Dfile.encoding=UTF-8
-Drudder.configFile=/opt/rudder/etc/rudder-web.properties
-Drudder.authFile=/opt/rudder/etc/rudder-users.xml
-Dinventoryweb.configFile=/opt/rudder/etc/inventory-web.properties
-Dlogback.configurationFile=/opt/rudder/etc/logback.xml
-Drun.mode=production" 

JAVA_OPTIONS_DEFAULT="${JAVA_OPTIONS}
${JAVA_OPTIONS_COMMONS}
-XX:+CMSClassUnloadingEnabled
-XX:+UseConcMarkSweepGC" 

JAVA_OPTIONS_G1GC="${JAVA_OPTIONS}
${JAVA_OPTIONS_COMMONS}
-XX:+UseG1GC
-XX:+UnlockExperimentalVMOptions
-XX:MaxGCPauseMillis=500" 

# you can add the following options to get more debug info about gc:
JAVA_OPTIONS_G1GC_DEBUG=" 
-XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCApplicationConcurrentTime
-verbose:gc
-XX:+PrintGCDetails
-Xloggc:/tmp/g1gc.log" 

# JAVA_OPTIONS_G1GC="${JAVA_OPTIONS_G1GC} ${JAVA_OPTIONS_G1GC_DEBUG}" 

# use default to come back to rudder default GC configuration
#JAVA_OPTIONS=${JAVA_OPTIONS_DEFAULT}
JAVA_OPTIONS=${JAVA_OPTIONS_G1GC}

You can also adapt the quantity of ram given to rudder-jetty in file /etc/default/rudder-jetty, with the JAVA_XMX parameter

After restarting rudder-jetty service, you should be able to see the parameter list with ps aux | grep jetty

[root@server]# ps aux | grep jetty
root      5486 38.3 30.2 10216788 4921580 ?    Sl   15:32  99:25 /bin/java -server -Xms9000m -Xmx9000m -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=500 
-XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -verbose:gc -XX:+PrintGCDetails -Xloggc:/tmp/g1gc.log -Dfile.encoding=UTF-8 
-Drudder.configFile=/opt/rudder/etc/rudder-web.properties -Drudder.authFile=/opt/rudder/etc/rudder-users.xml -Dinventoryweb.configFile=/opt/rudder/etc/inventory-web.properties 
-Dlogback.configurationFile=/opt/rudder/etc/logback.xml -Drun.mode=production -Djetty.home=/opt/rudder/jetty -Djetty.base=/opt/rudder/etc/rudder-jetty-base -Djava.io.tmpdir=/var/rudder
/tmp/jetty -jar /opt/rudder/jetty/start.jar OPTIONS=Server jetty.state=/opt/rudder/etc/rudder-jetty-base/rudder-jetty.state jetty-started.xml

Subtasks

Bug #15116: Document use G1GC to better deal with high volume of nodes and reportsDiscussionFrançois ARMANDActions

Associated revisions

Revision 98239b61 (diff)
Added by François ARMAND 2 months ago

Fixes #15075: Use G1GC to better deal with high volume of nodes and reports

History

#1

Updated by François ARMAND 2 months ago

  • Description updated (diff)
#2

Updated by François ARMAND 2 months ago

  • Description updated (diff)
#3

Updated by François ARMAND 2 months ago

  • Status changed from Discussion to In progress

So, it was decided that:

- CMS remains the default GC,
- we change /opt/rudder/etc/rudder-jetty.conf to make the GC configurable in JAVA_OPTIONS
- the configuration is documented to /etc/defaults/rudder-jetty
- add a section in performance (or requirement) that explain that for big HEAP (6Go or more), G1GC should be used.

#4

Updated by François ARMAND 2 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from François ARMAND to Alexis MOUSSET
  • Pull Request set to https://github.com/Normation/rudder-packages/pull/1971
#5

Updated by François ARMAND 2 months ago

  • Status changed from Pending technical review to Pending release
#9

Updated by Alexis MOUSSET about 2 months ago

  • Subject changed from Use G1GC to better deal with high volume of nodes and reports to Allow overriding gc options for the webapp JVM to better deal with high volumes
#10

Updated by Vincent MEMBRÉ about 2 months ago

This bug has been fixed in Rudder 5.0.12 which was released today.

Also available in: Atom PDF