Rudder

UPDATE / WORKAROUND: use 2 CPU¶

Speleogy in proxmox forum let us to think that perhaps the problem was the box configuration with only one cpu.
It seems to be confirmed in comment 10 (https://issues.rudder.io/issues/18717#note-10))

original report¶

One user is not able to start rudder v 6.1.6 or 6.2.0-rc1, neither on centos 7 nor ubuntu 18.04, on a proxmox (KVM) fresh environnement (installation done following rudder documentation).
On centos 7, java version is openJDK 1.8.0_272.

Only logs are:

- in journald/dmesg:

30 11:06:09+00:00##root@#Check if service httpd is started at boot was correct
Nov 30 06:06:48 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: @@Common@@log_info@@hasPolicyServer-root@@common-root@@00@@Check apache process@@None@@2020-11-30 11:06:09+00:00##root@#Ensure service httpd is started at boot was correct
Nov 30 06:06:48 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: @@server-roles@@result_success@@server-roles@@server-roles-directive@@0@@Check apache process@@None@@2020-11-30 11:06:09+00:00##root@#apache service running was correct
Nov 30 06:06:48 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: @@server-roles@@result_success@@server-roles@@server-roles-directive@@0@@Check apache boot script@@None@@2020-11-30 11:06:09+00:00##root@#apache service enabled was correct
Nov 30 06:06:49 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: [INFO] Executing is-active on rudder-jetty using the systemctl method
Nov 30 06:06:49 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: @@Common@@log_info@@hasPolicyServer-root@@common-root@@00@@Check jetty process@@None@@2020-11-30 11:06:09+00:00##root@#Check if the service rudder-jetty is started could not be repaired
Nov 30 06:06:49 xxxxxx cf-agent[13383]: CFEngine(agent) rudder Executing 'no timeout' ... '/bin/systemctl --no-ask-password reset-failed rudder-jetty.service'
Nov 30 06:06:50 xxxxxx cf-agent[13383]: CFEngine(agent) rudder Completed execution of '/bin/systemctl --no-ask-password reset-failed rudder-jetty.service'
Nov 30 06:06:50 xxxxxx cf-agent[13383]: CFEngine(agent) rudder Executing 'no timeout' ... '/bin/systemctl --no-ask-password start rudder-jetty.service'
Nov 30 06:06:51 xxxxxx systemd: Starting Jetty Web Application Server...
Nov 30 06:06:52 xxxxxx su: (to postgres) root on none
Nov 30 06:06:56 xxxxxx rudder-jetty.sh: Setting umask to 0007

- in /var/log/rudder/webapp/YYYY_MM_dd.stderrout.log:

2020-11-30 04:26:00.543:INFO:oejs.SetUIDListener:main: Setting umask=07
2020-11-30 04:26:00.742:INFO:oejs.Server:main: jetty-9.4.32.v20200930; built: 2020-09-30T16:16:37.804Z; git: de97d26f7bd222a0e16831e353d702a7a422f711; jvm 1.8.0_272-b10
2020-11-30 04:26:01.822:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///opt/rudder/share/webapps/] at interval 0

After changing jetty log level to DEBUG in /opt/rudder/etc/rudder-jetty-base/resources/jetty-logging.properties, we observe that the debug log stop with:

2020-11-30 08:53:54.487:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.IterableOnceOps
2020-11-30 08:53:54.503:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.IterableOps
2020-11-30 08:53:54.511:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.IterableFactoryDefaults
2020-11-30 08:53:54.518:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.Iterable
2020-11-30 08:53:54.526:DBUG:oejut.ShutdownThread:JettyShutdownThread:
java.lang.IllegalStateException: !STOPPED
at org.eclipse.jetty.server.handler.HandlerWrapper.destroy(HandlerWrapper.java:140)
at org.eclipse.jetty.util.thread.ShutdownThread.run(ShutdownThread.java:134)
2020-11-30 08:53:54.528:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.immutable.Iterable
2020-11-30 08:53:54.531:DBUG:oejw.WebAppContext:main: isSystemResource==false scala.collection.Seq jar:file:/var/rudder/tmp/jetty/jetty-rudder.war.dir/webapp/WEB-INF/lib/scala-library-2.13.3.jar!/scala/collection/Seq.class
2020-11-30 08:53:54.556:DBUG:oejw.WebAppContext:main: isSystemResource==false scala.PartialFunction jar:file:/var/rudder/tmp/jetty/jetty-rudder.war.dir/webapp/WEB-INF/lib/scala-library-2.13.3.jar!/scala/PartialFunction.class
08:53:54.526:DBUG:oejut.ShutdownThread:JettyShutdownThread:
java.lang.IllegalStateException: !STOPPED

This looks like https://github.com/eclipse/jetty.project/issues/1855 appart that we don't have several war, nor that we even use JSP.