Bug #18717
closed
In proxmox/kvm env with only one CPU, Rudder-jetty doesn't start without telling why
Description
UPDATE / WORKAROUND: use 2 CPU¶
Speleogy in proxmox forum let us to think that perhaps the problem was the box configuration with only one cpu.
It seems to be confirmed in comment 10 (https://issues.rudder.io/issues/18717#note-10))
original report¶
One user is not able to start rudder v 6.1.6 or 6.2.0-rc1, neither on centos 7 nor ubuntu 18.04, on a proxmox (KVM) fresh environnement (installation done following rudder documentation).
On centos 7, java version is openJDK 1.8.0_272.
Only logs are:
- in journald/dmesg:
30 11:06:09+00:00##root@#Check if service httpd is started at boot was correct Nov 30 06:06:48 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: @@Common@@log_info@@hasPolicyServer-root@@common-root@@00@@Check apache process@@None@@2020-11-30 11:06:09+00:00##root@#Ensure service httpd is started at boot was correct Nov 30 06:06:48 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: @@server-roles@@result_success@@server-roles@@server-roles-directive@@0@@Check apache process@@None@@2020-11-30 11:06:09+00:00##root@#apache service running was correct Nov 30 06:06:48 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: @@server-roles@@result_success@@server-roles@@server-roles-directive@@0@@Check apache boot script@@None@@2020-11-30 11:06:09+00:00##root@#apache service enabled was correct Nov 30 06:06:49 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: [INFO] Executing is-active on rudder-jetty using the systemctl method Nov 30 06:06:49 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: @@Common@@log_info@@hasPolicyServer-root@@common-root@@00@@Check jetty process@@None@@2020-11-30 11:06:09+00:00##root@#Check if the service rudder-jetty is started could not be repaired Nov 30 06:06:49 xxxxxx cf-agent[13383]: CFEngine(agent) rudder Executing 'no timeout' ... '/bin/systemctl --no-ask-password reset-failed rudder-jetty.service' Nov 30 06:06:50 xxxxxx cf-agent[13383]: CFEngine(agent) rudder Completed execution of '/bin/systemctl --no-ask-password reset-failed rudder-jetty.service' Nov 30 06:06:50 xxxxxx cf-agent[13383]: CFEngine(agent) rudder Executing 'no timeout' ... '/bin/systemctl --no-ask-password start rudder-jetty.service' Nov 30 06:06:51 xxxxxx systemd: Starting Jetty Web Application Server... Nov 30 06:06:52 xxxxxx su: (to postgres) root on none Nov 30 06:06:56 xxxxxx rudder-jetty.sh: Setting umask to 0007
- in /var/log/rudder/webapp/YYYY_MM_dd.stderrout.log:
2020-11-30 04:26:00.543:INFO:oejs.SetUIDListener:main: Setting umask=07 2020-11-30 04:26:00.742:INFO:oejs.Server:main: jetty-9.4.32.v20200930; built: 2020-09-30T16:16:37.804Z; git: de97d26f7bd222a0e16831e353d702a7a422f711; jvm 1.8.0_272-b10 2020-11-30 04:26:01.822:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///opt/rudder/share/webapps/] at interval 0
After changing jetty log level to DEBUG in /opt/rudder/etc/rudder-jetty-base/resources/jetty-logging.properties, we observe that the debug log stop with:
2020-11-30 08:53:54.487:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.IterableOnceOps 2020-11-30 08:53:54.503:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.IterableOps 2020-11-30 08:53:54.511:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.IterableFactoryDefaults 2020-11-30 08:53:54.518:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.Iterable 2020-11-30 08:53:54.526:DBUG:oejut.ShutdownThread:JettyShutdownThread: java.lang.IllegalStateException: !STOPPED at org.eclipse.jetty.server.handler.HandlerWrapper.destroy(HandlerWrapper.java:140) at org.eclipse.jetty.util.thread.ShutdownThread.run(ShutdownThread.java:134) 2020-11-30 08:53:54.528:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.immutable.Iterable 2020-11-30 08:53:54.531:DBUG:oejw.WebAppContext:main: isSystemResource==false scala.collection.Seq jar:file:/var/rudder/tmp/jetty/jetty-rudder.war.dir/webapp/WEB-INF/lib/scala-library-2.13.3.jar!/scala/collection/Seq.class 2020-11-30 08:53:54.556:DBUG:oejw.WebAppContext:main: isSystemResource==false scala.PartialFunction jar:file:/var/rudder/tmp/jetty/jetty-rudder.war.dir/webapp/WEB-INF/lib/scala-library-2.13.3.jar!/scala/PartialFunction.class 08:53:54.526:DBUG:oejut.ShutdownThread:JettyShutdownThread: java.lang.IllegalStateException: !STOPPED
This looks like https://github.com/eclipse/jetty.project/issues/1855 appart that we don't have several war, nor that we even use JSP.
Updated by François ARMAND almost 4 years ago
Update: it works on ubuntu on AWS in place of proxmox, so it may be a virtualisation related problem.
Updated by Vincent MEMBRÉ almost 4 years ago
- Target version changed from 6.1.7 to 6.1.8
Updated by Vincent MEMBRÉ almost 4 years ago
- Target version changed from 6.1.8 to 6.1.9
Updated by Vincent MEMBRÉ almost 4 years ago
- Target version changed from 6.1.9 to 6.1.10
Updated by Vincent MEMBRÉ over 3 years ago
- Target version changed from 6.1.10 to 6.1.11
Updated by Vincent MEMBRÉ over 3 years ago
- Target version changed from 6.1.11 to 6.1.12
Updated by matthieu castellazzi over 3 years ago
Good morning,
A quick comment to let you know that I have exactly the same issue.
I'm using proxmox (6.3.6 now) on my homelab and have been running the latest versions of rudder on it for the last few years without any issues.
All started to go wrong late 2020 and I had to spend some time trying to debug the thing without success. I cannot tell if it was a proxmox of rudder update that started the issue.
I tried to roll back my proxmox below version 6 without sucess.
Same issue if using proxmox VM or CT. Same issue if using Ubuntu 18.04, debian 10, Fedora 33 or CentOS 7 (my preference).
A simple instance of jetty (without rudder) starts fine on the lastest CentOS 7 and latest proxmox.
I've tried to update the 9.4 jetty version used by rudder to 10 and 11, not better.
So somehow jetty is failing to start rudder on top of proxmox, I'm not a java dev so I've trouble to understand where things are going wrong.
Looks like something in rudder war is not playing nicely with jetty....
I have 3 other instances of rudder at work, working nicely on VMware for years but at home I would prefer to stick with proxmox for economic and ease of use reasons.
Thanks you for not closing this ticket and still trying to fix it. And thanks for all the good work you are putting into rudder !
Matthieu
Updated by François ARMAND over 3 years ago
Hello Matthieu,
Since we don't have access to a proxmox env, it's hard for us to understand what going wrong.
All the information we found were case that don't apply (in the linked ticket) or about the need to have at least 2 CPU in the box (but why it would have change with recent rudder?).
The best would be if you could provide an access to a proxmox box where we can test & try things (the access can be limited to me).
Updated by François ARMAND over 3 years ago
- Subject changed from Rudder-jetty doesn't start without telling why to In proxmox env, Rudder-jetty doesn't start without telling why
Updated by matthieu castellazzi over 3 years ago
Hello Francois,
I've setup a proxmox env for you on a dedicated server and did a quick rudder install and this time it worked nicely. :-(
The only change I made was to allocate 2 CPUS... At home I was allocating only 1 to save some resources (I'm pretty sure it used to run on 1 at some time in the past).
I should have read more carefully your last comment about the 2 CPUs requirement... my bad.
So it is all good from my side, now I know that jetty won't start rudder on a single CPU.
Thanks for your help anyway, will be able to use again rudder on my home lab now.
Keep on the good work :-)
Cheers, Matthieu
Updated by François ARMAND over 3 years ago
- Subject changed from In proxmox env, Rudder-jetty doesn't start without telling why to In proxmox env with only one CPU, Rudder-jetty doesn't start without telling why
- Description updated (diff)
- Status changed from New to Resolved
Oh, this is fantastic!
To be clear, I have zero idea as for why the jvm needs 2 CPU on proxmod. We have all our testing box with one CPU (on vbox), and it works (with corresponding speed and all, but it works).
I'm very glad it worked.
For reference, there is that old thread about proxmox and 2 CPU for the JVM: https://forum.proxmox.com/threads/problem-with-java-aplications.9380/
Updated by François ARMAND over 3 years ago
- Subject changed from In proxmox env with only one CPU, Rudder-jetty doesn't start without telling why to In proxmox/kvm env with only one CPU, Rudder-jetty doesn't start without telling why
Updated by François ARMAND over 3 years ago
@matthieu: would you mind trying to limit to 1 to see if it breaks the new env?
And yes, there is still something strange: it uses to work with 1 cpu. It can be a change in kvm (but you tested in both centos 7 and ubuntu 18_04, which are not exactly recent distro), in proxmox, in the jvm version, in the kernel thread management (something related to meltdown/spectre & all mitigation?), in rudder (either jvm config or - unlikely given the logs - app)...