Project

General

Profile

Actions

Bug #18717

closed

In proxmox/kvm env with only one CPU, Rudder-jetty doesn't start without telling why

Added by François ARMAND 6 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
N/A
Category:
System integration
Target version:
Severity:
User visibility:
Effort required:
Priority:
0

Description

UPDATE / WORKAROUND: use 2 CPU

Speleogy in proxmox forum let us to think that perhaps the problem was the box configuration with only one cpu.
It seems to be confirmed in comment 10 (https://issues.rudder.io/issues/18717#note-10))

original report

One user is not able to start rudder v 6.1.6 or 6.2.0-rc1, neither on centos 7 nor ubuntu 18.04, on a proxmox (KVM) fresh environnement (installation done following rudder documentation).
On centos 7, java version is openJDK 1.8.0_272.

Only logs are:

- in journald/dmesg:

30 11:06:09+00:00##root@#Check if service httpd is started at boot was correct
Nov 30 06:06:48 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: @@Common@@log_info@@hasPolicyServer-root@@common-root@@00@@Check apache process@@None@@2020-11-30 11:06:09+00:00##root@#Ensure service httpd is started at boot was correct
Nov 30 06:06:48 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: @@server-roles@@result_success@@server-roles@@server-roles-directive@@0@@Check apache process@@None@@2020-11-30 11:06:09+00:00##root@#apache service running was correct
Nov 30 06:06:48 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: @@server-roles@@result_success@@server-roles@@server-roles-directive@@0@@Check apache boot script@@None@@2020-11-30 11:06:09+00:00##root@#apache service enabled was correct
Nov 30 06:06:49 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: [INFO] Executing is-active on rudder-jetty using the systemctl method
Nov 30 06:06:49 xxxxxx cf-agent[13383]: CFEngine(agent) rudder R: @@Common@@log_info@@hasPolicyServer-root@@common-root@@00@@Check jetty process@@None@@2020-11-30 11:06:09+00:00##root@#Check if the service rudder-jetty is started could not be repaired
Nov 30 06:06:49 xxxxxx cf-agent[13383]: CFEngine(agent) rudder Executing 'no timeout' ... '/bin/systemctl --no-ask-password reset-failed rudder-jetty.service'
Nov 30 06:06:50 xxxxxx cf-agent[13383]: CFEngine(agent) rudder Completed execution of '/bin/systemctl --no-ask-password reset-failed rudder-jetty.service'
Nov 30 06:06:50 xxxxxx cf-agent[13383]: CFEngine(agent) rudder Executing 'no timeout' ... '/bin/systemctl --no-ask-password start rudder-jetty.service'
Nov 30 06:06:51 xxxxxx systemd: Starting Jetty Web Application Server...
Nov 30 06:06:52 xxxxxx su: (to postgres) root on none
Nov 30 06:06:56 xxxxxx rudder-jetty.sh: Setting umask to 0007

- in /var/log/rudder/webapp/YYYY_MM_dd.stderrout.log:

2020-11-30 04:26:00.543:INFO:oejs.SetUIDListener:main: Setting umask=07
2020-11-30 04:26:00.742:INFO:oejs.Server:main: jetty-9.4.32.v20200930; built: 2020-09-30T16:16:37.804Z; git: de97d26f7bd222a0e16831e353d702a7a422f711; jvm 1.8.0_272-b10
2020-11-30 04:26:01.822:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///opt/rudder/share/webapps/] at interval 0

After changing jetty log level to DEBUG in /opt/rudder/etc/rudder-jetty-base/resources/jetty-logging.properties, we observe that the debug log stop with:

2020-11-30 08:53:54.487:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.IterableOnceOps
2020-11-30 08:53:54.503:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.IterableOps
2020-11-30 08:53:54.511:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.IterableFactoryDefaults
2020-11-30 08:53:54.518:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.Iterable
2020-11-30 08:53:54.526:DBUG:oejut.ShutdownThread:JettyShutdownThread:
java.lang.IllegalStateException: !STOPPED
at org.eclipse.jetty.server.handler.HandlerWrapper.destroy(HandlerWrapper.java:140)
at org.eclipse.jetty.util.thread.ShutdownThread.run(ShutdownThread.java:134)
2020-11-30 08:53:54.528:DBUG:oejw.WebAppClassLoader:main: WAP webapp loaded interface scala.collection.immutable.Iterable
2020-11-30 08:53:54.531:DBUG:oejw.WebAppContext:main: isSystemResource==false scala.collection.Seq jar:file:/var/rudder/tmp/jetty/jetty-rudder.war.dir/webapp/WEB-INF/lib/scala-library-2.13.3.jar!/scala/collection/Seq.class
2020-11-30 08:53:54.556:DBUG:oejw.WebAppContext:main: isSystemResource==false scala.PartialFunction jar:file:/var/rudder/tmp/jetty/jetty-rudder.war.dir/webapp/WEB-INF/lib/scala-library-2.13.3.jar!/scala/PartialFunction.class
08:53:54.526:DBUG:oejut.ShutdownThread:JettyShutdownThread:
java.lang.IllegalStateException: !STOPPED

This looks like https://github.com/eclipse/jetty.project/issues/1855 appart that we don't have several war, nor that we even use JSP.

Actions #1

Updated by François ARMAND 5 months ago

Update: it works on ubuntu on AWS in place of proxmox, so it may be a virtualisation related problem.

Actions #2

Updated by Vincent MEMBRÉ 5 months ago

  • Target version changed from 6.1.7 to 6.1.8
Actions #3

Updated by Vincent MEMBRÉ 4 months ago

  • Target version changed from 6.1.8 to 6.1.9
Actions #4

Updated by Vincent MEMBRÉ 4 months ago

  • Target version changed from 6.1.9 to 6.1.10
Actions #5

Updated by Vincent MEMBRÉ 2 months ago

  • Target version changed from 6.1.10 to 6.1.11
Actions #6

Updated by Vincent MEMBRÉ about 2 months ago

  • Target version changed from 6.1.11 to 6.1.12
Actions #7

Updated by matthieu castellazzi about 2 months ago

Good morning,

A quick comment to let you know that I have exactly the same issue.
I'm using proxmox (6.3.6 now) on my homelab and have been running the latest versions of rudder on it for the last few years without any issues.
All started to go wrong late 2020 and I had to spend some time trying to debug the thing without success. I cannot tell if it was a proxmox of rudder update that started the issue.
I tried to roll back my proxmox below version 6 without sucess.
Same issue if using proxmox VM or CT. Same issue if using Ubuntu 18.04, debian 10, Fedora 33 or CentOS 7 (my preference).
A simple instance of jetty (without rudder) starts fine on the lastest CentOS 7 and latest proxmox.
I've tried to update the 9.4 jetty version used by rudder to 10 and 11, not better.
So somehow jetty is failing to start rudder on top of proxmox, I'm not a java dev so I've trouble to understand where things are going wrong.
Looks like something in rudder war is not playing nicely with jetty....

I have 3 other instances of rudder at work, working nicely on VMware for years but at home I would prefer to stick with proxmox for economic and ease of use reasons.

Thanks you for not closing this ticket and still trying to fix it. And thanks for all the good work you are putting into rudder !

Matthieu

Actions #8

Updated by François ARMAND about 2 months ago

Hello Matthieu,

Since we don't have access to a proxmox env, it's hard for us to understand what going wrong.
All the information we found were case that don't apply (in the linked ticket) or about the need to have at least 2 CPU in the box (but why it would have change with recent rudder?).

The best would be if you could provide an access to a proxmox box where we can test & try things (the access can be limited to me).

Actions #9

Updated by François ARMAND about 2 months ago

  • Subject changed from Rudder-jetty doesn't start without telling why to In proxmox env, Rudder-jetty doesn't start without telling why
Actions #10

Updated by matthieu castellazzi about 2 months ago

Hello Francois,

I've setup a proxmox env for you on a dedicated server and did a quick rudder install and this time it worked nicely. :-(

The only change I made was to allocate 2 CPUS... At home I was allocating only 1 to save some resources (I'm pretty sure it used to run on 1 at some time in the past).
I should have read more carefully your last comment about the 2 CPUs requirement... my bad.
So it is all good from my side, now I know that jetty won't start rudder on a single CPU.

Thanks for your help anyway, will be able to use again rudder on my home lab now.

Keep on the good work :-)

Cheers, Matthieu

Actions #11

Updated by François ARMAND about 1 month ago

  • Subject changed from In proxmox env, Rudder-jetty doesn't start without telling why to In proxmox env with only one CPU, Rudder-jetty doesn't start without telling why
  • Description updated (diff)
  • Status changed from New to Resolved

Oh, this is fantastic!

To be clear, I have zero idea as for why the jvm needs 2 CPU on proxmod. We have all our testing box with one CPU (on vbox), and it works (with corresponding speed and all, but it works).

I'm very glad it worked.

For reference, there is that old thread about proxmox and 2 CPU for the JVM: https://forum.proxmox.com/threads/problem-with-java-aplications.9380/

Actions #12

Updated by François ARMAND about 1 month ago

  • Subject changed from In proxmox env with only one CPU, Rudder-jetty doesn't start without telling why to In proxmox/kvm env with only one CPU, Rudder-jetty doesn't start without telling why
Actions #13

Updated by François ARMAND about 1 month ago

@matthieu: would you mind trying to limit to 1 to see if it breaks the new env?

And yes, there is still something strange: it uses to work with 1 cpu. It can be a change in kvm (but you tested in both centos 7 and ubuntu 18_04, which are not exactly recent distro), in proxmox, in the jvm version, in the kernel thread management (something related to meltdown/spectre & all mitigation?), in rudder (either jvm config or - unlikely given the logs - app)...

Actions

Also available in: Atom PDF