Bug #21053: Rudder-jetty doesn't start in 7.0 - Rudder - Issue Tracker

Actions

Copy link

Bug #21053

closed

Rudder-jetty doesn't start in 7.0

Added by Nicolas CHARLES almost 2 years ago. Updated 8 months ago.

Status:

Resolved

Priority:

N/A

Assignee:

Category:

System integration

Target version:

7.2.11

Pull Request:

Severity:

UX impact:

It bothers me each time

User visibility:

Effort required:

Priority:

Name check:

To do

Fix check:

To do

Regression:

Description

I don't really know how it happened, but I got to a state where rudder-jetty wouldn't start
Nothing happens when running

service rudder-jetty start
Job for rudder-jetty.service failed because the control process exited with error code.
See "systemctl status rudder-jetty.service" and "journalctl -xe" for details.

No logs in /var/log/rudder nor systemctl

running

bash /opt/rudder/bin/rudder-jetty.sh start
/opt/rudder/etc/rudder-jetty.conf: line 72: warning: setlocale: LC_NUMERIC: cannot change locale ()
/opt/rudder/etc/rudder-jetty.conf: line 72: warning: setlocale: LC_NUMERIC: cannot change locale ()
/opt/rudder/etc/rudder-jetty.conf: line 73: warning: setlocale: LC_TIME: cannot change locale ()
/opt/rudder/etc/rudder-jetty.conf: line 73: warning: setlocale: LC_TIME: cannot change locale ()
Setting umask to 0007
Starting Jetty: process already running.

shows that it thinks it is running, but it is not

command

start-stop-daemon -k 0007 -S -p/var/rudder/run/rudder-jetty.pid -d/opt/rudder/etc/rudder-jetty-base -b -m -a /bin/java -- -server -Xms1024m -Xmx1024m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dfile.encoding=UTF-8 -Drudder.configFile=/opt/rudder/etc/rudder-web.properties -Drudder.authFile=/opt/rudder/etc/rudder-users.xml -Dlogback.configurationFile=/opt/rudder/etc/logback.xml -Drun.mode=production -XX:+CrashOnOutOfMemoryError -Djetty.home=/opt/rudder/jetty -Djetty.base=/opt/rudder/etc/rudder-jetty-base -Djava.io.tmpdir=/var/rudder/tmp/jetty -jar /opt/rudder/jetty/start.jar OPTIONS=Server jetty.state=/opt/rudder/etc/rudder-jetty-base/rudder-jetty.state jetty-started.xml start-log-file=/var/log/rudder/webapp/java.log
process already running.

so that's the culprit

file /var/rudder/run/rudder-jetty.pid does exist, with PID 877
but no process with that ID

moving the pid file and running does

 service rudder-jetty start
Job for rudder-jetty.service failed because the control process exited with error code.
See "systemctl status rudder-jetty.service" and "journalctl -xe" for details.

but service is started ...

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Nicolas CHARLES almost 2 years ago

Interstingly, file ./opt/rudder/etc/rudder-jetty-base/rudder-jetty.state exists in 6.2 but not in 7.0, even thought files /opt/rudder/bin/rudder-jetty.sh are identical

Actions

Copy link

Updated by Nicolas CHARLES almost 2 years ago

Killing rudder-jetty and doing a status returns:
In 6.2

root@server:/# systemctl status rudder-jetty
● rudder-jetty.service - Jetty Web Application Server
   Loaded: loaded (/lib/systemd/system/rudder-jetty.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Thu 2022-05-05 10:11:06 CEST; 57s ago
  Process: 6423 ExecStart=/opt/rudder/bin/rudder-jetty.sh start (code=exited, status=0/SUCCESS)
 Main PID: 6487 (code=killed, signal=KILL)

May 05 10:11:06 server systemd[1]: rudder-jetty.service: Main process exited, code=killed, status=9/KILL
May 05 10:11:06 server systemd[1]: rudder-jetty.service: Failed with result 'signal'.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

in 7.0

root@server:/var/rudder/configuration-repository/techniques/system# systemctl status rudder-jetty
● rudder-jetty.service - Jetty Web Application Server
     Loaded: loaded (/lib/systemd/system/rudder-jetty.service; enabled; vendor preset: enabled)
     Active: failed (Result: signal) since Wed 2022-05-04 15:54:26 UTC; 30s ago
    Process: 19265 ExecStart=/opt/rudder/bin/rudder-jetty.sh start (code=exited, status=0/SUCCESS)
   Main PID: 19329 (code=killed, signal=KILL)
        CPU: 1min 50.807s

May 04 14:01:10 server systemd[1]: rudder-jetty.service: Failed with result 'exit-code'.
May 04 14:01:10 server systemd[1]: Stopped Jetty Web Application Server.
May 04 14:01:10 server systemd[1]: rudder-jetty.service: Consumed 40.332s CPU time.
May 04 14:01:10 server systemd[1]: Starting Jetty Web Application Server...
May 04 14:01:10 server rudder-jetty.sh[19265]: Setting umask to 0007
May 04 14:01:30 server rudder-jetty.sh[19265]: Starting Jetty: . . . . OK Wed May  4 14:01:30 UTC 2022
May 04 14:01:30 server systemd[1]: Started Jetty Web Application Server.
May 04 15:54:26 server systemd[1]: rudder-jetty.service: Main process exited, code=killed, status=9/KILL
May 04 15:54:26 server systemd[1]: rudder-jetty.service: Failed with result 'signal'.
May 04 15:54:26 server systemd[1]: rudder-jetty.service: Consumed 1min 50.807s CPU time.

running the start does work in this case though

There's clearly a difference of behaviour between 6.2 and 7.0, but I cannot reliably recreate a state where rudder-jetty is not starting

Actions

Copy link

Updated by Vincent MEMBRÉ almost 2 years ago

Target version changed from 7.0.3 to 7.0.4

Actions

Copy link

Updated by Félix DALLIDET almost 2 years ago

Related to Bug #21240: Rudder-jetty service can not be started added

Actions

Copy link

Updated by Vincent MEMBRÉ almost 2 years ago

Target version changed from 7.0.4 to 7.0.5

Actions

Copy link

Updated by Vincent MEMBRÉ almost 2 years ago

Target version changed from 7.0.5 to 7.0.6

Actions

Copy link

Updated by François ARMAND over 1 year ago

UX impact set to I hate Rudder for that
Regression set to No

Actions

Copy link

Updated by François ARMAND over 1 year ago

UX impact changed from I hate Rudder for that to It bothers me each time

Actions

Copy link

Updated by Vincent MEMBRÉ over 1 year ago

Target version changed from 7.0.6 to 7.0.7

Actions

Copy link

#10

Updated by Vincent MEMBRÉ over 1 year ago

Target version changed from 7.0.7 to 7.0.8

Actions

Copy link

#11

Updated by Vincent MEMBRÉ over 1 year ago

Target version changed from 7.0.8 to 1002

Actions

Copy link

#12

Updated by Vincent MEMBRÉ over 1 year ago

Target version changed from 1002 to 7.1.7

Actions

Copy link

#13

Updated by Vincent MEMBRÉ over 1 year ago

Target version changed from 7.1.7 to 7.1.8

Actions

Copy link

#14

Updated by Vincent MEMBRÉ over 1 year ago

Target version changed from 7.1.8 to 7.1.9

Actions

Copy link

#15

Updated by Vincent MEMBRÉ over 1 year ago

Target version changed from 7.1.9 to 7.1.10

Actions

Copy link

#16

Updated by Vincent MEMBRÉ about 1 year ago

Target version changed from 7.1.10 to 1016

Actions

Copy link

#17

Updated by Vincent MEMBRÉ about 1 year ago

Status changed from New to Rejected

We cannot reproduce and we don't have much details, I"m rejecting

Actions

Copy link

#18

Updated by Alexis Mousset about 1 year ago

Target version changed from 1016 to 7.2.5

Actions

Copy link

#19

Updated by Félix DALLIDET about 1 year ago

Status changed from Rejected to New
Target version changed from 7.2.5 to 7.2.6

Re opening the ticket as I had the same issue in 7.3.0. Jetty could not start, was not producing any log file under /var/log/rudder and the system journal was spamming:

Apr 17 11:56:00 server rudder-jetty.sh[29749]: tail: cannot open '/opt/rudder/etc/rudder-jetty-base/rudder-jetty.state' for reading: No such file or directory

Removing the pid file /var/rudder/run/rudder-jetty.pid did resolve the issue

Actions

Copy link

#20