Bug #5653: rudder-jetty init script may fail on success - Rudder - Issue Tracker

Actions

Copy link

Bug #5653

closed

rudder-jetty init script may fail on success

Added by Dennis Cabooter about 10 years ago. Updated almost 9 years ago.

Status:

Released

Priority:

Assignee:

Benoît PECCATTE

Category:

System integration

Target version:

2.10.13

Pull Request:

https://github.com/Normation/rudder-p...

Severity:

UX impact:

User visibility:

Effort required:

Priority:

Name check:

Fix check:

Regression:

Description

The init script of rudder-jetty is a bit fishy. It always reports failure:

Starting Jetty: . . . . . . . . . . . . . . FAILED Fri Oct 17 11:22:33 CEST 2014

But.. it is actually running:

# /etc/init.d/rudder-jetty status | tail -1
Jetty running pid=4469

Also it takes a long time before I'm able to login to the web interface.

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Jonathan CLARKE about 10 years ago

It looks like the web application take too long to start, and the init script times out waiting for it.

This shouldn't happen of course.

One benefit you will get soon is that Rudder 3.0 removed a rather slow migration script at startup so this probably won't happen in that version.

By any chance could you use a snapshot VM copy to upgrade to current 2.12 nightly version and see if it is still as slow?

Actions

Copy link

Updated by Dennis Cabooter about 10 years ago

Is it not possible to change something in 2.11.3 to test? Also /etc/init.d/jetty still exists and gives no errors.

Actions

Copy link

Updated by Dennis Cabooter about 10 years ago

After kill -9 jetty, the script now works as expected.

Actions

Copy link

Updated by Dennis Cabooter about 10 years ago

On 2.11.4 that is...

Actions

Copy link

Updated by Matthieu CERDA about 10 years ago

Category set to System integration
Status changed from New to Discussion
Assignee set to Jonathan CLARKE
Priority changed from N/A to 2
Target version set to 2.11.5

This init script is just an horror to look at, I have a working correction on a test machine but it involves rewrites at multiple parts of the script.

I think improving this script by using patches applied on the example jetty.sh is not really a good way to go:

We do not update Jetty that often (once every several months)
Patches are applying more and more roughly (fuzzes/offsets everywhere)
The origin script is just riddled with inconsistencies, flawed assumptions and general lack of shell script best practices awareness

I'd suggest adding a patch here to fix this problem on 2.10 / 2.11, and just using our own version of the script on 3.0+ onwards copied from jetty.sh and stored in SOURCES, and just diff the changes between the two releases jetty.sh when we update Jetty to eventually pick the improvements if necessary.

Looks like a good idea ?

Actions

Copy link

Updated by Matthieu CERDA about 10 years ago

Status changed from Discussion to Pending technical review
Assignee changed from Jonathan CLARKE to Matthieu CERDA
% Done changed from 0 to 100
Pull Request set to https://github.com/Normation/rudder-packages/pull/527

PR has been created: https://github.com/Normation/rudder-packages/pull/527

I chose to adapt the existing "Jetty is running" logic, to be more resilent to "already started" conditions that often resulted in a "FAILED", by relying a bit more on the jetty-state file that specifies if Jetty is running or not :)

Actions

Copy link

Updated by Matthieu CERDA almost 10 years ago

The important code bit is:

if [ $(ps aux|grep -E "java .* /opt/rudder/jetty7/start.jar"|grep -v grep|wc -l) -ne 0 ]
then
  if [ -e ${JETTY_STATE} ] && grep -q "STARTING" ${JETTY_STATE} || grep -q "STARTED" ${JETTY_STATE}
  then
    echo "already running." 
    exit 0
  fi
fi

I check if a Jetty process seems running in the process list, and if it looks linked to the jetty status file (STARTING or STARTED), and just leave it alone in this case.

If those conditions are not fulfilled, and only in this case, remove the state file and try to run Jetty just like before.

I also cleanup the state file properly when stopping Jetty gracefully.

Actions

Copy link

Updated by Matthieu CERDA almost 10 years ago

Assignee changed from Matthieu CERDA to Jonathan CLARKE

Actions

Copy link

Updated by Vincent MEMBRÉ almost 10 years ago

Target version changed from 2.11.5 to 2.11.6

Actions

Copy link

#10

Updated by Vincent MEMBRÉ almost 10 years ago

Target version changed from 2.11.6 to 2.11.7

Actions

Copy link

#11

Updated by Vincent MEMBRÉ almost 10 years ago

Target version changed from 2.11.7 to 2.11.8

Actions

Copy link

#12

Updated by Dennis Cabooter almost 10 years ago

This is still true on Rudder 3.0:

# /etc/init.d/rudder-jetty restart
Stopping Jetty: OK
Setting umask to 0007
Starting Jetty: . . . . . . . . . . . . . . FAILED Tue Feb 17 14:06:08 CET 2015

# /etc/init.d/rudder-jetty check | tail -1
Jetty running pid=18190

Actions

Copy link

#13

Updated by François ARMAND almost 10 years ago

Assignee changed from Jonathan CLARKE to Benoît PECCATTE

Benoit, could you take a look to that one ?

From my point of view, Matthieu analysis seems good. But well, I would prefer more experienced eyes :)

Actions

Copy link

#14

Updated by Benoît PECCATTE almost 10 years ago

Status changed from Pending technical review to Discussion
Assignee changed from Benoît PECCATTE to Matthieu CERDA

Actions

Copy link

#15

Updated by Vincent MEMBRÉ over 9 years ago

Target version changed from 2.11.8 to 2.11.9

Actions

Copy link

#16

Updated by Vincent MEMBRÉ over 9 years ago

Target version changed from 2.11.9 to 2.11.10

Actions

Copy link

#17

Updated by Matthieu CERDA over 9 years ago

Status changed from Discussion to Pending technical review
Assignee changed from Matthieu CERDA to Benoît PECCATTE
Pull Request changed from https://github.com/Normation/rudder-packages/pull/527 to https://github.com/Normation/rudder-packages/pull/633
Reproduced set to No