Project

General

Profile

Actions

Bug #5653

closed

rudder-jetty init script may fail on success

Added by Dennis Cabooter over 6 years ago. Updated over 5 years ago.

Status:
Released
Priority:
2
Category:
System integration
Target version:
Severity:
User visibility:
Effort required:
Priority:

Description

The init script of rudder-jetty is a bit fishy. It always reports failure:

Starting Jetty: . . . . . . . . . . . . . . FAILED Fri Oct 17 11:22:33 CEST 2014

But.. it is actually running:

# /etc/init.d/rudder-jetty status | tail -1
Jetty running pid=4469

Also it takes a long time before I'm able to login to the web interface.


Related issues

Related to Rudder - Bug #5768: Bad "last seen" time when the timezone given by java does not matches the system oneNewActions
Actions #1

Updated by Jonathan CLARKE over 6 years ago

It looks like the web application take too long to start, and the init script times out waiting for it.

This shouldn't happen of course.

One benefit you will get soon is that Rudder 3.0 removed a rather slow migration script at startup so this probably won't happen in that version.

By any chance could you use a snapshot VM copy to upgrade to current 2.12 nightly version and see if it is still as slow?

Actions #2

Updated by Dennis Cabooter over 6 years ago

Is it not possible to change something in 2.11.3 to test? Also /etc/init.d/jetty still exists and gives no errors.

Actions #3

Updated by Dennis Cabooter over 6 years ago

After kill -9 jetty, the script now works as expected.

Actions #4

Updated by Dennis Cabooter over 6 years ago

On 2.11.4 that is...

Actions #5

Updated by Matthieu CERDA over 6 years ago

  • Category set to System integration
  • Status changed from New to Discussion
  • Assignee set to Jonathan CLARKE
  • Priority changed from N/A to 2
  • Target version set to 2.11.5

This init script is just an horror to look at, I have a working correction on a test machine but it involves rewrites at multiple parts of the script.

I think improving this script by using patches applied on the example jetty.sh is not really a good way to go:
  • We do not update Jetty that often (once every several months)
  • Patches are applying more and more roughly (fuzzes/offsets everywhere)
  • The origin script is just riddled with inconsistencies, flawed assumptions and general lack of shell script best practices awareness

I'd suggest adding a patch here to fix this problem on 2.10 / 2.11, and just using our own version of the script on 3.0+ onwards copied from jetty.sh and stored in SOURCES, and just diff the changes between the two releases jetty.sh when we update Jetty to eventually pick the improvements if necessary.

Looks like a good idea ?

Actions #6

Updated by Matthieu CERDA over 6 years ago

  • Status changed from Discussion to Pending technical review
  • Assignee changed from Jonathan CLARKE to Matthieu CERDA
  • % Done changed from 0 to 100
  • Pull Request set to https://github.com/Normation/rudder-packages/pull/527

PR has been created: https://github.com/Normation/rudder-packages/pull/527

I chose to adapt the existing "Jetty is running" logic, to be more resilent to "already started" conditions that often resulted in a "FAILED", by relying a bit more on the jetty-state file that specifies if Jetty is running or not :)

Actions #7

Updated by Matthieu CERDA over 6 years ago

The important code bit is:

if [ $(ps aux|grep -E "java .* /opt/rudder/jetty7/start.jar"|grep -v grep|wc -l) -ne 0 ]
then
  if [ -e ${JETTY_STATE} ] && grep -q "STARTING" ${JETTY_STATE} || grep -q "STARTED" ${JETTY_STATE}
  then
    echo "already running." 
    exit 0
  fi
fi

I check if a Jetty process seems running in the process list, and if it looks linked to the jetty status file (STARTING or STARTED), and just leave it alone in this case.

If those conditions are not fulfilled, and only in this case, remove the state file and try to run Jetty just like before.

I also cleanup the state file properly when stopping Jetty gracefully.

Actions #8

Updated by Matthieu CERDA over 6 years ago

  • Assignee changed from Matthieu CERDA to Jonathan CLARKE
Actions #9

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 2.11.5 to 2.11.6
Actions #10

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 2.11.6 to 2.11.7
Actions #11

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 2.11.7 to 2.11.8
Actions #12

Updated by Dennis Cabooter about 6 years ago

This is still true on Rudder 3.0:

# /etc/init.d/rudder-jetty restart
Stopping Jetty: OK
Setting umask to 0007
Starting Jetty: . . . . . . . . . . . . . . FAILED Tue Feb 17 14:06:08 CET 2015
# /etc/init.d/rudder-jetty check | tail -1
Jetty running pid=18190
Actions #13

Updated by François ARMAND about 6 years ago

  • Assignee changed from Jonathan CLARKE to Benoît PECCATTE

Benoit, could you take a look to that one ?

From my point of view, Matthieu analysis seems good. But well, I would prefer more experienced eyes :)

Actions #14

Updated by Benoît PECCATTE about 6 years ago

  • Status changed from Pending technical review to Discussion
  • Assignee changed from Benoît PECCATTE to Matthieu CERDA
Actions #15

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 2.11.8 to 2.11.9
Actions #16

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 2.11.9 to 2.11.10
Actions #17

Updated by Matthieu CERDA about 6 years ago

  • Status changed from Discussion to Pending technical review
  • Assignee changed from Matthieu CERDA to Benoît PECCATTE
  • Pull Request changed from https://github.com/Normation/rudder-packages/pull/527 to https://github.com/Normation/rudder-packages/pull/633
  • Reproduced set to No
Actions #18

Updated by Matthieu CERDA about 6 years ago

  • Status changed from Pending technical review to Pending release

Applied in changeset rudder-packages|commit:be6d258649a8a08cc2008d4333f5372562d03526.

Actions #19

Updated by Benoît PECCATTE about 6 years ago

Applied in changeset rudder-packages|commit:22f38c0a3c3a7fbd44d9c13e5f761dd0a5dec8ea.

Actions #20

Updated by Vincent MEMBRÉ about 6 years ago

  • Subject changed from Fishy init script rudder-jetty to rudder-jetty init script may fail on success
Actions #21

Updated by Vincent MEMBRÉ about 6 years ago

This bug has been fixed in Rudder 2.11.10 and 3.0.4 which were released today.

Actions #22

Updated by Vincent MEMBRÉ about 6 years ago

  • Status changed from Pending release to Released
Actions #23

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 2.11.10 to 2.10.13
Actions #24

Updated by Vincent MEMBRÉ over 5 years ago

This was fixed in 2.10.13 and not 2.11.10 as previously stated

Actions

Also available in: Atom PDF