Project

General

Profile

Bug #14281

Manage rudder webapp shutdown correctly in case of fatal exception at startup

Added by Félix DALLIDET 5 months ago. Updated 2 months ago.

Status:
Released
Priority:
N/A
Assignee:
Vincent MEMBRÉ
Category:
System integration
Target version:
5.0.9
Pull Request:
https://github.com/Normation/rudder/p...
Severity:
Major - prevents use of part of Rudder | no simple workaround
User visibility:
Operational - other Techniques | Rudder settings | Plugins
Effort required:
Priority:
82
Tags:
Sponsored

Description

Example of an error that we would like to catch:
Error at rudder restart liftActor:NoClassDefFoundError

Exception in thread "pool-7-thread-7" java.lang.NoClassDefFoundError: net/liftweb/actor/ActorLogger$
  at net.liftweb.actor.SpecializedLiftActor$$anonfun$exceptionHandler$1.applyOrElse(LiftActor.scala:345)
  at net.liftweb.actor.SpecializedLiftActor$$anonfun$exceptionHandler$1.applyOrElse(LiftActor.scala:344)
  at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
  at net.liftweb.actor.SpecializedLiftActor.proc2(LiftActor.scala:324)
  at net.liftweb.actor.SpecializedLiftActor.$anonfun$processMailbox$1(LiftActor.scala:226)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
  at net.liftweb.actor.SpecializedLiftActor.around(LiftActor.scala:240)
  at net.liftweb.actor.SpecializedLiftActor.around$(LiftActor.scala:239)
  at com.normation.rudder.batch.AutomaticReportsCleaning$LADatabaseCleaner.around(AutomaticReportsCleaner.scala:311)
  at net.liftweb.actor.SpecializedLiftActor.processMailbox(LiftActor.scala:226)
  at net.liftweb.actor.SpecializedLiftActor.$anonfun$$bang$3(LiftActor.scala:189)
  at net.liftweb.actor.LAScheduler$$anon$2$$anon$3.run(LiftActor.scala:76)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)

This happens when old classes are not cleanly unloaded, most likely because jetty didn't stop correctly.

WORKAROUND: stop rudder-jetty ; check that jetty is stopped ; start rudder-jetty

Other example include "java.lang.OutOfMemoryError", "java.lang.StackOverflowError", ... In fact I think all java.errors.

Files

2019_03_30.stderrout.log.132711187 (40.4 KB) 2019_03_30.stderrout.log.132711187 François ARMAND, 2019-03-30 14:45

Subtasks

Bug #14568: Exception at Rudder start: com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'rudder.jvm'ReleasedNicolas CHARLESActions

Related issues

Related to Rudder - Bug #2843: Rudder can fail to generate promises when Java is lacking memoryRejectedActions

Associated revisions

Revision a6c78ff6 (diff)
Added by François ARMAND 3 months ago

Fixes #14281: Error at rudder restart liftActor:NoClassDefFoundError

History

#1

Updated by Félix DALLIDET 5 months ago

It happend when installing the auth-backend plugin which was already installed on the server.
The install did not seem to trigger any webapp restart.

#2

Updated by Alexis MOUSSET 4 months ago

  • Description updated (diff)
#3

Updated by Alexis MOUSSET 4 months ago

Seen on 4.3.1 on SLES 11 today.

#4

Updated by François ARMAND 4 months ago

  • Tags set to Sponsored
  • Target version changed from 5.0.7 to 4.3.10
  • Severity set to Major - prevents use of part of Rudder | no simple workaround
  • User visibility set to Operational - other Techniques | Rudder settings | Plugins
  • Priority changed from 0 to 84

We also saw it during a normal reboot, so I'm setting the priority to a higher level. We need to at least find a way to stop the webapp, a NoClassDefFound is always a big problem.

#5

Updated by François ARMAND 4 months ago

  • Description updated (diff)
  • Category set to System integration
  • Assignee set to François ARMAND

The problem with that exception is that it happens on a separed thread pool.

The general idea would be to have something looking at the log file and doing other things when that happen.
Something =
- rudder (but if rudder is broken, not good)
- jetty (but that means special handled code in jetty)
- something out of rudder/jetty (script in cron)

#6

Updated by François ARMAND 4 months ago

  • Target version changed from 4.3.10 to 4.3.11
#7

Updated by François ARMAND 3 months ago

  • Target version changed from 4.3.11 to 5.0.9
  • Priority changed from 84 to 83

Trying to see if that it can be corrected in 5.0 at least.

#8

Updated by François ARMAND 3 months ago

  • Status changed from New to In progress
#9

Updated by François ARMAND 3 months ago

  • Description updated (diff)

I'm trying an implementation with `Thread.UncaughtExceptionHandler`: https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.UncaughtExceptionHandler.html

#10

Updated by François ARMAND 3 months ago

  • Related to Bug #2843: Rudder can fail to generate promises when Java is lacking memory added
#11

Updated by François ARMAND 3 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from François ARMAND to Vincent MEMBRÉ
  • Pull Request set to https://github.com/Normation/rudder/pull/2167
#12

Updated by François ARMAND 3 months ago

  • Status changed from Pending technical review to Pending release
#13

Updated by François ARMAND 3 months ago

It works even better than expected: the shutdown is a clean one, see attachement.

#14

Updated by Vincent MEMBRÉ 2 months ago

  • Subject changed from Error at rudder restart liftActor:NoClassDefFoundError to Manage rudder webapp shutdown correctly in case of fatal exception at startup
#15

Updated by Vincent MEMBRÉ 2 months ago

  • Status changed from Pending release to Released
  • Priority changed from 83 to 82

This bug has been fixed in Rudder 5.0.9 which was released today.

Also available in: Atom PDF