Bug #14281
Manage rudder webapp shutdown correctly in case of fatal exception at startup
Description
Example of an error that we would like to catch:
Error at rudder restart liftActor:NoClassDefFoundError
Exception in thread "pool-7-thread-7" java.lang.NoClassDefFoundError: net/liftweb/actor/ActorLogger$ at net.liftweb.actor.SpecializedLiftActor$$anonfun$exceptionHandler$1.applyOrElse(LiftActor.scala:345) at net.liftweb.actor.SpecializedLiftActor$$anonfun$exceptionHandler$1.applyOrElse(LiftActor.scala:344) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34) at net.liftweb.actor.SpecializedLiftActor.proc2(LiftActor.scala:324) at net.liftweb.actor.SpecializedLiftActor.$anonfun$processMailbox$1(LiftActor.scala:226) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) at net.liftweb.actor.SpecializedLiftActor.around(LiftActor.scala:240) at net.liftweb.actor.SpecializedLiftActor.around$(LiftActor.scala:239) at com.normation.rudder.batch.AutomaticReportsCleaning$LADatabaseCleaner.around(AutomaticReportsCleaner.scala:311) at net.liftweb.actor.SpecializedLiftActor.processMailbox(LiftActor.scala:226) at net.liftweb.actor.SpecializedLiftActor.$anonfun$$bang$3(LiftActor.scala:189) at net.liftweb.actor.LAScheduler$$anon$2$$anon$3.run(LiftActor.scala:76) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
This happens when old classes are not cleanly unloaded, most likely because jetty didn't stop correctly.
WORKAROUND: stop rudder-jetty ; check that jetty is stopped ; start rudder-jetty
Other example include "java.lang.OutOfMemoryError", "java.lang.StackOverflowError", ... In fact I think all java.errors.
Files
Subtasks
Related issues
Updated by Félix DALLIDET over 1 year ago
It happend when installing the auth-backend plugin which was already installed on the server.
The install did not seem to trigger any webapp restart.
Updated by
Updated by François ARMAND over 1 year ago
- Tags set to Sponsored
- Target version changed from 5.0.7 to 4.3.10
- Severity set to Major - prevents use of part of Rudder | no simple workaround
- User visibility set to Operational - other Techniques | Rudder settings | Plugins
- Priority changed from 0 to 84
We also saw it during a normal reboot, so I'm setting the priority to a higher level. We need to at least find a way to stop the webapp, a NoClassDefFound is always a big problem.
Updated by François ARMAND over 1 year ago
- Description updated (diff)
- Category set to System integration
- Assignee set to François ARMAND
The problem with that exception is that it happens on a separed thread pool.
The general idea would be to have something looking at the log file and doing other things when that happen.
Something =
- rudder (but if rudder is broken, not good)
- jetty (but that means special handled code in jetty)
- something out of rudder/jetty (script in cron)
Updated by François ARMAND over 1 year ago
- Target version changed from 4.3.11 to 5.0.9
- Priority changed from 84 to 83
Trying to see if that it can be corrected in 5.0 at least.
Updated by François ARMAND over 1 year ago
- Description updated (diff)
I'm trying an implementation with `Thread.UncaughtExceptionHandler`: https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.UncaughtExceptionHandler.html
Updated by François ARMAND over 1 year ago
- Related to Bug #2843: Rudder can fail to generate promises when Java is lacking memory added
Updated by François ARMAND over 1 year ago
- Status changed from In progress to Pending technical review
- Assignee changed from François ARMAND to Vincent MEMBRÉ
- Pull Request set to https://github.com/Normation/rudder/pull/2167
Updated by François ARMAND over 1 year ago
- Status changed from Pending technical review to Pending release
Applied in changeset rudder|a6c78ff6e1f29163bc6301ca7f892628769f785a.
Updated by François ARMAND over 1 year ago
It works even better than expected: the shutdown is a clean one, see attachement.
Updated by Vincent MEMBRÉ over 1 year ago
- Subject changed from Error at rudder restart liftActor:NoClassDefFoundError to Manage rudder webapp shutdown correctly in case of fatal exception at startup
Updated by Vincent MEMBRÉ over 1 year ago
- Status changed from Pending release to Released
- Priority changed from 83 to 82
This bug has been fixed in Rudder 5.0.9 which was released today.