Bug #14281
closed
Manage rudder webapp shutdown correctly in case of fatal exception at startup
Added by Félix DALLIDET almost 6 years ago.
Updated over 2 years ago.
Category:
System integration
Severity:
Major - prevents use of part of Rudder | no simple workaround
User visibility:
Operational - other Techniques | Rudder settings | Plugins
Description
Example of an error that we would like to catch:
Error at rudder restart liftActor:NoClassDefFoundError
Exception in thread "pool-7-thread-7" java.lang.NoClassDefFoundError: net/liftweb/actor/ActorLogger$
at net.liftweb.actor.SpecializedLiftActor$$anonfun$exceptionHandler$1.applyOrElse(LiftActor.scala:345)
at net.liftweb.actor.SpecializedLiftActor$$anonfun$exceptionHandler$1.applyOrElse(LiftActor.scala:344)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
at net.liftweb.actor.SpecializedLiftActor.proc2(LiftActor.scala:324)
at net.liftweb.actor.SpecializedLiftActor.$anonfun$processMailbox$1(LiftActor.scala:226)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at net.liftweb.actor.SpecializedLiftActor.around(LiftActor.scala:240)
at net.liftweb.actor.SpecializedLiftActor.around$(LiftActor.scala:239)
at com.normation.rudder.batch.AutomaticReportsCleaning$LADatabaseCleaner.around(AutomaticReportsCleaner.scala:311)
at net.liftweb.actor.SpecializedLiftActor.processMailbox(LiftActor.scala:226)
at net.liftweb.actor.SpecializedLiftActor.$anonfun$$bang$3(LiftActor.scala:189)
at net.liftweb.actor.LAScheduler$$anon$2$$anon$3.run(LiftActor.scala:76)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
This happens when old classes are not cleanly unloaded, most likely because jetty didn't stop correctly.
WORKAROUND: stop rudder-jetty ; check that jetty is stopped ; start rudder-jetty
Other example include "java.lang.OutOfMemoryError", "java.lang.StackOverflowError", ... In fact I think all java.errors.
Files
It happend when installing the auth-backend plugin which was already installed on the server.
The install did not seem to trigger any webapp restart.
- Description updated (diff)
Seen on 4.3.1 on SLES 11 today.
- Translation missing: en.field_tag_list set to Sponsored
- Target version changed from 5.0.7 to 4.3.10
- Severity set to Major - prevents use of part of Rudder | no simple workaround
- User visibility set to Operational - other Techniques | Rudder settings | Plugins
- Priority changed from 0 to 84
We also saw it during a normal reboot, so I'm setting the priority to a higher level. We need to at least find a way to stop the webapp, a NoClassDefFound is always a big problem.
- Description updated (diff)
- Category set to System integration
- Assignee set to François ARMAND
The problem with that exception is that it happens on a separed thread pool.
The general idea would be to have something looking at the log file and doing other things when that happen.
Something =
- rudder (but if rudder is broken, not good)
- jetty (but that means special handled code in jetty)
- something out of rudder/jetty (script in cron)
- Target version changed from 4.3.10 to 4.3.11
- Target version changed from 4.3.11 to 5.0.9
- Priority changed from 84 to 83
Trying to see if that it can be corrected in 5.0 at least.
- Status changed from New to In progress
- Description updated (diff)
- Related to Bug #2843: Rudder can fail to generate promises when Java is lacking memory added
- Status changed from In progress to Pending technical review
- Assignee changed from François ARMAND to Vincent MEMBRÉ
- Pull Request set to https://github.com/Normation/rudder/pull/2167
- Status changed from Pending technical review to Pending release
It works even better than expected: the shutdown is a clean one, see attachement.
- Subject changed from Error at rudder restart liftActor:NoClassDefFoundError to Manage rudder webapp shutdown correctly in case of fatal exception at startup
- Status changed from Pending release to Released
- Priority changed from 83 to 82
This bug has been fixed in Rudder 5.0.9 which was released today.
- Priority changed from 82 to 73
Also available in: Atom
PDF