Project

General

Profile

Bug #13014

StackOverflowError during policy generation in JavascriptEngine - debian 9.5 with jdk 1.8.0_181

Added by Marco Kirchhoff 5 months ago. Updated about 2 months ago.

Status:
Released
Priority:
N/A
Category:
Security
Target version:
Severity:
Critical - prevents main use of Rudder | no workaround | data loss | security
User visibility:
Operational - other Techniques | Technique editor | Rudder settings
Effort required:
Medium
Priority:
57

Description

After upgrading from 4.2.7 to 4.3.3 policy generation fails with a StackOverflowError.

I determined that the exception is caused by "com.normation.rudder.services.policies.JsEngine$SandboxSecurityManager.checkPermission" calling itself recursively while trying to check for the permission to load a missing class (which appears to be "FilePermission").

stacktrace.txt (63.1 KB) stacktrace.txt Marco Kirchhoff, 2018-07-20 13:36

Related issues

Related to Rudder - Bug #12448: Failed generation with "Could not initialize class javax.crypto.JceSecurity"Released

Associated revisions

Revision 33e18852 (diff)
Added by François ARMAND 2 months ago

Fixes #13014: StackOverflowError during policy generation in JavascriptEngine - debian 9.5 with jdk 1.8.0_181

History

#1 Updated by François ARMAND 3 months ago

  • Category set to Security
  • Assignee set to François ARMAND
  • Target version set to 4.3.5
  • User visibility set to Operational - other Techniques | Technique editor | Rudder settings
  • Priority changed from 0 to 50

Sorry for the long delay.

I'm not able to reproduce that problem, but the jvm security manager is a quite sensitive beast. From what I see, it may be that a (new ?) required permission is missing, and so the check for the permission itself trigger a permission check, which loops.

Would you mind sharing:

- your exact OS,
- your exact JVM flavor and version ?

We corrected some problems with Java 9/10 support (https://www.rudder-project.org/redmine/issues/12548) but the manifestation of the problem were not the same exactly.

Sorry for that.

#3 Updated by Marco Kirchhoff 2 months ago

Thanks for getting back to me.
Here's the information you asked for.

# uname -a
Linux rudder 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08) x86_64 GNU/Linux

# lsb_release -a
Distributor ID: Debian
Description:    Debian GNU/Linux 9.5 (stretch)
Release:        9.5
Codename:       stretch

# dpkg --list | grep -e rudder -e ncf
ii  ncf                              4.3.4-stretch0                 all          ncf - CFEngine framework
ii  ncf-api-virtualenv               4.3.4-stretch0                 all          ncf - CFEngine framework
ii  rudder-agent                     4.3.4-stretch0                 amd64        Configuration management and audit tool - agent
ii  rudder-inventory-endpoint        4.3.4-stretch0                 all          Configuration management and audit tool - service to receive inventory data
ii  rudder-inventory-ldap            4.3.4-stretch0                 amd64        Configuration management and audit tool - OpenLDAP
ii  rudder-jetty                     4.3.4-stretch0                 all          Configuration management and audit tool - Jetty application server
ii  rudder-reports                   4.3.4-stretch0                 all          Configuration management and audit tool - reports database
ii  rudder-server-relay              4.3.4-stretch0                 amd64        Configuration management and audit tool - Server relay package
ii  rudder-server-root               4.3.4-stretch0                 all          Configuration management and audit tool - root server base package
ii  rudder-techniques                4.3.4-stretch0                 all          Configuration management and audit tool - techniques
ii  rudder-webapp                    4.3.4-stretch0                 all          Configuration management and audit tool - webapp

# java -version
openjdk version "1.8.0_181" 
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-1~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

#4 Updated by François ARMAND 2 months ago

  • Related to Bug #12448: Failed generation with "Could not initialize class javax.crypto.JceSecurity" added

#5 Updated by François ARMAND 2 months ago

  • Subject changed from StackOverflowError during policy generation in JavascriptEngine to StackOverflowError during policy generation in JavascriptEngine - debian 9.5 with jdk 1.8.0_181
  • Severity changed from Major - prevents use of part of Rudder | no simple workaround to Critical - prevents main use of Rudder | no workaround | data loss | security
  • Effort required set to Medium
  • Priority changed from 50 to 57

1.8.0_172 is pretty recent, but I just tested with it for the same release of Rudder, and I was not able to reproduce the problem. I tested both oracle JDK and open JDK.

Finally, I tested with the exact same version of Debian / Open JDK, and I got the error. The strangest think is that even with Oracle JDK which I installed afterward, I get the error.
So there's something very fishy going on here. I fear it might be quite hard to debug :/

A workaround it to not use JS field in directive or disable these directive until we find a correction - but depending on your use case, it may be as good as not using Rudder at all.

#6 Updated by François ARMAND 2 months ago

Some more information on that subject:

- I reproduced it on centos 7, some JVM version.
- it seems to be here:

        at com.normation.rudder.services.policies.JsEngine$SandboxSecurityManager.checkPermission(JavascriptEngine.scala:681)
        at sun.misc.URLClassPath.check(URLClassPath.java:642)
        at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1268)
        at sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:1249)
        at sun.misc.URLClassPath.findResource(URLClassPath.java:215)
        at java.net.URLClassLoader$2.run(URLClassLoader.java:569)
        at java.net.URLClassLoader$2.run(URLClassLoader.java:567)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findResource(URLClassLoader.java:566)
        at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:546)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at com.normation.rudder.services.policies.JsEngine$SandboxSecurityManager.checkPermission(JavascriptEngine.scala:681)
        at sun.misc.URLClassPath.check(URLClassPath.java:642)

Which correspond to code:

676    override def checkPermission(permission: Permission): Unit = {
677      if(isSandboxed.get) {
678        // there some perms which are hard to specify in that file
679        permission match {
680          // jar and classes
681          case x: FilePermission if( x.getActions == "read" && (x.getName.contains(".class") || x.getName.endsWith(".jar")  || x.getName.endsWith(".so") ) ) => // ok
682          // configuration files
683          case x: FilePermission if( x.getActions == "read" && (x.getName.endsWith(".cfg") || x.getName.endsWith(".properties") ) )  => // ok
684          // cert files and config
685   ....

So it is the check itself which cause a check :/

#7 Updated by François ARMAND 2 months ago

So, more information: the problem is due to security check that are initiated during other security checks. Typically, trying to execute:

             ....
681          case x: FilePermission if( x.getActions == "read" && (x.getName.contains(".class") || x.getName.endsWith(".jar")  || x.getName.endsWith(".so") ) ) => // ok

needs to load FilePermission class file, abd that need to resolve URL (more permission), and then load a file from fs (more perm).
It seems that something changed in later JVM 1.8 JDK releases about what classes are loaded when. And that results in infinite loops due to impossibility to linearize the graphe of dependance check.

I thought about two resolutions, a correct one and a hacky one. Unfortunatly, I'm not able to implement the correct one for now, and I'm not sure I want to amass the quantity of java priviledges & security to be able to do it.
So, the correct solution is to be able to make the check permission implementation logic not subject to the permission check. This would be correct because the code for the check is statically knows and immutable, so we can assess its security. But I don't know of a way to escape the current security contexte (and it seems logical, since it's kind of the goal of the security context...)
S, the hacky one is to preload relevant classes out of the security context so that we don't have to do the resolution check later on. Of course, this is extremelly fragile.

#8 Updated by François ARMAND 2 months ago

  • Status changed from New to In progress

#9 Updated by François ARMAND 2 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from François ARMAND to Vincent MEMBRÉ
  • Pull Request set to https://github.com/Normation/rudder/pull/2048

#10 Updated by Normation Quality Assistant 2 months ago

  • Assignee changed from Vincent MEMBRÉ to François ARMAND

#11 Updated by François ARMAND 2 months ago

  • Status changed from Pending technical review to Pending release

#12 Updated by Vincent MEMBRÉ about 2 months ago

  • Status changed from Pending release to Released
This bug has been fixed in Rudder 4.3.5 and 5.0.1 which were released today.
Changelog
Changelog

Also available in: Atom PDF