Rudder

It is a problem occuring in the underlying LDAP lib (unbound).

So the connection, for some reason, can not be done because the socket is closed, and so the connection never reach the LDAP server, which is why OpenLDAP log are void at that point of time.

It happens when we try to aquire a connection in NodeConfigurationCacheRepository.scala #l445:

  def save(caches: Set[NodeConfigurationHash]): Box[Set[NodeId]] = {
    val updatedIds = caches.map(_.id)
    for {
      ldap          <- ldapCon  // <========= HERE
      existingEntry <- ldap.get(rudderDit.NODE_CONFIGS.dn)

This code is totally standard, we are using it again and again everywhere. If it was repeatable, I would say that something went wrong with the LDAP connection pool before / during the deployment, and that for some reason, the fault happen around the same time (based on number of open con?). Or perhaps something during the deployment, just before that action, corrupt the connection pool -perhaps a bad data.

I fear that I won't be able to make more assumption.

So, some questions:

• does restarting Rudder solved the problem?
• cleaning cache ? (but I really believe that the problem is not linked to NodeHash, but something before)

Next step is trying to set unboundId at debug or trace level, and see what the logs are telling.

One of our user experience the same problem on a Rudder 4.1.7.

The server is quite loaded and has > 10k nodes.

I'm able to reproduce by faking a huge Node Configuration entry by changing that method in NodeConfigurationCacheRepository:

  private[this] def toLdap(nodeConfigs: Set[NodeConfigurationHash]): LDAPEntry = {
    val cc = if(nodeConfigs.nonEmpty) {
      val c = nodeConfigs.head
       for(i <- 0 to 10000) yield c.copy(
         id = NodeId(c.id.value+i.toString),
          policyHash = c.policyHash ++ (for(j <- 1 to 100) yield PolicyHash( Cf3PolicyDraftId(RuleId("test--dir-" + j), DirectiveId("test-rule-"+j)) , j))
        )
    } else {
      Set[NodeConfigurationHash]()
    }
    val caches = (cc ++ nodeConfigs).map{ x => write(x) }
    val entry = rudderDit.NODE_CONFIGS.model
    entry +=! (A_NODE_CONFIG, caches.toSeq:_*)
    entry
  }

I was able to minify it with that program:

object NodesConfigurationErrorMain {

  val rwLdap =
    new RWPooledSimpleAuthConnectionProvider(
      host = "localhost",
      port = 1389,
      authDn = "cn=Manager,cn=rudder-configuration",
      authPw = "secret",
      poolSize = 2
  )

  def str(size: Int) = new String( Array.fill[Char](size)('x'))

  def main(args: Array[String]): Unit = {
    val dn = new DN("cn=Nodes Configuration,ou=Rudder,cn=rudder-configuration")
    val entry = LDAPEntry(dn)

    entry += new Attribute("objectClass", "top", "nodeConfigurations")
    entry += new Attribute("cn", "Nodes Configuration")

//here, with 454, we get:
//Failure(Can not save (add) 'cn=Nodes Configuration,ou=Rudder,cn=rudder-configuration': An error occurred while attempting to send the LDAP message to server localhost:1389:  SocketException(message='Socket closed', trace='socketWrite(SocketOutputStream.java:118) / write(SocketOutputStream.java:155) / write(BufferedOutputStream.java:122) / write(ByteStringBuffer.java:1664) ....
    val values = for(i <- 0 until 167) yield i + "-"+str(100000) ++ str(453)
    entry += new Attribute("nodeConfig", values:_*)
    println(for {
      ldap <- rwLdap
      _ <- ldap.delete(dn)
      _ <- ldap.save(entry)
    } yield {
      "done" 
    })

  }
}

To be clearer: in the previous code snippets, it's really the total sum size of the entry which is the problem. I can decrease the size of each value and add more of them, or increase it and add even less.

It is more likely because somewhere, we hit an int overflow, likely in the ASN.1 encoding or the actual write of the code into the socket.

Bug opened in ldap sdk: https://github.com/pingidentity/ldapsdk/issues/47

Actually, the problem is just that OpenLDAP limit the size of request to avoid DOS by people sending huge, useless request.

The correction is to increase "sockbuf_max_incoming_auth" in slapd.conf.

# Specify the maximum incoming LDAP PDU size for authenticated sessions. The default is 4194303.
# This is too little for us, we have big entries (like Nodes Configuration)
sockbuf_max_incoming_auth 1000000000

So, to So, to sum up that problem:

- OpenLDAP has (like all LDAP server) a protection agains denial of service attack that limit the size of request it processes. This is to avoid having an attacker sending big, useless request with hard to decode ASN.1, and make the server burn CPU on them.
- that protection is by "PDU" (protocol data unit), so it's not a diret mapping into request size
- but it explains why we sometime get LDAP error "socket close" - when too much data comes, openldap just close the socket to protect itself.
- you get the following message with OpenLDAP log level "conns trace" (yeah, that's why we never saw it - the log level needed is extremelly deep)

Jul 23 10:03:19 server rudder-slapd[31057]: connection_get(10): got connid=1002
Jul 23 10:03:19 server rudder-slapd[31057]: connection_read(10): checking for input on id=1002
Jul 23 10:03:19 server rudder-slapd[31057]: ber_get_next on fd 10 failed errno=34 (Numerical result out of range)
Jul 23 10:03:19 server rudder-slapd[31057]: connection_read(10): input error=-2 id=1002, closing.
Jul 23 10:03:19 server rudder-slapd[31057]: connection_closing: readying conn=1002 sd=10 for close
Jul 23 10:03:19 server rudder-slapd[31057]: connection_close: conn=1002 sd=10