Bug #6780
closedNode not included in dynamic group due to openldap bug with modrdn not showing node children
Description
Some queries are sometimes (seems only on Centos/RedHat) not returning the proper list of nodes
The cause is the idlcache of openldap, which store the cache of ONE and SUB queries for entries. However, if we move an entry (with modrdn) with subchildren, it fails to properly update the subchildren in the cache - hene no result
Deactivating the idlcache, or restarting slapd circumvent the issue
A patch was proposed by Jon (attached), and a ticket opened at on OpenLDAP bug tracker ( http://www.openldap.org/its/index.cgi/Incoming?id=8378 )
Below, old ticket description
Some times a week The Rudder web interface stops working properly. Searching times out and the pie charts on the dashboard don't appear. There is no error in slapd.log, but there is an indication that LDAP is the culprit in the webapp logs:
[2015-06-22 14:59:21] ERROR com.normation.ldap.sdk.ROPooledSimpleAuthConnectionProvider - Can't execute LDAP request com.unboundid.ldap.sdk.LDAPSearchException: A client-side timeout was encountered while waiting 300000ms for a response to search request with message ID 3, base DN 'cn=rudder-configuration', scope SUB, and filter '(&(|(objectClass=rudderNode)(&(objectClass=node)(entryDN:dnOneLevelMatch:=ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration)))(modifyTimestamp>=20150622100542.001Z))' from server localhost:389.
As Nicolas Charles suggested on IRC, I stopped rudder-jetty and rudder-slapd. However, rudder-jetty is started automagically within 5 minutes and i had to forcestop slapd. Then i reindexed LDAP and started the services. The problem didn't go away unfortunatly.
Files
Updated by Nicolas CHARLES about 9 years ago
- Assignee changed from Nicolas CHARLES to François ARMAND
i'm assigning this issue to Francois, as he's more literate in ldap than I am
Updated by Nicolas CHARLES about 9 years ago
- File slapd.conf slapd.conf added
- File DB_CONFIG DB_CONFIG added
- File networkInterface networkInterface added
I'm having a probably related issue: on a VM, on Centos6, when the server has been used quite a lot, the groups don't get updated when accepting a node.
This is not related to the webapp (i connected from a remote webapp to double check this point). It simply seems that the ldap is not taking into account new entries in complex search (with regex at least)
I lost the webapp trace from ldap, but I'm searching from Linux node, with Centos OS, and IP Adress matching regex 192.168.42.1[3|4].
Accepting a node with ip 192.168.42.13 didn't update the group, and it shows in the webapp trace (it simply didn't came find the network interface 192.168.42.13
Rebooting the server solves the issue, as well as reindexing the ldap.
Some info:
[root@server openldap-data]# ls -al total 108624 drwxr-xr-x 2 root root 4096 30 juin 10:00 . drwxr-xr-x 5 root root 4096 4 juin 17:53 .. -rw-r--r-- 1 root root 4096 30 juin 10:00 alock -rw------- 1 root root 24576 30 juin 10:00 __db.001 -rw------- 1 root root 65896448 30 juin 10:08 __db.002 -rw------- 1 root root 1073741824 30 juin 10:08 __db.003 -rw------- 1 root root 2359296 30 juin 10:07 __db.004 -rw------- 1 root root 35831808 30 juin 10:08 __db.005 -rw------- 1 root root 32768 30 juin 10:01 __db.006 -rw-r--r-- 1 root root 580 3 juin 08:28 DB_CONFIG -rw------- 1 root root 372736 23 juin 18:48 dn2id.bdb -rw------- 1 root root 2228224 24 juin 13:35 id2entry.bdb -rw------- 1 root root 10485760 30 juin 10:01 log.0000000003 -rw------- 1 root root 65536 23 juin 18:48 objectClass.bdb
and attached my DB_CONFIG and slapd.conf, as well as a simple ldap research
Updated by Vincent MEMBRÉ about 9 years ago
- Target version changed from 3.0.7 to 3.0.8
Updated by Nicolas CHARLES about 9 years ago
Some more data:
i can reproduce my bug (wrong search) quite reliably. I need to accept a node after having burdened a bit ldap
- search directly on nodes work ok (on hostname, OS, primary ip adresse)
- search combine with sub element (mountpoint, networkinterface) fail to find the new node
- search combine with hardware (bios name) fail
- Node does show up on the web interface, with complete inventory
request that work and don't works
ldapsearch -H ldap://localhost:389/ -D cn=manager,cn=rudder-configuration -w 3cfd665c2c8b -b 'ou=Machines,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration' -s sub '(&(objectClass=machine)(objectClass=*))' 1.1 motherBoardUuid
Correcly finds the new node
ldapsearch -H ldap://localhost:389/ -D cn=manager,cn=rudder-configuration -w 3cfd665c2c8b -b 'networkInterface=eth1,nodeId=11c74371-062c-4b9d-a089-d28cc1b5f637,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration' -s base
correctly find the network interface
ldapsearch -H ldap://localhost:389/ -D cn=manager,cn=rudder-configuration -w 3cfd665c2c8b -b 'nodeId=11c74371-062c-4b9d-a089-d28cc1b5f637,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration' -s one '(&(objectClass=networkInterfaceLogicalElement)(objectClass=*))'
correctly find the network interface
ldapsearch -H ldap://localhost:389/ -D cn=manager,cn=rudder-configuration -w 3cfd665c2c8b -b 'ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration' -s sub '(&(objectClass=networkInterfaceLogicalElement)(objectClass=*))' 1.1 ipHostNumber
Does not find the network interface
ldapsearch -H ldap://localhost:389/ -D cn=manager,cn=rudder-configuration -w 3cfd665c2c8b -b 'ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration' -s sub '(&(objectClass=networkInterfaceLogicalElement)(objectClass=*))' 1.1 ipHostNumber
Does not find the network interface
ldapsearch -H ldap://localhost:389/ -D cn=manager,cn=rudder-configuration -w 3cfd665c2c8b -b 'ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration' -s sub 'networkInterfaceMacAddress=08:00:27:06:97:21'
does not find the network interface
ldapsearch -H ldap://localhost:389/ -D cn=manager,cn=rudder-configuration -w 3cfd665c2c8b -b 'nodeId=11c74371-062c-4b9d-a089-d28cc1b5f637,ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration' -s sub '(&(objectClass=networkInterfaceLogicalElement)(objectClass=*))' 1.1 ipHostNumber
finds the network interface
Weird, huh ?
Updated by Nicolas CHARLES about 9 years ago
uname -a Linux server 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Centos 6.3
Updated by Nicolas CHARLES about 9 years ago
restarting rudder-slapd solves the issue
Updated by Vincent MEMBRÉ about 9 years ago
- Target version changed from 3.0.8 to 3.0.9
Updated by Nicolas CHARLES almost 9 years ago
- Related to Bug #6931: Update OpenLDAP to 2.4.41 added
Updated by Nicolas CHARLES almost 9 years ago
- Status changed from New to Rejected
This was probably sovled by #6931 ; rejecting this ticket
Dennis, if the issue still occurs, feel free to reopen this ticket
Thank you !
Nicolas
Updated by Nicolas CHARLES over 8 years ago
Ha, i still have search error
@(#) $OpenLDAP: slapd 2.4.41 (Dec 10 2015 03:57:37) $ root@centos-builder-6-64.labo.normation.com:/usr/src/rudder-packages/package/BUILD/openldap-source/servers/slapd
a query on Operating system type linux + operating system name centos ++ ip address regex 192.168.42.1[3|4] fails at finding a given node
This specific node acceptation was super slow (more than 1 minutes without anything)
I do not have any error in the logs
replacing regex by ip == 192.168.42.14 does not work
searching Network Interface IP address is defined doesn't return my new node
restarting ldap solves the issue
Updated by Nicolas CHARLES over 8 years ago
the server is a centos6 server, which as been upgraded fro m 3.0~alpha1 to 3.15, in a lot of different phases
Updated by Nicolas CHARLES over 8 years ago
- Status changed from Rejected to New
- Target version changed from 3.0.9 to 3.0.14
I'm reviving this ticket, as it's been seen reapparing in the wild
Updated by François ARMAND over 8 years ago
- Subject changed from The Rudder web interface fails on LDAP to Node not included in dynamic group due to openldap bug with modrdn not showing node children
- Status changed from New to In progress
So, it appears again in CentOS 7 only, not sur why. Hard to reproduce, but we did it!
It appears that it seems to be a bug with modrdn on hdb that don't update the children grand-parents relation when moved, so the children still think they are on the old place, which explains why a search on the new grand parent does not show the children, but one directly on the parent DOES show them.
The bug seems to be related to idlcachesize, because setting it 0 make the problem not reproducible.
It seems that the last version of openldap (2.4.44) does not have the problem, but more confirmation need to be done since nothing in the changelog between our version (2.4.41) and that one is related.
Updated by Nicolas CHARLES over 8 years ago
- File openldap-dn2id-modrdn-idlcache-sub.patch openldap-dn2id-modrdn-idlcache-sub.patch added
- Description updated (diff)
- Assignee changed from François ARMAND to Benoît PECCATTE
- Target version changed from 3.0.14 to 2.11.19
Updated by Jonathan CLARKE over 8 years ago
Bug reported upstream at http://www.openldap.org/its/index.cgi/Incoming?id=8378
Updated by Alexis Mousset over 8 years ago
- Assignee changed from Benoît PECCATTE to Alexis Mousset
Updated by Alexis Mousset over 8 years ago
- Status changed from In progress to Pending technical review
- Assignee changed from Alexis Mousset to Jonathan CLARKE
- Pull Request set to https://github.com/Normation/rudder-packages/pull/886
Updated by Alexis Mousset over 8 years ago
- Status changed from Pending technical review to Pending release
- % Done changed from 0 to 100
Applied in changeset rudder-packages|1895911270cbf14a7ed42ee98e2684b49ff1a1a1.
Updated by Vincent MEMBRÉ over 8 years ago
- Status changed from Pending release to Released