Architecture #17128
closedreview index for LDAP
Added by Nicolas CHARLES over 4 years ago. Updated about 4 years ago.
Description
A long long time ago ( https://issues.rudder.io/issues/5965 ) we removed indexes within Rudder, by adding cache
Then we move to mdb, which doesn't have cache, but is super fast
It is fast, but some edge cases are unfortunately slow, or super slow. See for example https://issues.rudder.io/issues/12937 , or https://issues.rudder.io/issues/14945
We could improve perfs there, but we need also to measure what we are doing, as adding indexes make writing slower (and write is costly in ldap)
Updated by Nicolas CHARLES over 4 years ago
- Related to Bug #5965: LDAP configuration is not optimized for Rudder use case added
Updated by Nicolas CHARLES over 4 years ago
- Related to Bug #12937: In Rudder 6.2.0 inventory processing merge_uuid part get extremelly slow on debian added
Updated by Nicolas CHARLES over 4 years ago
- Related to Bug #14945: Missing index in ldap directory added
Updated by Nicolas CHARLES over 4 years ago
- Related to Bug #2751: Some ldap are not indexed added
Updated by Nicolas CHARLES over 4 years ago
as a reminder, indexes must be set with a stopped rudder-slapd, and then the command /opt/rudder/sbin/slapindex must be run - it is fairly slow (there's a -q option for quick)
unfortunately, we don't have the logs anymore of what is unidexed because of https://issues.rudder.io/issues/10429
Updated by Nicolas CHARLES over 4 years ago
On 6.1, after 1 days of operation, I have the following missing indexes (with occurences)
Equality¶
208 (agentName) not indexed
63694 (cn) not indexed
195618 (directiveId) not indexed
29 (isDynamic) not indexed
8 (isSystem) not indexed
26261 (nodeGroupId) not indexed
6117 (nodeId) not indexed
12905 (osName) not indexed
78 (policyServerId) not indexed
4 (software) not indexed
50578 (softwareVersion) not indexed
Inequality¶
14162 (modifyTimestamp) not indexed
Substring¶
416 (agentName) not indexed
3 (cn) not indexed
1 (ruleTarget) not indexed
stats¶
Dynamic group update are between 2m26 and 5M9, median at 3m19
Fetch rules at the begining of policy generation is between 17 and 342, median at 55ms
Inventory processing is between 3s711 and 9s57, with a median at 6s50
Get node infos: min 232, max 15501, av 444, median 368
Policy generation is around 13 or 14 minutes
Updated by Nicolas CHARLES over 4 years ago
testing with indexes
index cn eq index softwareVersion eq index directiveId eq
Interestingly, indexing with
/opt/rudder/sbin/slapindex -q
takes 2secondes on load platform
Stats¶
Dynamic groups update are between 2m30 and 3m30 (but there are only 12 recomputations)
Fetch rules at the begining of policy generation is between 21 and 100, median at 55
Inventory processing is between 160 and 928ms with a median at 215 ms
Get node info: min 171, max 63902, average 255, median 211
Policy generation is around 12m30-13m50
There aren't any visible regression. I had one IO issues when copying too much inventories at once but that was my mistake
Web interface feels faster (but that's hard to measure)
Remaining ldap missing index¶
For equality
88 (agentName) not indexed 11 (isDynamic) not indexed 18 (isSystem) not indexed 10105 (nodeGroupId) not indexed 6264 (nodeId) not indexed 4959 (osName) not indexed 33 (policyServerId) not indexed 8 (software) not indexed
Updated by Nicolas CHARLES over 4 years ago
Interestingly, add ing index
index nodeGroupId eq
breaks the computation of dynamic groups for relays
[2020-04-22 00:17:04] ERROR scheduledJob - Error when updating dynamic group 'hasPolicyServer-0b448d1d-16c0-408e-a97c-4c8dfd1d352d' <- Inconsistency: Error when retrieving the entry for NodeGroup 'hasPolicyServer-0b448d1d-16c0-408e-a97c-4c8dfd1d352d
Updated by Nicolas CHARLES over 4 years ago
Finally, these index make everything much faster when a node is already there for inentory processing, but it's much less impressive when node is not yet accepted: we are still in the 3-5 secondes area if inventory was never seen
Updated by Nicolas CHARLES over 4 years ago
- Status changed from New to In progress
- Assignee set to Nicolas CHARLES
Updated by Nicolas CHARLES over 4 years ago
- Assignee changed from Nicolas CHARLES to François ARMAND
- Pull Request set to https://github.com/Normation/rudder-packages/pull/2249
Updated by Nicolas CHARLES over 4 years ago
- Status changed from In progress to Pending technical review
Updated by Nicolas CHARLES over 4 years ago
- Status changed from Pending technical review to Pending release
Applied in changeset rudder-packages|bfb0e6e8f1378166012c132d5fee05d8b238b259.
Updated by François ARMAND over 4 years ago
- Tracker changed from Bug to Architecture
- Priority deleted (
0)
Updated by Vincent MEMBRÉ over 4 years ago
This bug has been fixed in Rudder 6.1.0~beta1 which was released today.
Updated by Vincent MEMBRÉ over 4 years ago
- Related to Bug #17215: create migration script for LDAP indexes added
Updated by Vincent MEMBRÉ over 4 years ago
- Status changed from Pending release to Released
Updated by François ARMAND about 4 years ago
- Fix check changed from To do to Checked