Architecture #14939
openInefficient storage of software in LDAP directory
Added by Nicolas CHARLES over 5 years ago. Updated 15 days ago.
Description
Software details are store in a specific branch, with deduplication, so that's great, but we store the full dn of these software in a node.
So every node get 2000 entries like
softwareId=000ddc3b-6c0a-4394-bb52-ad9cf012658d,ou=Software,ou=Inventories,cn=rudder-configuration
there is about twice as much data as necessary there, we could simply have the uuid and not the full DN to the software
Impact is double:- ldap directory is bigger than necessary, so its slower than necessary
- we need much more memory for everything:
when we get a node, we need to transport the thousands softwares dn, convert them to string, and manipulate it
memory impact is huge
we keep moving from softwareid that's an id to software dn that's a dn
Getting, from all the nodes, all the softwares entry, with 1000 nodes result in 1.5GB of memory, and a huge strain on GC
Graph shows big spike where all softwares are fetched, the frop is a GC
Files
graph.png (52.8 KB) graph.png | memory usage | Nicolas CHARLES, 2019-05-22 22:34 | |
memory.png (34.3 KB) memory.png | classes | Nicolas CHARLES, 2019-05-22 22:34 |
Updated by Vincent MEMBRÉ about 5 years ago
- Target version changed from 6.0.0~beta1 to 6.0.0
Updated by Alexis Mousset almost 5 years ago
- Target version changed from 6.0.0 to 6.0.1
Updated by Vincent MEMBRÉ almost 5 years ago
- Target version changed from 6.0.1 to 6.0.2
Updated by Vincent MEMBRÉ almost 5 years ago
- Target version changed from 6.0.2 to 6.0.3
Updated by Vincent MEMBRÉ almost 5 years ago
- Target version changed from 6.0.3 to 6.0.4
Updated by Vincent MEMBRÉ over 4 years ago
- Target version changed from 6.0.4 to 6.0.5
Updated by Vincent MEMBRÉ over 4 years ago
- Target version changed from 6.0.5 to 6.0.6
Updated by Vincent MEMBRÉ over 4 years ago
- Target version changed from 6.0.6 to 6.0.7
Updated by Vincent MEMBRÉ over 4 years ago
- Target version changed from 6.0.7 to 6.0.8
Updated by François ARMAND over 4 years ago
- Severity set to Major - prevents use of part of Rudder | no simple workaround
- User visibility set to Operational - other Techniques | Rudder settings | Plugins
- Priority changed from 0 to 43
We need a clear path to get that thing better.
The problem is that we need to support for a time both encoding, since people won't be able to migrate in one go (it would be likely too long ?), and they may have dyn group based on software.
We can:
- just change dn to UUID (note that the type of softwareDn is "dn", so we will need an other attribute name in all case for uuid only; )
- change encoding and store soft in node directly.
1/ allows to have global dedup, which is good for highly similar infra with lots of identical distro/stack.
2/ allows to never have cleaning batch, which are consuming and lead to period of horrible performances
Updated by François ARMAND over 4 years ago
- Related to Architecture #14923: Dynamic groups with regex on software are long to build delaying generation added
Updated by Vincent MEMBRÉ over 4 years ago
- Target version changed from 6.0.8 to 6.0.9
Updated by Vincent MEMBRÉ about 4 years ago
- Target version changed from 6.0.9 to 6.0.10
Updated by Vincent MEMBRÉ about 4 years ago
- Target version changed from 6.0.10 to 798
Updated by Benoît PECCATTE over 3 years ago
- Target version changed from 798 to 6.1.14
Updated by Vincent MEMBRÉ over 3 years ago
- Target version changed from 6.1.14 to 6.1.15
Updated by Vincent MEMBRÉ over 3 years ago
- Target version changed from 6.1.15 to 6.1.16
Updated by Vincent MEMBRÉ about 3 years ago
- Target version changed from 6.1.16 to 6.1.17
Updated by Vincent MEMBRÉ about 3 years ago
- Target version changed from 6.1.17 to 6.1.18
Updated by Vincent MEMBRÉ almost 3 years ago
- Target version changed from 6.1.18 to 6.1.19
Updated by François ARMAND almost 3 years ago
- Tracker changed from Bug to Architecture
- Severity deleted (
Major - prevents use of part of Rudder | no simple workaround) - User visibility deleted (
Operational - other Techniques | Rudder settings | Plugins) - Priority deleted (
43)
Updated by Vincent MEMBRÉ over 2 years ago
- Target version changed from 6.1.19 to 6.1.20
Updated by Vincent MEMBRÉ over 2 years ago
- Target version changed from 6.1.20 to 6.1.21
Updated by Vincent MEMBRÉ over 2 years ago
- Target version changed from 6.1.21 to old 6.1 issues to relocate
Updated by Alexis Mousset over 1 year ago
- Target version changed from old 6.1 issues to relocate to 8.0.0~alpha1
Updated by Vincent MEMBRÉ over 1 year ago
- Target version changed from 8.0.0~alpha1 to 8.0.0~beta1
Updated by Alexis Mousset over 1 year ago
- Target version changed from 8.0.0~beta1 to 8.1.0~alpha1
Updated by Vincent MEMBRÉ 10 months ago
- Target version changed from 8.1.0~alpha1 to 8.1.0~beta1
Updated by Vincent MEMBRÉ 9 months ago
- Target version changed from 8.1.0~beta1 to 8.1.0~beta2
Updated by Vincent MEMBRÉ 8 months ago
- Target version changed from 8.1.0~beta2 to 8.1.0~rc1
Updated by Vincent MEMBRÉ 8 months ago
- Target version changed from 8.1.0~rc1 to 8.1.0
Updated by Vincent MEMBRÉ 8 months ago
- Target version changed from 8.1.0 to 8.1.1
Updated by Vincent MEMBRÉ 7 months ago
- Target version changed from 8.1.1 to 8.1.2
Updated by Vincent MEMBRÉ 7 months ago
- Target version changed from 8.1.2 to 8.1.3
Updated by Vincent MEMBRÉ 6 months ago
- Target version changed from 8.1.3 to 8.1.4
Updated by Vincent MEMBRÉ 5 months ago
- Target version changed from 8.1.4 to 8.1.5
Updated by Vincent MEMBRÉ 5 months ago
- Target version changed from 8.1.5 to 8.1.6
Updated by Vincent MEMBRÉ 4 months ago
- Target version changed from 8.1.6 to 8.1.7
Updated by Vincent MEMBRÉ 2 months ago
- Target version changed from 8.1.7 to 8.1.8
Updated by Vincent MEMBRÉ 15 days ago
- Target version changed from 8.1.8 to 8.1.9