Project

General

Profile

Actions

Architecture #14939

open

Inefficient storage of software in LDAP directory

Added by Nicolas CHARLES almost 5 years ago. Updated about 3 hours ago.

Status:
New
Priority:
N/A
Assignee:
-
Category:
Performance and scalability
Target version:
Effort required:
Name check:
Fix check:
Regression:

Description

Software details are store in a specific branch, with deduplication, so that's great, but we store the full dn of these software in a node.
So every node get 2000 entries like

softwareId=000ddc3b-6c0a-4394-bb52-ad9cf012658d,ou=Software,ou=Inventories,cn=rudder-configuration

there is about twice as much data as necessary there, we could simply have the uuid and not the full DN to the software

Impact is double:
  • ldap directory is bigger than necessary, so its slower than necessary
  • we need much more memory for everything:

when we get a node, we need to transport the thousands softwares dn, convert them to string, and manipulate it
memory impact is huge
we keep moving from softwareid that's an id to software dn that's a dn

Getting, from all the nodes, all the softwares entry, with 1000 nodes result in 1.5GB of memory, and a huge strain on GC

Graph shows big spike where all softwares are fetched, the frop is a GC


Files

graph.png (52.8 KB) graph.png memory usage Nicolas CHARLES, 2019-05-22 22:34
memory.png (34.3 KB) memory.png classes Nicolas CHARLES, 2019-05-22 22:34

Related issues 1 (1 open0 closed)

Related to Rudder - Architecture #14923: Dynamic groups with regex on software are long to build delaying generationNewFrançois ARMANDActions
Actions #1

Updated by Nicolas CHARLES almost 5 years ago

Actions #2

Updated by Nicolas CHARLES almost 5 years ago

Actions #3

Updated by Nicolas CHARLES almost 5 years ago

  • Description updated (diff)
Actions #4

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 6.0.0~beta1 to 6.0.0
Actions #5

Updated by Alexis Mousset over 4 years ago

  • Target version changed from 6.0.0 to 6.0.1
Actions #6

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 6.0.1 to 6.0.2
Actions #7

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 6.0.2 to 6.0.3
Actions #8

Updated by Vincent MEMBRÉ about 4 years ago

  • Target version changed from 6.0.3 to 6.0.4
Actions #9

Updated by Vincent MEMBRÉ about 4 years ago

  • Target version changed from 6.0.4 to 6.0.5
Actions #10

Updated by Vincent MEMBRÉ about 4 years ago

  • Target version changed from 6.0.5 to 6.0.6
Actions #11

Updated by Vincent MEMBRÉ almost 4 years ago

  • Target version changed from 6.0.6 to 6.0.7
Actions #12

Updated by Vincent MEMBRÉ almost 4 years ago

  • Target version changed from 6.0.7 to 6.0.8
Actions #13

Updated by François ARMAND almost 4 years ago

  • Severity set to Major - prevents use of part of Rudder | no simple workaround
  • User visibility set to Operational - other Techniques | Rudder settings | Plugins
  • Priority changed from 0 to 43

We need a clear path to get that thing better.
The problem is that we need to support for a time both encoding, since people won't be able to migrate in one go (it would be likely too long ?), and they may have dyn group based on software.

We can:
- just change dn to UUID (note that the type of softwareDn is "dn", so we will need an other attribute name in all case for uuid only; )
- change encoding and store soft in node directly.

1/ allows to have global dedup, which is good for highly similar infra with lots of identical distro/stack.
2/ allows to never have cleaning batch, which are consuming and lead to period of horrible performances

Actions #14

Updated by François ARMAND almost 4 years ago

  • Related to Architecture #14923: Dynamic groups with regex on software are long to build delaying generation added
Actions #15

Updated by Vincent MEMBRÉ almost 4 years ago

  • Target version changed from 6.0.8 to 6.0.9
Actions #16

Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 6.0.9 to 6.0.10
Actions #17

Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 6.0.10 to 798
Actions #18

Updated by Benoît PECCATTE almost 3 years ago

  • Target version changed from 798 to 6.1.14
Actions #19

Updated by Vincent MEMBRÉ almost 3 years ago

  • Target version changed from 6.1.14 to 6.1.15
Actions #20

Updated by Vincent MEMBRÉ almost 3 years ago

  • Target version changed from 6.1.15 to 6.1.16
Actions #21

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.16 to 6.1.17
Actions #22

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.17 to 6.1.18
Actions #23

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.18 to 6.1.19
Actions #24

Updated by François ARMAND about 2 years ago

  • Tracker changed from Bug to Architecture
  • Severity deleted (Major - prevents use of part of Rudder | no simple workaround)
  • User visibility deleted (Operational - other Techniques | Rudder settings | Plugins)
  • Priority deleted (43)
Actions #25

Updated by Vincent MEMBRÉ about 2 years ago

  • Target version changed from 6.1.19 to 6.1.20
Actions #26

Updated by Vincent MEMBRÉ almost 2 years ago

  • Target version changed from 6.1.20 to 6.1.21
Actions #27

Updated by Vincent MEMBRÉ almost 2 years ago

  • Target version changed from 6.1.21 to old 6.1 issues to relocate
Actions #28

Updated by Alexis Mousset about 1 year ago

  • Target version changed from old 6.1 issues to relocate to 8.0.0~alpha1
Actions #29

Updated by Vincent MEMBRÉ 10 months ago

  • Target version changed from 8.0.0~alpha1 to 8.0.0~beta1
Actions #30

Updated by Alexis Mousset 9 months ago

  • Target version changed from 8.0.0~beta1 to 8.1.0~alpha1
Actions #31

Updated by Vincent MEMBRÉ 3 months ago

  • Target version changed from 8.1.0~alpha1 to 8.1.0~beta1
Actions #32

Updated by Vincent MEMBRÉ about 2 months ago

  • Target version changed from 8.1.0~beta1 to 8.1.0~beta2
Actions #33

Updated by Vincent MEMBRÉ about 1 month ago

  • Target version changed from 8.1.0~beta2 to 8.1.0~rc1
Actions #34

Updated by Vincent MEMBRÉ 19 days ago

  • Target version changed from 8.1.0~rc1 to 8.1.0
Actions #35

Updated by Vincent MEMBRÉ 14 days ago

  • Target version changed from 8.1.0 to 8.1.1
Actions #36

Updated by Vincent MEMBRÉ about 3 hours ago

  • Target version changed from 8.1.1 to 8.1.2
Actions

Also available in: Atom PDF