Project

General

Profile

Bug #9609

Deleted node should be periodically fully erased in LDAP (after some ttl)

Added by Janos Mattyasovszky almost 3 years ago. Updated 3 months ago.

Status:
Released
Priority:
N/A
Category:
Performance and scalability
Target version:
Severity:
Minor - inconvenience | misleading | easy workaround
User visibility:
Operational - other Techniques | Technique editor | Rudder settings
Effort required:
Small
Priority:
0

Description

When you delete a node from Rudder, the inventories in the LDAP Backend are not deleted, just moved to the "ou=Removed Inventories". Having this behavior will basically pile up all ever accepted nodes with time, with currently no real way (except deleting them directly in LDAP, which is not nice) to either permanently delete them or to have a housekeeping on the entries to "age out" after X amount of time.

My proposals:
  • Add an API Feature to permanently delete the nodes from Rudder
  • Implement some kind of housekeeping that can be activated to delete removed entries older than X (like you currently have a TTL for reports)

Related issues

Related to Rudder - Bug #9719: Node deletes are not properly cleaned upNewActions
Related to Rudder - Bug #14707: Impossible to recreate a node deletedRejectedActions
Related to Rudder - Bug #14923: Dynamic groups with regex on software are long to build delaying generationNewActions

Associated revisions

Revision 3cbcc100 (diff)
Added by Nicolas CHARLES 5 months ago

Fixes #9609: Deleted node should be periodically fully erased in LDAP (after some ttl)

History

#1

Updated by Jonathan CLARKE almost 3 years ago

I agree it would be nice to have a mechanism to delete old nodes for housekeeping purposes.

However it should be noted that at chest glance the design of OpenLDAP very much reduces any negative impact unused entries may have:

  • The total number of entries in the directory can be in the millions before OpenLDAP will show slowdown, for example.
  • OpenLDAP caches are only loaded with entries and search results that are actually used, so unless you query the deleted nodes, they will never enter the cache (which is where all results should be served from)
  • We don't use any indexes (apart from the mandatory objectClass which is just one entry per object) so the unused entries will not weigh in here either (see https://github.com/Normation/rudder-packages/blob/master/rudder-inventory-ldap/SOURCES/slapd.conf#L52)

However, I see that we warm up the cache by reading all entries in https://github.com/Normation/rudder-packages/blob/master/rudder-inventory-ldap/SOURCES/rudder-inventory-ldap.init#L445, including these deleted entries. That is a waste of cache space. We should change the warmup script.

#2

Updated by Janos Mattyasovszky almost 3 years ago

Depends on how you see it, but currently there is a potential to have thousands of dead entries every year, and after a couple of years the removed would outweight the active entries by almost 2:1...

Having them in LDAP does not give any benefit against not having them, which OTOH would speed up startup (as already mentioned) and not to talk about backup/restore times...

Our current slapcat-based backup is uncompressed 1.6GB big, and takes quite some minutes to make, so having BACKUP_AT_SHUTDOWN="1" in the rudder-slapd also makes me question it if this backup is right in that place to be performed at basically each slapd restart, taking a lot of extra minutes to restart slapd.

I don't say slapd is not good to have millions of entries, just the current usage is not a very optimal one...

I'll do some testing tomorrow to see how much space/time these deleted inventories actually take up in % of disk space of backup / time for backup.

#3

Updated by Janos Mattyasovszky almost 3 years ago

  • Related to Bug #9719: Node deletes are not properly cleaned up added
#4

Updated by Benoît PECCATTE over 2 years ago

  • Category set to Performance and scalability
#5

Updated by Benoît PECCATTE over 2 years ago

  • Severity set to Minor - inconvenience | misleading | easy workaround
  • User visibility set to Operational - other Techniques | Technique editor | Rudder settings
  • Priority set to 16
#6

Updated by Jonathan CLARKE over 2 years ago

  • Assignee deleted (Nicolas CHARLES)
#7

Updated by Benoît PECCATTE over 2 years ago

  • Priority changed from 16 to 15
#8

Updated by Benoît PECCATTE over 2 years ago

  • Priority changed from 15 to 29
#9

Updated by Benoît PECCATTE almost 2 years ago

  • Priority changed from 29 to 27
#10

Updated by François ARMAND 11 months ago

  • Subject changed from Deleting a node does not permanently delete the LDAP entries to Deleted node should be periodically fully erased in LDAP (after some ttl)
  • Effort required set to Small
  • Priority changed from 27 to 0
#11

Updated by Alexis MOUSSET 6 months ago

  • Related to Bug #14707: Impossible to recreate a node deleted added
#12

Updated by Nicolas CHARLES 5 months ago

  • Related to Bug #14923: Dynamic groups with regex on software are long to build delaying generation added
#13

Updated by Nicolas CHARLES 5 months ago

  • Target version set to 5.0.12
#14

Updated by Nicolas CHARLES 5 months ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES
#15

Updated by Nicolas CHARLES 5 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/2229
#16

Updated by Rudder Quality Assistant 5 months ago

  • Status changed from Pending technical review to Discussion
  • Assignee changed from François ARMAND to Nicolas CHARLES
#17

Updated by Nicolas CHARLES 5 months ago

  • Status changed from Discussion to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
#18

Updated by Rudder Quality Assistant 5 months ago

  • Status changed from Pending technical review to Discussion
  • Assignee changed from François ARMAND to Nicolas CHARLES
#19

Updated by Nicolas CHARLES 5 months ago

  • Status changed from Discussion to Pending release
#24

Updated by Vincent MEMBRÉ 3 months ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 5.0.12 which was released today.

Also available in: Atom PDF