Project

General

Profile

Actions

Bug #9609

closed

Deleted node should be periodically fully erased in LDAP (after some ttl)

Added by Janos Mattyasovszky over 7 years ago. Updated over 4 years ago.

Status:
Released
Priority:
N/A
Category:
Performance and scalability
Target version:
Severity:
Minor - inconvenience | misleading | easy workaround
UX impact:
User visibility:
Operational - other Techniques | Technique editor | Rudder settings
Effort required:
Small
Priority:
0
Name check:
Reviewed
Fix check:
Checked
Regression:

Description

When you delete a node from Rudder, the inventories in the LDAP Backend are not deleted, just moved to the "ou=Removed Inventories". Having this behavior will basically pile up all ever accepted nodes with time, with currently no real way (except deleting them directly in LDAP, which is not nice) to either permanently delete them or to have a housekeeping on the entries to "age out" after X amount of time.

My proposals:
  • Add an API Feature to permanently delete the nodes from Rudder
  • Implement some kind of housekeeping that can be activated to delete removed entries older than X (like you currently have a TTL for reports)

Related issues 3 (1 open2 closed)

Related to Rudder - Question #9719: Node deletes are not properly cleaned upResolvedActions
Related to Rudder - Bug #14707: Impossible to recreate a node deletedRejectedFrançois ARMANDActions
Related to Rudder - Architecture #14923: Dynamic groups with regex on software are long to build delaying generationNewFrançois ARMANDActions
Actions #1

Updated by Jonathan CLARKE over 7 years ago

I agree it would be nice to have a mechanism to delete old nodes for housekeeping purposes.

However it should be noted that at chest glance the design of OpenLDAP very much reduces any negative impact unused entries may have:

  • The total number of entries in the directory can be in the millions before OpenLDAP will show slowdown, for example.
  • OpenLDAP caches are only loaded with entries and search results that are actually used, so unless you query the deleted nodes, they will never enter the cache (which is where all results should be served from)
  • We don't use any indexes (apart from the mandatory objectClass which is just one entry per object) so the unused entries will not weigh in here either (see https://github.com/Normation/rudder-packages/blob/master/rudder-inventory-ldap/SOURCES/slapd.conf#L52)

However, I see that we warm up the cache by reading all entries in https://github.com/Normation/rudder-packages/blob/master/rudder-inventory-ldap/SOURCES/rudder-inventory-ldap.init#L445, including these deleted entries. That is a waste of cache space. We should change the warmup script.

Actions #2

Updated by Janos Mattyasovszky over 7 years ago

Depends on how you see it, but currently there is a potential to have thousands of dead entries every year, and after a couple of years the removed would outweight the active entries by almost 2:1...

Having them in LDAP does not give any benefit against not having them, which OTOH would speed up startup (as already mentioned) and not to talk about backup/restore times...

Our current slapcat-based backup is uncompressed 1.6GB big, and takes quite some minutes to make, so having BACKUP_AT_SHUTDOWN="1" in the rudder-slapd also makes me question it if this backup is right in that place to be performed at basically each slapd restart, taking a lot of extra minutes to restart slapd.

I don't say slapd is not good to have millions of entries, just the current usage is not a very optimal one...

I'll do some testing tomorrow to see how much space/time these deleted inventories actually take up in % of disk space of backup / time for backup.

Actions #3

Updated by Janos Mattyasovszky over 7 years ago

  • Related to Question #9719: Node deletes are not properly cleaned up added
Actions #4

Updated by Benoît PECCATTE about 7 years ago

  • Category set to Performance and scalability
Actions #5

Updated by Benoît PECCATTE almost 7 years ago

  • Severity set to Minor - inconvenience | misleading | easy workaround
  • User visibility set to Operational - other Techniques | Technique editor | Rudder settings
  • Priority set to 16
Actions #6

Updated by Jonathan CLARKE almost 7 years ago

  • Assignee deleted (Nicolas CHARLES)
Actions #7

Updated by Benoît PECCATTE almost 7 years ago

  • Priority changed from 16 to 15
Actions #8

Updated by Benoît PECCATTE almost 7 years ago

  • Priority changed from 15 to 29
Actions #9

Updated by Benoît PECCATTE over 6 years ago

  • Priority changed from 29 to 27
Actions #10

Updated by François ARMAND over 5 years ago

  • Subject changed from Deleting a node does not permanently delete the LDAP entries to Deleted node should be periodically fully erased in LDAP (after some ttl)
  • Effort required set to Small
  • Priority changed from 27 to 0
Actions #11

Updated by Alexis Mousset almost 5 years ago

  • Related to Bug #14707: Impossible to recreate a node deleted added
Actions #12

Updated by Nicolas CHARLES almost 5 years ago

  • Related to Architecture #14923: Dynamic groups with regex on software are long to build delaying generation added
Actions #13

Updated by Nicolas CHARLES almost 5 years ago

  • Target version set to 5.0.12
Actions #14

Updated by Nicolas CHARLES almost 5 years ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES
Actions #15

Updated by Nicolas CHARLES almost 5 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/2229
Actions #16

Updated by Rudder Quality Assistant almost 5 years ago

  • Status changed from Pending technical review to Discussion
  • Assignee changed from François ARMAND to Nicolas CHARLES
Actions #17

Updated by Nicolas CHARLES almost 5 years ago

  • Status changed from Discussion to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
Actions #18

Updated by Rudder Quality Assistant almost 5 years ago

  • Status changed from Pending technical review to Discussion
  • Assignee changed from François ARMAND to Nicolas CHARLES
Actions #19

Updated by Nicolas CHARLES almost 5 years ago

  • Status changed from Discussion to Pending release
Actions #20

Updated by Vincent MEMBRÉ almost 5 years ago

  • Name check set to To do
Actions #21

Updated by Vincent MEMBRÉ almost 5 years ago

  • Fix check set to To do
Actions #22

Updated by François ARMAND almost 5 years ago

  • Fix check changed from To do to Checked
Actions #23

Updated by Alexis Mousset almost 5 years ago

  • Name check changed from To do to Reviewed
Actions #24

Updated by Vincent MEMBRÉ over 4 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 5.0.12 which was released today.

Actions

Also available in: Atom PDF