Project

General

Profile

Actions

Bug #10405

closed

Bug #10373: Upgrading from 4.0 to 4.1 failed on Centos7.3, and purged LDAP directory

Bad way to calculate the size of mdb database for LDAP data

Added by François ARMAND over 7 years ago. Updated over 7 years ago.

Status:
Released
Priority:
N/A
Category:
System integration
Target version:
Severity:
Critical - prevents main use of Rudder | no workaround | data loss | security
UX impact:
User visibility:
Operational - other Techniques | Technique editor | Rudder settings
Effort required:
Small
Priority:
68
Name check:
Fix check:
Regression:

Description

With the parent task merged, we are now correctly setting the mdb database size during migration.

But we are not at all using the correct way to calculate it, because we are still using the old way which was for a cache, so depending on RAM, and not for a storage size.

The problem is that it is not trivial to update mdb storage size if limit is reached, and the consequences can be very grave (losing data).

So, we need to:

- find the normal database size for a set of production install,
- find a sufficiently high value so that, if possible, any installation can run a couple of thousand node without even knowing that that limit exists (tracked on that ticket)
- document all of that (tracked in #10404)

Actions #1

Updated by François ARMAND over 7 years ago

  • Subject changed from By way to calcul to size of mdb database for LDAP data to Bad way to calcul to size of mdb database for LDAP data
  • Category set to System integration
Actions #3

Updated by Nicolas CHARLES over 7 years ago

The parameter maxsize defines the maximum size the database is allowed to grow (see https://sites.google.com/site/bmaupinwiki/home/applications/ldap/optimizing-openldap )
There is no reason to not put a huge number, as it will only use what is necessary by the inventories and configurations

For instance, i put maxsize 24644718592 , but got a 436 MB file with 2000 auto-generated inventories
7000 real nodes translate to 4.3 GB

It would be really sad to have a catastrophic failure at some point because value was too conservative

Actions #4

Updated by François ARMAND over 7 years ago

So, there doesn't seem to have a problem to set, say, 10GB (or even 100GB) for that? If so, why there is even a size?

Actions #5

Updated by Nicolas CHARLES over 7 years ago

Don't know - someone more versed in openldap may have an answer

Actions #6

Updated by François ARMAND over 7 years ago

Perhaps we should also add to the stats info we can collect that info, to have a better idea of the actual size.

Actions #7

Updated by François ARMAND over 7 years ago

  • Subject changed from Bad way to calcul to size of mdb database for LDAP data to Bad way to calculate the size of mdb database for LDAP data
Actions #8

Updated by Jonathan CLARKE over 7 years ago

  • User visibility set to Operational - other Techniques | Technique editor | Rudder settings
  • Effort required set to Small
Actions #9

Updated by François ARMAND over 7 years ago

  • Assignee set to Benoît PECCATTE

So, given http://www.openldap.org/lists/openldap-technical/201208/msg00177.html ("set it to a few hundred GB") and the constraint that it must be adressable, we can set it to:

- 2 GB on 32 bits proc (note: it must be documented that 32bit proc won't support more than a couple thousands nodes)
- 100Go on 64 bits. The current size for ~7000 nodes is ~2,5 GB, so that is more than one order of magnitude above.

Actions #10

Updated by Benoît PECCATTE over 7 years ago

  • Status changed from New to In progress
Actions #11

Updated by Benoît PECCATTE over 7 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Benoît PECCATTE to Alexis Mousset
  • Pull Request set to https://github.com/Normation/rudder-packages/pull/1290
Actions #12

Updated by Benoît PECCATTE over 7 years ago

  • Status changed from Pending technical review to Pending release
Actions #13

Updated by Benoît PECCATTE over 7 years ago

  • Priority set to 68
Actions #14

Updated by Benoît PECCATTE over 7 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 4.1.0 which was released today.

Actions

Also available in: Atom PDF