User story #4560
closed
Keep only the LDIF dump of the inventory when we accept/refuse it, not every time we receive a new inventory
Description
We keep on the policy server all the inventory received, and timestamp them.
They can add up to a fairly large amount of space (several GB), after several months/years
It would be great if Rudder could automagically compress all those older than x days, and maybe even delete those older than y days (but the compress would be enough for a start)
Updated by Vincent MEMBRÉ over 10 years ago
- Target version changed from 2.6.11 to 2.6.12
Updated by Christophe Nowicki over 10 years ago
Logrotate looks perfect for that ;-)
http://iain.cx/src/logrotate/
Updated by François ARMAND over 10 years ago
Some more information about that ticket:
- files in /var/rudder/inventories/debug can be deactivated with the "trace.ldif.in.file" logger option which is set to "off" by default (see comment in logback.xml: [1])
- files in /var/rudder/inventories/historical are from InventoryHistorizationPostCommit. Their should be at least an option to disable that, disabled by default.
[1]
Debug LDAP write operations
===========================
This logger allow to trace LDAP writes operation related
receiving inventories and to output them in LDIF file
(the output directory path
is configured in the main configuration file)
The trace is done only if level is set to "trace"
WARNING: setting the level to trace may have major
performance issue, as A LOT of LDIF files will have
to be written.
You should activate that log only for debugging purpose.
Updated by François ARMAND over 10 years ago
- Assignee set to Jonathan CLARKE
So more more info: at the begining of Rudder, these files were saved to keep a trace of inventories received by the endpoint and be able to show inventories diff to the user. The feature was remove a long time ago, because it was not OK to display LDIF to an end user.
So, with more thinking, I believe we should just delete the feature that keeps a LDIF of each received inventories :
- the actual implementation is naive (same inventories are kept for example) and a correct implementation should output datas in something else than LDIF format (JSON ?), and at least use Git internally to store them in an efficient way.
- it takes a lot of place for nothing (the diffing inventory feature is not in Rudder for now, so there is no need to keep dead code (we could add it back if needed))
- a similar feature may be implemented in shell from the script that takes received inventories and send them to the endpoint (so if someone need that kind of history, it could be done, with files in XML in place of LDIF)
But we should do it in 2.10 nevertheless.
Jon, what do you thing ?
Note that the directory /var/rudder/inventories/historical should be kept, because we need to have one historical LDIF at the moment of the node acceptation/refusal to be able to display the information to the user. But that historical LDIF is generated elsewhere, when the node is accepted/refused.
Updated by François ARMAND over 10 years ago
- Status changed from New to Discussion
- Target version changed from 2.6.12 to 2.10.0~beta1
Updated by François ARMAND over 10 years ago
- Subject changed from Historical inventories add up to a lot of used space to Do not write an LDIF dump of inventories when they are received
- Status changed from Discussion to In progress
- Assignee changed from Jonathan CLARKE to François ARMAND
- Priority changed from N/A to 1 (highest)
We decided that the feature, implemented as it is, is just a dead weight for a future working implementation. So for now, just remove that dead code which is impacting diskspace.
Updated by François ARMAND over 10 years ago
- Status changed from In progress to Pending technical review
- Assignee changed from François ARMAND to Nicolas CHARLES
Updated by Nicolas CHARLES over 10 years ago
- Pull Request set to https://github.com/Normation/ldap-inventory/pull/32
The PR is there https://github.com/Normation/ldap-inventory/pull/32
Updated by Nicolas CHARLES over 10 years ago
- Status changed from Pending technical review to In progress
- Assignee changed from Nicolas CHARLES to Francois BAYART
Francois,
I have some concerns about this PR (see the comments in github), and I seem to be misunderstanding something.
Is there a PR missing from rudder ?
Updated by François ARMAND over 10 years ago
- Status changed from In progress to Pending technical review
- Assignee changed from Francois BAYART to Nicolas CHARLES
Replies available
Updated by François ARMAND over 10 years ago
- Status changed from Pending technical review to Pending release
- % Done changed from 0 to 100
Applied in changeset ldap-inventory:commit:8831d1b6259f421b1518219fd32c84b04ab45a72.
Updated by Nicolas CHARLES over 10 years ago
Applied in changeset ldap-inventory:commit:98888327c1fe91e16f23ec200a38825df794eade.
Updated by Vincent MEMBRÉ over 10 years ago
- Tracker changed from Bug to User story
- Subject changed from Do not write an LDIF dump of inventories when they are received to Keep only the LDIF dump of the inventory when we accept/refuse it, not every time we receive a new inventory
Updated by Vincent MEMBRÉ over 10 years ago
- Category changed from 39 to Performance and scalability
Updated by Vincent MEMBRÉ over 10 years ago
- Status changed from Pending release to Released
This bug has been fixed in Rudder 2.10.0~beta1, which was released today.
Check out:
The release announcement: http://www.rudder-project.org/pipermail/rudder-announce/2014-March/000084.html
The full ChangeLog: http://www.rudder-project.org/foswiki/bin/view/System/Documentation:ChangeLog210
Download information: https://www.rudder-project.org/site/get-rudder/downloads/