Bug #12937
Updated by François ARMAND over 3 years ago
After a time, inventory processing on a rudder which manage debian system get *extremelly* slow - up to several *minutes* by inventory, versus the expected <1s duration. This cause chain problem with the accumulation of to-be-processed inventories in @/var/rudder/inventories/incoming@ and @/var/rudder/inventories/accepted-nodes-updates@.
The main observable consequence is new nodes taking several hours to appear in "pending node" in UI.
Analysis:
This was an old ticket which got a brand new hype in 6.2. The underlying root cause was that the number of software was growing linearly with the number of debian-like inventory processed (because of bug on a check condition on the "source name" and "source version" attribute, only present on debian-like since 6.2).
After a time, the huge number of software (in the million range for a middle-sized infra of just 500 linux, vs ~50k in normal operationnal condition, so in the range of a 10k node installation, without the corresponding sized root server) was exacerbing the `merge_uuid` action for them.
The PR correcting that ticket also add a new API endpoint allowing to clean-up unused software entries (see doc API for @system/maintenance/purgeSoftware@). The purge is triggered every night, but you might want to trigger it after install rudder 6.2.1 to start diminishing the number of garbage software as soon as possible.
History:
With Rudder 4.1, some inventories are painfully slow to process
Checking the logs, in trace, it shows
<pre>
[2018-07-11 16:50:35] DEBUG com.normation.inventory.services.provisioning.NodeInventoryDNFinderService - Server Id 'NodeId(YYYYYYY-YYYYYYY-YYYYYYY-YYYYYYY-YYYYYYY)' found in DIT 'AcceptedInventory' with id finder 'use_existing_id'
[2018-07-11 16:50:35] DEBUG com.normation.inventory.services.provisioning.MachineDNFinderService - Processing machine id finder use_existing_id
[2018-07-11 16:50:35] DEBUG com.normation.inventory.services.provisioning.MachineDNFinderService - Processing machine id finder check_mother_board_uuid_accepted
[2018-07-11 16:50:35] DEBUG com.normation.inventory.services.provisioning.MachineDNFinderService - Machine Id 'MachineUuid(XXXXX-XXXXX-XXXXX-XXXXX-XXXXX)' found with id finder 'check_mother_board_uuid_accepted'
[2018-07-11 16:50:35] TRACE com.normation.inventory.ldap.provisioning.DefaultReportSaver - Precommit 'pre_commit_inventory:merge_uuid': 30645 ms
[2018-07-11 16:50:35] TRACE com.normation.inventory.ldap.provisioning.DefaultReportSaver - Precommit 'pre_commit_inventory:check_machine_cn': 0 ms
[2018-07-11 16:50:35] TRACE com.normation.inventory.ldap.provisioning.DefaultReportSaver - Precommit 'pre_commit_inventory:set_last_inventory_date': 0 ms
[2018-07-11 16:50:35] TRACE com.normation.inventory.ldap.provisioning.DefaultReportSaver - Precommit 'pre_commit_inventory:add_ip_values': 0 ms
[2018-07-11 16:50:35] TRACE com.normation.inventory.ldap.provisioning.DefaultReportSaver - Precommit 'pre_commit_inventory:log_inventory': 0 ms
[2018-07-11 16:50:35] TRACE com.normation.inventory.ldap.provisioning.DefaultReportSaver - Pre commit report: 30647 ms
</pre>
There is no reason for such check to be so slow.
h3. WORKAROUND
It may helps to delete unreferenced software, and perhaps the batch that should do that is in error (you can @grep unreferenced /var/log/rudder/webapp/*@ to see if there is errors or a lot of them)
You can do it manually:
So, first, MAKE A BACKUP OF YOUR LDAP: https://docs.rudder.io/reference/6.2/administration/procedures.html#_migration_backups_and_restores
Then, create de directory for work, and go in it. Then:
- create a directory for work, and go in it. Then:
- all softwares:
<pre>
ldapsearch -LLL -o ldif-wrap=no -h localhost -p 389 -x -D "cn=Manager,cn=rudder-configuration" -w LDAP_PASS_FROM_rudder-passwords_file -b "ou=Software,ou=Inventories,cn=rudder-configuration" -s one 1.1 | sort | uniq | cut -d: -f2 > all-soft-sorted.dns
</pre>
- softwares used in accepted nodes:
<pre>
ldapsearch -LLL -o ldif-wrap=no -h localhost -p 389 -x -D "cn=Manager,cn=rudder-configuration" -w LDAP_PASS -b "ou=Nodes,ou=Accepted Inventories,ou=Inventories,cn=rudder-configuration" -s one 'nodeId=*' software | grep softwareId | sort | uniq | cut -d: -f2 > nodes-soft-sorted.dns
</pre>
- software to delete (ie all - nodes):
<pre>
grep -f nodes-soft-sorted.dns -v all-soft-sorted.dns > soft-to-delete.dns
</pre>
- then delete unreferenced:
<pre>
ldapdelete -h localhost -p 389 -x -D "cn=Manager,cn=rudder-configuration" -w LDAP_PASS -c -f soft-to-delete.dns
</pre>