Bug #19585
closed
Sometime inventory processing is not done when inventory is receveived
Description
it took a full 6 minutes for the webapp to identify that inventory was sitting in /var/rudder/inventories/accepted-nodes-updates/
Edit: so the problem is due to a lock not correctly removed in every case when there is several file processed: it seems to happen more frequently if both signature and inventory file are gzipped, and with sign first.
When that happens, the first file (gz) takes the lock, then the second (gz) can not be processed (because lock hold and gzip processing), then we get an other for the unlocked file (sign) which wait for inventory (because still gz), and the gz file is never processed and the lock is not removed.
Then, I'm not sure how, the next (or the next next) inventory processing round remove the lock and the next can be processed Ok (because there's lots of file unzipped on the fs, so things can go on).
Test correction
- Take a pair of inventory/signature (for ex in received)
- gzip ocs
- cp inv.ocs.gz sig.sign /var/rudder/inventories/incoming
Do it several times (4 of 5): it should always lead to a webapp log "new inventory .." "inventory parsed in XXX" and NOT the "inventory already processing" one.
Updated by François ARMAND over 3 years ago
- Translation missing: en.field_tag_list set to Blocks 7.0-beta
Updated by François ARMAND about 3 years ago
- Target version changed from 7.0.0~beta1 to 900
Do you have more information on that one? With so little, it's impossible for me to reproduce. All my simple tests show that inventories put in /var/rudder/inventories/incoming are instantly processed.
Updated by Nicolas CHARLES about 3 years ago
it's when installing Rudder, with rtf. Nodes don"t show up in pending nodes
Updated by Vincent MEMBRÉ about 3 years ago
- Target version changed from 900 to 7.0.0~beta2
Updated by Nicolas CHARLES about 3 years ago
- Related to Architecture #19920: root inventory is missing and need to be resent after install added
Updated by Nicolas CHARLES about 3 years ago
- Assignee set to Nicolas CHARLES
- Target version changed from 7.0.0~beta2 to 6.2.11
as detailed in related issue, it's because the signature file is handled before, and sometimes it locks the non signature file
Updated by Nicolas CHARLES about 3 years ago
- Status changed from New to In progress
Updated by Nicolas CHARLES about 3 years ago
- Status changed from In progress to Pending technical review
- Assignee changed from Nicolas CHARLES to François ARMAND
- Pull Request set to https://github.com/Normation/rudder/pull/3897
Updated by Nicolas CHARLES about 3 years ago
- Status changed from Pending technical review to In progress
Updated by Vincent MEMBRÉ about 3 years ago
- Target version changed from 6.2.11 to 6.2.12
Updated by Vincent MEMBRÉ almost 3 years ago
- Target version changed from 6.2.12 to 6.2.13
Updated by François ARMAND almost 3 years ago
- Tracker changed from Bug to Architecture
- Subject changed from webapp is slow to detect inventories received in /var/rudder/inventories/accepted-nodes-updates/ to Sometime inventory processing is not done when inventory is receveived
- Status changed from In progress to New
- Target version changed from 6.2.13 to 7.1.0~beta1
- Priority deleted (
0)
The solution to that is to rework inventory incoming to make it more resilient to restart or delays in signature/main file reception
Updated by Vincent MEMBRÉ over 2 years ago
- Target version changed from 7.1.0~beta1 to 7.1.0~beta2
Updated by Vincent MEMBRÉ over 2 years ago
- Target version changed from 7.1.0~beta2 to 7.1.0~rc1
Updated by Alexis Mousset over 2 years ago
- Target version changed from 7.1.0~rc1 to 7.2.0~beta1
Updated by François ARMAND over 2 years ago
- Tracker changed from Architecture to Bug
- Target version changed from 7.2.0~beta1 to 6.1.20
- Priority set to 0
So, it appears that there is two problems:
- one is that processing can be interrupted between the moment inventory is processing and file removed. It breaks. That needs a rework of the arch of inventory processing.
- but there is also a case where a lock is not correctly released when it should, leading to have wait some more inventory processing (generaly 2) to have it removed, and so that encompasses cases for "inventory stuck and not processed but hard to reproduce and goes away with touch" (because touching one file 2 times happen quickly when testing).
So: I propose to correct the lock condition with that ticket, and the rearch in #19920 .
Updated by François ARMAND over 2 years ago
- Status changed from New to In progress
Updated by François ARMAND over 2 years ago
- Status changed from In progress to Pending technical review
- Assignee changed from François ARMAND to Nicolas CHARLES
- Pull Request changed from https://github.com/Normation/rudder/pull/3897 to https://github.com/Normation/rudder/pull/4271
Updated by François ARMAND over 2 years ago
- Status changed from Pending technical review to Pending release
Applied in changeset rudder|ec2489c408ef6b076aa86ee4a8b6d9bb2ecd8121.
Updated by François ARMAND over 2 years ago
- Description updated (diff)
- Fix check changed from To do to Checked
I can confirm that this bug was reproducible with a nightly version before the patch, and not anymore afterward.
Updated by Alexis Mousset over 2 years ago
- Translation missing: en.field_tag_list deleted (
Blocks 7.0-beta)
Updated by François ARMAND over 2 years ago
- Has duplicate Bug #20994: Some inventories are stuck in the incoming folder added
Updated by Vincent MEMBRÉ over 2 years ago
- Status changed from Pending release to Released