Bug #19585
closed
Sometime inventory processing is not done when inventory is receveived
Added by Nicolas CHARLES almost 3 years ago.
Updated almost 2 years ago.
Category:
Web - Nodes & inventories
Description
it took a full 6 minutes for the webapp to identify that inventory was sitting in /var/rudder/inventories/accepted-nodes-updates/
Edit: so the problem is due to a lock not correctly removed in every case when there is several file processed: it seems to happen more frequently if both signature and inventory file are gzipped, and with sign first.
When that happens, the first file (gz) takes the lock, then the second (gz) can not be processed (because lock hold and gzip processing), then we get an other for the unlocked file (sign) which wait for inventory (because still gz), and the gz file is never processed and the lock is not removed.
Then, I'm not sure how, the next (or the next next) inventory processing round remove the lock and the next can be processed Ok (because there's lots of file unzipped on the fs, so things can go on).
Test correction
- Take a pair of inventory/signature (for ex in received)
- gzip ocs
- cp inv.ocs.gz sig.sign /var/rudder/inventories/incoming
Do it several times (4 of 5): it should always lead to a webapp log "new inventory .." "inventory parsed in XXX" and NOT the "inventory already processing" one.
- Translation missing: en.field_tag_list set to Blocks 7.0-beta
- Target version changed from 7.0.0~beta1 to 900
Do you have more information on that one? With so little, it's impossible for me to reproduce. All my simple tests show that inventories put in /var/rudder/inventories/incoming are instantly processed.
it's when installing Rudder, with rtf. Nodes don"t show up in pending nodes
- Target version changed from 900 to 7.0.0~beta2
- Related to Architecture #19920: root inventory is missing and need to be resent after install added
- Assignee set to Nicolas CHARLES
- Target version changed from 7.0.0~beta2 to 6.2.11
as detailed in related issue, it's because the signature file is handled before, and sometimes it locks the non signature file
- Status changed from New to In progress
- Status changed from In progress to Pending technical review
- Assignee changed from Nicolas CHARLES to François ARMAND
- Pull Request set to https://github.com/Normation/rudder/pull/3897
- Status changed from Pending technical review to In progress
proposed PR does not work
- Target version changed from 6.2.11 to 6.2.12
- Target version changed from 6.2.12 to 6.2.13
- Tracker changed from Bug to Architecture
- Subject changed from webapp is slow to detect inventories received in /var/rudder/inventories/accepted-nodes-updates/ to Sometime inventory processing is not done when inventory is receveived
- Status changed from In progress to New
- Target version changed from 6.2.13 to 7.1.0~beta1
- Priority deleted (
0)
The solution to that is to rework inventory incoming to make it more resilient to restart or delays in signature/main file reception
- Target version changed from 7.1.0~beta1 to 7.1.0~beta2
- Target version changed from 7.1.0~beta2 to 7.1.0~rc1
- Target version changed from 7.1.0~rc1 to 7.2.0~beta1
- Tracker changed from Architecture to Bug
- Target version changed from 7.2.0~beta1 to 6.1.20
- Priority set to 0
So, it appears that there is two problems:
- one is that processing can be interrupted between the moment inventory is processing and file removed. It breaks. That needs a rework of the arch of inventory processing.
- but there is also a case where a lock is not correctly released when it should, leading to have wait some more inventory processing (generaly 2) to have it removed, and so that encompasses cases for "inventory stuck and not processed but hard to reproduce and goes away with touch" (because touching one file 2 times happen quickly when testing).
So: I propose to correct the lock condition with that ticket, and the rearch in #19920 .
- Description updated (diff)
- Status changed from New to In progress
- Status changed from In progress to Pending technical review
- Assignee changed from François ARMAND to Nicolas CHARLES
- Pull Request changed from https://github.com/Normation/rudder/pull/3897 to https://github.com/Normation/rudder/pull/4271
- Status changed from Pending technical review to Pending release
- Description updated (diff)
- Description updated (diff)
- Fix check changed from To do to Checked
I can confirm that this bug was reproducible with a nightly version before the patch, and not anymore afterward.
- Translation missing: en.field_tag_list deleted (
Blocks 7.0-beta)
- Has duplicate Bug #20994: Some inventories are stuck in the incoming folder added
- Status changed from Pending release to Released
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by 
Updated by