Project

General

Profile

Actions
Copy link

Bug #19585

closed

Sometime inventory processing is not done when inventory is receveived

Added by Nicolas CHARLES over 2 years ago. Updated almost 2 years ago.

Status:
Released
Priority:
N/A
Assignee:
Nicolas CHARLES
Category:
Web - Nodes & inventories
Target version:
6.1.20
Pull Request:
https://github.com/Normation/rudder/p...
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
To do
Fix check:
Checked
Regression:

Description

it took a full 6 minutes for the webapp to identify that inventory was sitting in /var/rudder/inventories/accepted-nodes-updates/

Edit: so the problem is due to a lock not correctly removed in every case when there is several file processed: it seems to happen more frequently if both signature and inventory file are gzipped, and with sign first.

When that happens, the first file (gz) takes the lock, then the second (gz) can not be processed (because lock hold and gzip processing), then we get an other for the unlocked file (sign) which wait for inventory (because still gz), and the gz file is never processed and the lock is not removed.

Then, I'm not sure how, the next (or the next next) inventory processing round remove the lock and the next can be processed Ok (because there's lots of file unzipped on the fs, so things can go on).

Test correction

  • Take a pair of inventory/signature (for ex in received)
  • gzip ocs
  • cp inv.ocs.gz sig.sign /var/rudder/inventories/incoming
    Do it several times (4 of 5): it should always lead to a webapp log "new inventory .." "inventory parsed in XXX" and NOT the "inventory already processing" one.

Related issues 2 (0 open — 2 closed)

Related to Rudder - Architecture #19920: root inventory is missing and need to be resent after installReleasedNicolas CHARLESActions
Has duplicate Rudder - Bug #20994: Some inventories are stuck in the incoming folderResolvedActions
Actions
Copy link
 #1

Updated by François ARMAND over 2 years ago

  • Translation missing: en.field_tag_list set to Blocks 7.0-beta
Actions
Copy link
 #2

Updated by Nicolas CHARLES over 2 years ago

this is still the case

Actions
Copy link
 #3

Updated by François ARMAND over 2 years ago

  • Target version changed from 7.0.0~beta1 to 900

Do you have more information on that one? With so little, it's impossible for me to reproduce. All my simple tests show that inventories put in /var/rudder/inventories/incoming are instantly processed.

Actions
Copy link
 #4

Updated by Nicolas CHARLES over 2 years ago

it's when installing Rudder, with rtf. Nodes don"t show up in pending nodes

Actions
Copy link
 #5

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 900 to 7.0.0~beta2
Actions
Copy link
 #6

Updated by Nicolas CHARLES over 2 years ago

  • Related to Architecture #19920: root inventory is missing and need to be resent after install added
Actions
Copy link
 #7

Updated by Nicolas CHARLES over 2 years ago

  • Assignee set to Nicolas CHARLES
  • Target version changed from 7.0.0~beta2 to 6.2.11

as detailed in related issue, it's because the signature file is handled before, and sometimes it locks the non signature file

Actions
Copy link
 #8

Updated by Nicolas CHARLES over 2 years ago

  • Status changed from New to In progress
Actions
Copy link
 #9

Updated by Nicolas CHARLES over 2 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/3897
Actions
Copy link
 #10

Updated by Nicolas CHARLES over 2 years ago

  • Status changed from Pending technical review to In progress
Actions
Copy link
 #11

Updated by Nicolas CHARLES over 2 years ago

proposed PR does not work

Actions
Copy link
 #12

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.2.11 to 6.2.12
Actions
Copy link
 #13

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.2.12 to 6.2.13
Actions
Copy link
 #14

Updated by François ARMAND about 2 years ago

  • Tracker changed from Bug to Architecture
  • Subject changed from webapp is slow to detect inventories received in /var/rudder/inventories/accepted-nodes-updates/ to Sometime inventory processing is not done when inventory is receveived
  • Status changed from In progress to New
  • Target version changed from 6.2.13 to 7.1.0~beta1
  • Priority deleted (0)

The solution to that is to rework inventory incoming to make it more resilient to restart or delays in signature/main file reception

Actions
Copy link
 #15

Updated by Vincent MEMBRÉ about 2 years ago

  • Target version changed from 7.1.0~beta1 to 7.1.0~beta2
Actions
Copy link
 #16

Updated by Vincent MEMBRÉ about 2 years ago

  • Target version changed from 7.1.0~beta2 to 7.1.0~rc1
Actions
Copy link
 #17

Updated by Alexis Mousset about 2 years ago

  • Target version changed from 7.1.0~rc1 to 7.2.0~beta1
Actions
Copy link
 #18

Updated by François ARMAND almost 2 years ago

  • Tracker changed from Architecture to Bug
  • Target version changed from 7.2.0~beta1 to 6.1.20
  • Priority set to 0

So, it appears that there is two problems:

- one is that processing can be interrupted between the moment inventory is processing and file removed. It breaks. That needs a rework of the arch of inventory processing.
- but there is also a case where a lock is not correctly released when it should, leading to have wait some more inventory processing (generaly 2) to have it removed, and so that encompasses cases for "inventory stuck and not processed but hard to reproduce and goes away with touch" (because touching one file 2 times happen quickly when testing).

So: I propose to correct the lock condition with that ticket, and the rearch in #19920 .

Actions
Copy link
 #19

Updated by François ARMAND almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #20

Updated by François ARMAND almost 2 years ago

  • Status changed from New to In progress
Actions
Copy link
 #21

Updated by François ARMAND almost 2 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from François ARMAND to Nicolas CHARLES
  • Pull Request changed from https://github.com/Normation/rudder/pull/3897 to https://github.com/Normation/rudder/pull/4271
Actions
Copy link
 #22

Updated by François ARMAND almost 2 years ago

  • Status changed from Pending technical review to Pending release
Actions
Copy link
 #23

Updated by François ARMAND almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #24

Updated by François ARMAND almost 2 years ago

  • Description updated (diff)
  • Fix check changed from To do to Checked

I can confirm that this bug was reproducible with a nightly version before the patch, and not anymore afterward.

Actions
Copy link
 #25

Updated by Alexis Mousset almost 2 years ago

  • Translation missing: en.field_tag_list deleted (Blocks 7.0-beta)
Actions
Copy link
 #26

Updated by François ARMAND almost 2 years ago

  • Has duplicate Bug #20994: Some inventories are stuck in the incoming folder added
Actions
Copy link
 #27

Updated by Vincent MEMBRÉ almost 2 years ago

  • Status changed from Pending release to Released
Actions
Copy link

Also available in: Atom PDF