Project

General

Profile

Actions
Copy link

Bug #15012

closed

When receving pleinty of inventories at the same time, the web interface starts to parse them all at once

Added by Nicolas CHARLES over 5 years ago. Updated about 5 years ago.

Status:
Released
Priority:
N/A
Assignee:
Nicolas CHARLES
Category:
Web - Nodes & inventories
Target version:
6.0.0~beta1
Pull Request:
https://github.com/Normation/rudder/p...
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
Fix check:
Regression:

Description

I've put around 350 inventories in incoming
Result, I have 350 lines

[2019-06-01 22:30:38] DEBUG inventory-processing - Start parsing inventory 'RDR-LOAD2-RandHost275.rudder.local275.ocs'
[2019-06-01 22:30:38] DEBUG inventory-processing - Start parsing inventory 'RDR-LOAD2-RandHost276.rudder.local276.ocs'
[2019-06-01 22:30:38] DEBUG inventory-processing - Start parsing inventory 'RDR-LOAD2-RandHost274.rudder.local274.ocs'
[2019-06-01 22:30:38] DEBUG inventory-processing - Start parsing inventory 'RDR-LOAD2-RandHost273.rudder.local273.ocs'
[2019-06-01 22:30:38] DEBUG inventory-processing - Start parsing inventory 'RDR-LOAD2-RandHost272.rudder.local272.ocs'
[2019-06-01 22:30:38] DEBUG inventory-processing - Start parsing inventory 'RDR-LOAD2-RandHost270.rudder.local270.ocs'

A full GC

then 

[2019-06-01 22:31:15] DEBUG inventory-processing - Inventory 'RDR-LOAD2-RandHost349.rudder.local349.ocs' parsed in 170 milliseconds ms, now saving
[2019-06-01 22:31:15] DEBUG inventory-processing - Inventory 'RDR-LOAD2-RandHost310.rudder.local310.ocs' parsed in 0 milliseconds ms, now saving
[2019-06-01 22:31:15] DEBUG inventory-processing - Inventory 'RDR-LOAD2-RandHost294.rudder.local294.ocs' parsed in 0 milliseconds ms, now saving
[2019-06-01 22:31:15] DEBUG inventory-processing - Inventory 'RDR-LOAD2-RandHost27.rudder.local27.ocs' parsed in 0 milliseconds ms, now saving
[2019-06-01 22:31:15] DEBUG inventory-processing - Inventory 'RDR-LOAD2-RandHost268.rudder.local268.ocs' parsed in 0 milliseconds ms, now saving

(not all files were parsed)

[2019-06-01 22:31:17] DEBUG inventory-processing - Inventory 'RDR-LOAD2-RandHost368.rudder.local368.ocs' for node '19ab73c0-214a-423a-94f6-a2f86dff3d56' pre-processed in 39 seconds and 466 milliseconds ms
[2019-06-01 22:31:17] WARN  inventory-processing - Rejecting Inventory 'RDR-LOAD2-RandHost368.rudder.local368.ocs' for Node '19ab73c0-214a-423a-94f6-a2f86dff3d56' because processing queue is full.
[2019-06-01 22:31:17] WARN  inventory-processing - Rejecting Inventory 'RDR-LOAD2-RandHost48.rudder.local48.ocs' for Node '8b66ada8-49b5-491e-a7fd-65ee3118db5a' because processing queue is full.
[2019-06-01 22:31:17] DEBUG inventory-processing - Inventory 'RDR-LOAD2-RandHost79.rudder.local79.ocs' for node 'aa2a4e0e-79c0-49b4-aa47-48c23bbd6fcd' pre-processed in 31 seconds and 647 milliseconds ms
[2019-06-01 22:31:17] WARN  inventory-processing - Rejecting Inventory 'RDR-LOAD2-RandHost79.rudder.local79.ocs' for Node 'aa2a4e0e-79c0-49b4-aa47-48c23bbd6fcd' because processing queue is full.
[2019-06-01 22:31:17] DEBUG inventory-processing - Inventory 'RDR-LOAD2-RandHost164.rudder.local164.ocs' for node '24089e7c-f7b2-4ea3-b330-84475afffe0f' pre-processed in 40 seconds and 850 milliseconds ms
[2019-06-01 22:31:17] WARN  inventory-processing - Rejecting Inventory 'RDR-LOAD2-RandHost164.rudder.local164.ocs' for Node '24089e7c-f7b2-4ea3-b330-84475afffe0f' because processing queue is full.
[2019-06-01 22:31:17] DEBUG inventory-processing - Inventory 'RDR-LOAD2-RandHost339.rudder.local339.ocs' for node '1acbaf70-5be7-4d4a-8889-e37c23d328b3' pre-processed in 39 seconds and 461 milliseconds ms

the parsing should probably be less agressive

note: those who were rejected because a a full queue went to failed

Actions
Copy link

Also available in: Atom PDF