Project

General

Profile

Actions

Bug #16773

closed

Batch of new nodes can overflow rudder server with inventories

Added by François ARMAND about 4 years ago. Updated over 3 years ago.

Status:
Released
Priority:
N/A
Category:
Performance and scalability
Target version:
Severity:
Major - prevents use of part of Rudder | no simple workaround
UX impact:
User visibility:
Operational - other Techniques | Rudder settings | Plugins
Effort required:
Priority:
45
Name check:
To do
Fix check:
Checked
Regression:

Description

We decided to make non accepted nodes send their inventories more often: https://issues.rudder.io/issues/9676 "An agent run with initial promises should send its inventory more often"

The unforseen effect of that decision is that if you had a bunch of nodes at the same time (in the hundreds), they start spamming Rudder server with inventories. And inventories will be rejected because the processing queue is full quite often.
If you are not lucky, it will always be the same node that will be processed.

We should add safeguards on the server side to reject inventories for new nodes that are already in the processing queue (and only new nodes, I believe).
We should also make nodes send their inventory more often only for one or two hours. Problems descibed in ticket #9676 don't matche the case of a node still not accepted after, say, 3 days.


Files

clipboard-202011132228-mpb2z.png (137 KB) clipboard-202011132228-mpb2z.png François ARMAND, 2020-11-13 22:28

Related issues 1 (0 open1 closed)

Related to Rudder - User story #9676: An agent run with initial promises should send its inventory more oftenReleasedNicolas CHARLESActions
Actions

Also available in: Atom PDF