Project

General

Profile

Actions
Copy link

Bug #26611

open

invalidateComplianceRequest get saturated with UpdateCompliance messages

Added by François ARMAND 25 days ago. Updated 11 days ago.

Status:
New
Priority:
1 (highest)
Assignee:
-
Category:
Performance and scalability
Target version:
8.2.6
Pull Request:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
To do
Fix check:
To do
Regression:
No

Description

The queue for updating compliance cache get saturated by `UpdateCompliance` and `ExpiredCompliance` messages.
This cause RAM exaustion, since `UpdateCompliance` contains a whole `NodeStatusReport`.

There is also an accumulation of `NodeStatusReport` with something updating postgresql, likely "lastCompliance" table (even if we're not sure).

The accumulation is massive, going to several hundred of thousands of update message for an instance with only 10k nodes, no change in configuration, and 10 reports received per second.

This leads to rudder crashing.

Subtasks 5 (5 open — 0 closed)

Bug #26683: Don't check for expiration nodes that just got new reportsPending releaseVincent MEMBRÉActions
Bug #26692: Remove semaphore and use a sliding queue for compliance invalidationPending releaseVincent MEMBRÉActions
Bug #26725: Reverting change in queue since it leads to CPU over usePending releaseVincent MEMBRÉActions
Bug #26729: Remove semaphore and use a sliding queue for compliance invalidation (blocking version)Pending technical reviewVincent MEMBRÉActions
Bug #26703: Make AggregatedStatusReport correct wrt equalsPending technical reviewVincent MEMBRÉActions
Actions
Copy link
 #1

Updated by François ARMAND 23 days ago

  • Priority changed from N/A to 1 (highest)
Actions
Copy link
 #2

Updated by François ARMAND 15 days ago

  • Subtask #26683 added
Actions
Copy link
 #3

Updated by François ARMAND 14 days ago

  • Subtask #26692 added
Actions
Copy link
 #4

Updated by François ARMAND 11 days ago

  • Subtask #26703 added
Actions
Copy link

Also available in: Atom PDF