Project

General

Profile

Actions
Copy link

Bug #26611

closed

invalidateComplianceRequest get saturated with UpdateCompliance messages

Added by François ARMAND about 2 months ago. Updated 2 days ago.

Status:
Released
Priority:
1 (highest)
Assignee:
-
Category:
Performance and scalability
Target version:
8.2.6
Pull Request:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
To do
Fix check:
Checked
Regression:
No

Description

The queue for updating compliance cache get saturated by `UpdateCompliance` and `ExpiredCompliance` messages.
This cause RAM exaustion, since `UpdateCompliance` contains a whole `NodeStatusReport`.

There is also an accumulation of `NodeStatusReport` with something updating postgresql, likely "lastCompliance" table (even if we're not sure).

The accumulation is massive, going to several hundred of thousands of update message for an instance with only 10k nodes, no change in configuration, and 10 reports received per second.

This leads to rudder crashing.

Subtasks 5 (0 open — 5 closed)

Bug #26683: Don't check for expiration nodes that just got new reportsReleasedVincent MEMBRÉActions
Bug #26692: Remove semaphore and use a sliding queue for compliance invalidationReleasedVincent MEMBRÉActions
Bug #26725: Reverting change in queue since it leads to CPU over useReleasedVincent MEMBRÉActions
Bug #26729: Remove semaphore and use a sliding queue for compliance invalidation (blocking version)ReleasedVincent MEMBRÉActions
Bug #26703: Make AggregatedStatusReport correct wrt equalsReleasedVincent MEMBRÉActions
Actions
Copy link

Also available in: Atom PDF