Project

General

Profile

Actions

Bug #26611

open

invalidateComplianceRequest get saturated with UpdateCompliance messages

Added by François ARMAND 4 days ago. Updated 2 days ago.

Status:
New
Priority:
1 (highest)
Assignee:
-
Category:
Performance and scalability
Target version:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
To do
Fix check:
To do
Regression:
No

Description

The queue for updating compliance cache get saturated by `UpdateCompliance` and `ExpiredCompliance` messages.
This cause RAM exaustion, since `UpdateCompliance` contains a whole `NodeStatusReport`.

There is also an accumulation of `NodeStatusReport` with something updating postgresql, likely "lastCompliance" table (even if we're not sure).

The accumulation is massive, going to several hundred of thousands of update message for an instance with only 10k nodes, no change in configuration, and 10 reports received per second.

This leads to rudder crashing.

Actions #1

Updated by François ARMAND 2 days ago

  • Priority changed from N/A to 1 (highest)
Actions

Also available in: Atom PDF