Project

General

Profile

Actions
Copy link

Bug #24652

closed

Rudder 8.1 slows down over time

Added by Nicolas CHARLES 8 months ago. Updated 5 months ago.

Status:
Released
Priority:
N/A
Assignee:
Nicolas CHARLES
Category:
Performance and scalability
Target version:
8.1.5
Pull Request:
https://github.com/Normation/rudder/p...
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
To do
Fix check:
Checked
Regression:
No

Description

2024_03_16.stderrout.log.gz:2024-03-16 06:28:42+0000 DEBUG dynamic-group.timing - Computing dynamic groups without dependencies finished in 3893943 ms
2024_03_16.stderrout.log.gz:2024-03-16 07:32:03+0000 DEBUG dynamic-group.timing - Computing dynamic groups without dependencies finished in 3601104 ms
2024_03_16.stderrout.log.gz:2024-03-16 08:02:09+0000 DEBUG dynamic-group.timing - Computing dynamic groups without dependencies finished in 720472 ms
2024_03_16.stderrout.log.gz:2024-03-16 10:00:48+0000 DEBUG dynamic-group.timing - Computing dynamic groups without dependencies finished in 5868962 ms
2024_03_16.stderrout.log.gz:2024-03-16 12:01:45+0000 DEBUG dynamic-group.timing - Computing dynamic groups without dependencies finished in 6048786 ms
2024_03_16.stderrout.log.gz:2024-03-16 14:02:46+0000 DEBUG dynamic-group.timing - Computing dynamic groups without dependencies finished in 6057161 ms
2024_03_16.stderrout.log.gz:2024-03-16 17:49:57+0000 DEBUG dynamic-group.timing - Computing dynamic groups without dependencies finished in 7617380 ms
2024_03_16.stderrout.log.gz:2024-03-16 20:03:33+0000 DEBUG dynamic-group.timing - Computing dynamic groups without dependencies finished in 7807798 ms

There are objects ( com.normation.rudder.services.reports.CacheComplianceQueueAction$ExpiredCompliance ) piling up in the heap dump, which might be related

Files

Download all files
clipboard-202404181612-sdcnh.png (86.3 KB) clipboard-202404181612-sdcnh.png François ARMAND, 2024-04-18 16:12
clipboard-202404181623-dsmww.png (245 KB) clipboard-202404181623-dsmww.png François ARMAND, 2024-04-18 16:23

Related issues 2 (0 open — 2 closed)

Related to Rudder - Bug #24712: ExpiredCompliance events are pilling upReleasedNicolas CHARLESActions
Related to Rudder - Bug #24713: Dynamic groups are slow to compute in Rudder 8.1ReleasedVincent MEMBRÉActions
Actions
Copy link
 #1

Updated by Nicolas CHARLES 8 months ago · Edited

  • Assignee set to Nicolas CHARLES

There are roughly 1 million of com.normation.rudder.services.reports.CacheComplianceQueueAction$ExpiredCompliance piling up in 30 minutes

Actions
Copy link
 #2

Updated by Vincent MEMBRÉ 8 months ago

  • Target version changed from 8.1.0~rc1 to 8.1.0
Actions
Copy link
 #3

Updated by Vincent MEMBRÉ 7 months ago

  • Target version changed from 8.1.0 to 8.1.1
Actions
Copy link
 #4

Updated by François ARMAND 7 months ago

Creating an other ticket for the ExpiredCompliance which are pilling up, it might not be the root cause but something else to investigate

Actions
Copy link
 #5

Updated by François ARMAND 7 months ago

  • Related to Bug #24712: ExpiredCompliance events are pilling up added
Actions
Copy link
 #6

Updated by François ARMAND 7 months ago

  • Related to Bug #24713: Dynamic groups are slow to compute in Rudder 8.1 added
Actions
Copy link Download all files
 #7

Updated by François ARMAND 7 months ago

We will need to see if the two linked tickets are enought to remove the slowdown over time.

After 24h on hour test machine, it seems to be ok with memory correctly reclamed when needed.
There is a lot of CPU spikes, they correlate with generation due to system-update campaign starting/ending.

Actions
Copy link
 #8

Updated by Vincent MEMBRÉ 7 months ago

  • Target version changed from 8.1.1 to 8.1.2
Actions
Copy link
 #9

Updated by Vincent MEMBRÉ 7 months ago

  • Target version changed from 8.1.2 to 8.1.3
Actions
Copy link
 #10

Updated by Vincent MEMBRÉ 6 months ago

  • Target version changed from 8.1.3 to 8.1.4
Actions
Copy link
 #11

Updated by François ARMAND 5 months ago

Perhaps we have a track to explore: the node score `handleEvent` that does a database insert is done in the `performAction` linked to the `invalidateComplianceReport` queue.

I'm going to try to move it in `fetchRunAndCompliance` around the place where `ComplianceRepository.saveRunCompliance` is done, which is known to be slow.

Actions
Copy link
 #12

Updated by François ARMAND 5 months ago

  • Status changed from New to In progress
  • Assignee changed from Nicolas CHARLES to François ARMAND
Actions
Copy link
 #13

Updated by François ARMAND 5 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from François ARMAND to Clark ANDRIANASOLO
  • Pull Request set to https://github.com/Normation/rudder/pull/5737
Actions
Copy link
 #14

Updated by Vincent MEMBRÉ 5 months ago

  • Target version changed from 8.1.4 to 8.1.5
Actions
Copy link
 #15

Updated by François ARMAND 5 months ago

  • Assignee changed from Clark ANDRIANASOLO to Nicolas CHARLES
Actions
Copy link
 #16

Updated by Anonymous 5 months ago

  • Status changed from Pending technical review to Pending release
Actions
Copy link
 #17

Updated by Vincent MEMBRÉ 5 months ago

  • Fix check changed from To do to Checked
Actions
Copy link
 #18

Updated by Vincent MEMBRÉ 5 months ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 8.1.5 which was released today.

Actions
Copy link

Also available in: Atom PDF