Project

General

Profile

Actions

Bug #16208

closed

On a loaded system, the compliance computation is fairly expensive

Added by Nicolas CHARLES over 4 years ago. Updated over 4 years ago.

Status:
Released
Priority:
N/A
Category:
Performance and scalability
Target version:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
To do
Fix check:
Checked
Regression:

Description

Receiving thousands of reports per second, with 4500 nodes and 321 directive leads to a costly compliance computation

timing_debug log in trace show

[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - unexpected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - expected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - prepare data: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - get missing reports: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - unexpected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - compute compliance : 1ms
[2019-11-18 22:22:24] DEBUG debug_timing - Compliance: mergeCompareByRule global cost : 296ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: getNodeStatusReports - computing compliance for node NodeId(28e55618-6886-45ed-8992-657a3022e448): 297ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: computing policy status for NodeId(28e55618-6886-45ed-8992-657a3022e448): 1ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: Compute complianceForRun map: 268ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - prepare data: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - get missing reports: 27ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - unexpected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - expected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - prepare data: 1ms

the Compliance: Compute complianceForRun map part is really expensive, as it iterate over each rules, each directive and each components, and manipulate a lot of collections, changing them and creating them

it's even more complex to debug, as each step in this map is 0ms ...

Actions #1

Updated by Nicolas CHARLES over 4 years ago

a first good step would be to compute the logs not for each step, but for the aggregation of each step within the map creation

Actions #2

Updated by Nicolas CHARLES over 4 years ago

created a unit test to effectively measure with a lot of data
Most of the time seems spend in checkExpectedComponentWithReports

Actions #3

Updated by Nicolas CHARLES over 4 years ago

slashing completely checkExpectedComponentWithReports for tests save 50% of the computation time

Actions #4

Updated by Nicolas CHARLES over 4 years ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES
Actions #5

Updated by Nicolas CHARLES over 4 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/2622
Actions #6

Updated by Nicolas CHARLES over 4 years ago

with these changes, mergeCompareByRule is about 30% faster, the final test going from 11500 ms to 8000ms

Actions #7

Updated by Nicolas CHARLES over 4 years ago

  • Status changed from Pending technical review to Pending release
Actions #8

Updated by Nicolas CHARLES over 4 years ago

  • Fix check changed from To do to Checked
Actions #9

Updated by Vincent MEMBRÉ over 4 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 5.0.15 which was released today.

Actions

Also available in: Atom PDF