Project

General

Profile

Bug #16208

On a loaded system, the compliance computation is fairly expensive

Added by Nicolas CHARLES 7 months ago. Updated 6 months ago.

Status:
Released
Priority:
N/A
Category:
Performance and scalability
Target version:
Severity:
User visibility:
Effort required:
Priority:
0

Description

Receiving thousands of reports per second, with 4500 nodes and 321 directive leads to a costly compliance computation

timing_debug log in trace show

[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - unexpected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - expected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - prepare data: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - get missing reports: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - unexpected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - compute compliance : 1ms
[2019-11-18 22:22:24] DEBUG debug_timing - Compliance: mergeCompareByRule global cost : 296ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: getNodeStatusReports - computing compliance for node NodeId(28e55618-6886-45ed-8992-657a3022e448): 297ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: computing policy status for NodeId(28e55618-6886-45ed-8992-657a3022e448): 1ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: Compute complianceForRun map: 268ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - prepare data: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - get missing reports: 27ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - unexpected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - expected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - prepare data: 1ms

the Compliance: Compute complianceForRun map part is really expensive, as it iterate over each rules, each directive and each components, and manipulate a lot of collections, changing them and creating them

it's even more complex to debug, as each step in this map is 0ms ...

#1

Updated by Nicolas CHARLES 7 months ago

a first good step would be to compute the logs not for each step, but for the aggregation of each step within the map creation

#2

Updated by Nicolas CHARLES 6 months ago

created a unit test to effectively measure with a lot of data
Most of the time seems spend in checkExpectedComponentWithReports

#3

Updated by Nicolas CHARLES 6 months ago

slashing completely checkExpectedComponentWithReports for tests save 50% of the computation time

#4

Updated by Nicolas CHARLES 6 months ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES
#5

Updated by Nicolas CHARLES 6 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/2622
#6

Updated by Nicolas CHARLES 6 months ago

with these changes, mergeCompareByRule is about 30% faster, the final test going from 11500 ms to 8000ms

#7

Updated by Nicolas CHARLES 6 months ago

  • Status changed from Pending technical review to Pending release
#9

Updated by Vincent MEMBRÉ 6 months ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 5.0.15 which was released today.

Also available in: Atom PDF