Project

General

Profile

Bug #16208

On a loaded system, the compliance computation is fairly expensive

Added by Nicolas CHARLES 2 months ago. Updated about 2 months ago.

Status:
Released
Priority:
N/A
Category:
Performance and scalability
Target version:
Severity:
User visibility:
Effort required:
Priority:
0

Description

Receiving thousands of reports per second, with 4500 nodes and 321 directive leads to a costly compliance computation

timing_debug log in trace show

[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - unexpected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - expected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - prepare data: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - get missing reports: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - unexpected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - compute compliance : 1ms
[2019-11-18 22:22:24] DEBUG debug_timing - Compliance: mergeCompareByRule global cost : 296ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: getNodeStatusReports - computing compliance for node NodeId(28e55618-6886-45ed-8992-657a3022e448): 297ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: computing policy status for NodeId(28e55618-6886-45ed-8992-657a3022e448): 1ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: Compute complianceForRun map: 268ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - prepare data: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - get missing reports: 27ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - unexpected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - expected directives computation: 0ms
[2019-11-18 22:22:24] TRACE debug_timing - Compliance: mergeCompareByRule - prepare data: 1ms

the Compliance: Compute complianceForRun map part is really expensive, as it iterate over each rules, each directive and each components, and manipulate a lot of collections, changing them and creating them

it's even more complex to debug, as each step in this map is 0ms ...

Associated revisions

Revision de994cd6 (diff)
Added by Nicolas CHARLES 2 months ago

Fixes #16208: On a loaded system, the compliance computation is fairly expensive

History

#1

Updated by Nicolas CHARLES 2 months ago

a first good step would be to compute the logs not for each step, but for the aggregation of each step within the map creation

#2

Updated by Nicolas CHARLES 2 months ago

created a unit test to effectively measure with a lot of data
Most of the time seems spend in checkExpectedComponentWithReports

#3

Updated by Nicolas CHARLES 2 months ago

slashing completely checkExpectedComponentWithReports for tests save 50% of the computation time

#4

Updated by Nicolas CHARLES 2 months ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES
#5

Updated by Nicolas CHARLES 2 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/2622
#6

Updated by Nicolas CHARLES 2 months ago

with these changes, mergeCompareByRule is about 30% faster, the final test going from 11500 ms to 8000ms

#7

Updated by Nicolas CHARLES 2 months ago

  • Status changed from Pending technical review to Pending release
#9

Updated by Vincent MEMBRÉ about 2 months ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 5.0.15 which was released today.

Also available in: Atom PDF