Project

General

Profile

Bug #16083

Big memory usage when fetching/writing node configuration and expected reports

Added by Nicolas CHARLES 3 months ago. Updated about 2 months ago.

Status:
Released
Priority:
N/A
Category:
Performance and scalability
Target version:
Severity:
Major - prevents use of part of Rudder | no simple workaround
User visibility:
Infrequent - complex configurations | third party integrations
Effort required:
Priority:
41

Description

On large installation (10k+), the memory usage when writing expected reports is absurdly high; same when fetching data
We should batch these queries, because it means we are taking everything, serializing them all, and putting all theses serialization in big ldap/sql query, so we are effectively tripling (+ side effect) the memory usage at these operation

Hypothesis: batching by 200 to 500 would be sweet spot because it keeps the memory at bay, while still transfering enough data
To test this hypothesis, we need to make a parameter that can be changed (either by api, or by configuration parameter) and test several scenarios

Note: version prior 4.2 where not impacted, because we didn't manage data the same way


Subtasks

Bug #16228: Fetching nodes last run is not batchedReleasedFrançois ARMANDActions

Associated revisions

Revision bc70f083 (diff)
Added by Nicolas CHARLES 2 months ago

Fixes #16083: Big memory usage when fetching/writing ndoe configuration and expected reports

History

#1

Updated by Nicolas CHARLES 3 months ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES
#2

Updated by Nicolas CHARLES 3 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/2569
#3

Updated by Nicolas CHARLES 2 months ago

  • Status changed from Pending technical review to Pending release
#4

Updated by Alexis MOUSSET 2 months ago

  • Subject changed from Big memory usage when fetching/writing ndoe configuration and expected reports to Big memory usage when fetching/writing node configuration and expected reports
#7

Updated by Vincent MEMBRÉ about 2 months ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 5.0.15 which was released today.

Also available in: Atom PDF