Project

General

Profile

Actions
Copy link

Bug #16228

closed

Bug #16083: Big memory usage when fetching/writing node configuration and expected reports

Fetching nodes last run is not batched

Added by Nicolas CHARLES about 5 years ago. Updated almost 5 years ago.

Status:
Released
Priority:
N/A
Assignee:
François ARMAND
Category:
Performance and scalability
Target version:
5.0.15
Pull Request:
https://github.com/Normation/rudder/p...
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
Reviewed
Fix check:
Checked
Regression:

Description

we need to do the same as the parent ticket for getNodesLastRun
Query time is 25s with 5500 nodes, which is clearly too much (and too large)

Actions
Copy link
 #1

Updated by Nicolas CHARLES about 5 years ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES
Actions
Copy link
 #2

Updated by Nicolas CHARLES about 5 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/2623
Actions
Copy link
 #3

Updated by Nicolas CHARLES almost 5 years ago

ok, the query is really inefficient

it's much more efficient to do

select r.nodeid, r.date, r.nodeconfigid, r.complete, r.insertionid,
                         c.nodeid, c.nodeconfigid, c.begindate, c.enddate, c.configuration from
(select reportsexecution.nodeid, reportsexecution.date, reportsexecution.nodeconfigid, reportsexecution.complete, reportsexecution.insertionid from
reportsexecution where nodeid in  (values ('cfc8fe3d-09e6-4680-aea9-e347385753a2'),('540bc906-c672-49ce-acab-240d6e65676b'),('9b295838-6e8e-481b-927d-1e1891c317b2')) and (nodeid, insertionid) in
  (select nodeid, max(insertionid) as insertionid
       from reportsexecution
       where complete = true
       GROUP BY nodeid)) as r
left outer join nodeconfigurations as c
                         on r.nodeId = c.nodeid and r.nodeconfigid = c.nodeconfigid;

or 
select r.nodeid, r.date, r.nodeconfigid, r.complete, r.insertionid,
                         c.nodeid, c.nodeconfigid, c.begindate, c.enddate, c.configuration from
(select reportsexecution.nodeid, reportsexecution.date, reportsexecution.nodeconfigid, reportsexecution.complete, reportsexecution.insertionid from
reportsexecution where (nodeid, insertionid) in
  (select nodeid, max(insertionid) as insertionid
       from reportsexecution
       where complete = true
and nodeid in ((values ('cfc8fe3d-09e6-4680-aea9-e347385753a2'),('540bc906-c672-49ce-acab-240d6e65676b'),('9b295838-6e8e-481b-927d-1e1891c317b2')))
       GROUP BY nodeid)) as r
left outer join nodeconfigurations as c
on r.nodeId = c.nodeid and r.nodeconfigid = c.nodeconfigid;

Actions
Copy link
 #4

Updated by Nicolas CHARLES almost 5 years ago

latest query shows an order of magnitude of perfs improvement

Actions
Copy link
 #5

Updated by Nicolas CHARLES almost 5 years ago

  • Status changed from Pending technical review to Pending release
Actions
Copy link
 #6

Updated by Alexis Mousset almost 5 years ago

  • Name check changed from To do to Reviewed
Actions
Copy link
 #7

Updated by Nicolas CHARLES almost 5 years ago

  • Fix check changed from To do to Checked
Actions
Copy link
 #8

Updated by Vincent MEMBRÉ almost 5 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 5.0.15 which was released today.

Actions
Copy link

Also available in: Atom PDF