Project

General

Profile

Actions
Copy link

Bug #16228

closed

Bug #16083: Big memory usage when fetching/writing node configuration and expected reports

Fetching nodes last run is not batched

Bug #16228: Fetching nodes last run is not batched

Added by Nicolas CHARLES about 6 years ago. Updated about 6 years ago.

Status:
Released
Priority:
N/A
Assignee:
François ARMAND
Category:
Performance and scalability
Target version:
5.0.15
Pull Request:
https://github.com/Normation/rudder/p...
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
Reviewed
Fix check:
Checked
Regression:

Description

we need to do the same as the parent ticket for getNodesLastRun
Query time is 25s with 5500 nodes, which is clearly too much (and too large)

Updated by Nicolas CHARLES about 6 years ago Actions
Copy link
 #1

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES

Updated by Nicolas CHARLES about 6 years ago Actions
Copy link
 #2

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/2623

Updated by Nicolas CHARLES about 6 years ago Actions
Copy link
 #3

ok, the query is really inefficient

it's much more efficient to do

select r.nodeid, r.date, r.nodeconfigid, r.complete, r.insertionid,
                         c.nodeid, c.nodeconfigid, c.begindate, c.enddate, c.configuration from
(select reportsexecution.nodeid, reportsexecution.date, reportsexecution.nodeconfigid, reportsexecution.complete, reportsexecution.insertionid from
reportsexecution where nodeid in  (values ('cfc8fe3d-09e6-4680-aea9-e347385753a2'),('540bc906-c672-49ce-acab-240d6e65676b'),('9b295838-6e8e-481b-927d-1e1891c317b2')) and (nodeid, insertionid) in
  (select nodeid, max(insertionid) as insertionid
       from reportsexecution
       where complete = true
       GROUP BY nodeid)) as r
left outer join nodeconfigurations as c
                         on r.nodeId = c.nodeid and r.nodeconfigid = c.nodeconfigid;

or 
select r.nodeid, r.date, r.nodeconfigid, r.complete, r.insertionid,
                         c.nodeid, c.nodeconfigid, c.begindate, c.enddate, c.configuration from
(select reportsexecution.nodeid, reportsexecution.date, reportsexecution.nodeconfigid, reportsexecution.complete, reportsexecution.insertionid from
reportsexecution where (nodeid, insertionid) in
  (select nodeid, max(insertionid) as insertionid
       from reportsexecution
       where complete = true
and nodeid in ((values ('cfc8fe3d-09e6-4680-aea9-e347385753a2'),('540bc906-c672-49ce-acab-240d6e65676b'),('9b295838-6e8e-481b-927d-1e1891c317b2')))
       GROUP BY nodeid)) as r
left outer join nodeconfigurations as c
on r.nodeId = c.nodeid and r.nodeconfigid = c.nodeconfigid;

Updated by Nicolas CHARLES about 6 years ago Actions
Copy link
 #4

latest query shows an order of magnitude of perfs improvement

Updated by Nicolas CHARLES about 6 years ago Actions
Copy link
 #5

  • Status changed from Pending technical review to Pending release

Updated by Alexis Mousset about 6 years ago Actions
Copy link
 #6

  • Name check changed from To do to Reviewed

Updated by Nicolas CHARLES about 6 years ago Actions
Copy link
 #7

  • Fix check changed from To do to Checked

Updated by Vincent MEMBRÉ about 6 years ago Actions
Copy link
 #8

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 5.0.15 which was released today.

Actions
Copy link

Also available in: PDF Atom