Project

General

Profile

Actions

Architecture #9582

open

Query to get config of last run can be pretty expensive when there are a lot of new nodes runs

Added by Nicolas CHARLES over 7 years ago. Updated 7 days ago.

Status:
New
Priority:
N/A
Assignee:
-
Category:
Performance and scalability
Target version:
Effort required:
Name check:
Fix check:
Regression:

Description

The query to get last nodes runs configuration is ok, but when we have a significant number of nodes to filter on, it gets pretty slow (from 50ms to 2s with 900 nodes)
There is a threashold where it is cheaper to simply take all nodes, and filter in scala (as taking all nodes is more about 1s on my test system)

(note: this is also true in 3.1 implementation to get expected reports)

This needs more testing, obviously, but it could improve dashboard loading at web interface start, or when all nodes are answering at the same time

Actions #1

Updated by Nicolas CHARLES over 7 years ago

limit seems to be at about 400 nodes in the query out of 900 total node (a bit less than 50%) with a very large DB (8 days worth of reports and config)

with smaller (16 GB) dataset, difference seems even bigger

more test should be done with more and less nodes than 900, but it seems we can assume than more than 500 nodes in the query=> postfilter the whole list of nodes

Actions #2

Updated by Alexis Mousset over 7 years ago

  • Target version changed from 321 to 4.0.1
Actions #3

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 4.0.1 to 4.0.2
Actions #4

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 4.0.2 to 4.0.3
Actions #5

Updated by Vincent MEMBRÉ about 7 years ago

  • Target version changed from 4.0.3 to 4.0.4
Actions #6

Updated by Benoît PECCATTE almost 7 years ago

  • Severity set to Minor - inconvenience | misleading | easy workaround
  • User visibility set to Operational - other Techniques | Technique editor | Rudder settings
  • Priority set to 16
Actions #7

Updated by Vincent MEMBRÉ almost 7 years ago

  • Target version changed from 4.0.4 to 4.0.5
Actions #8

Updated by Vincent MEMBRÉ almost 7 years ago

  • Target version changed from 4.0.5 to 4.0.6
  • Priority changed from 16 to 15
Actions #9

Updated by Vincent MEMBRÉ almost 7 years ago

  • Target version changed from 4.0.6 to 4.0.7
Actions #10

Updated by Vincent MEMBRÉ almost 7 years ago

  • Target version changed from 4.0.7 to 357
Actions #11

Updated by Benoît PECCATTE almost 7 years ago

  • Priority changed from 15 to 29
Actions #12

Updated by Alexis Mousset over 6 years ago

  • Target version changed from 357 to 4.1.6
Actions #13

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 4.1.6 to 4.1.7
  • Priority changed from 29 to 28
Actions #14

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 4.1.7 to 4.1.8
Actions #15

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 4.1.8 to 4.1.9
  • Priority changed from 28 to 27
Actions #16

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 4.1.9 to 4.1.10
Actions #17

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 4.1.10 to 4.1.11
Actions #18

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 4.1.11 to 4.1.12
Actions #19

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 4.1.12 to 4.1.13
Actions #20

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.13 to 4.1.14
Actions #21

Updated by Benoît PECCATTE over 5 years ago

  • Target version changed from 4.1.14 to 4.1.15
Actions #22

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.15 to 4.1.16
Actions #23

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.16 to 4.1.17
Actions #24

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.17 to 4.1.18
  • Priority changed from 27 to 0
Actions #25

Updated by Vincent MEMBRÉ about 5 years ago

  • Target version changed from 4.1.18 to 4.1.19
Actions #26

Updated by Alexis Mousset about 5 years ago

  • Target version changed from 4.1.19 to 4.1.20
Actions #27

Updated by François ARMAND about 5 years ago

  • Target version changed from 4.1.20 to 4.1.21
Actions #28

Updated by Vincent MEMBRÉ almost 5 years ago

  • Target version changed from 4.1.21 to 4.1.22
Actions #29

Updated by Vincent MEMBRÉ almost 5 years ago

  • Target version changed from 4.1.22 to 4.1.23
Actions #30

Updated by Vincent MEMBRÉ almost 5 years ago

  • Target version changed from 4.1.23 to 4.1.24
Actions #31

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 4.1.24 to 588
Actions #32

Updated by François ARMAND over 4 years ago

  • Category changed from Web - Compliance & node report to Performance and scalability
  • Target version changed from 588 to 5.0.13
  • User visibility changed from Operational - other Techniques | Technique editor | Rudder settings to Operational - other Techniques | Rudder settings | Plugins
  • Priority changed from 0 to 27

@ncharles: it would be interesting to benchmark that on the load testing env.

Actions #33

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 5.0.13 to 5.0.14
Actions #34

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 5.0.14 to 5.0.15
Actions #35

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 5.0.15 to 5.0.16
Actions #36

Updated by Alexis Mousset about 4 years ago

  • Target version changed from 5.0.16 to 5.0.17
  • Priority changed from 27 to 28
Actions #37

Updated by Vincent MEMBRÉ almost 4 years ago

  • Target version changed from 5.0.17 to 5.0.18
Actions #38

Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 5.0.18 to 5.0.19
  • Priority changed from 28 to 29
Actions #39

Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 5.0.19 to 5.0.20
Actions #40

Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 5.0.20 to 797
  • Priority changed from 29 to 30
Actions #41

Updated by Benoît PECCATTE over 2 years ago

  • Target version changed from 797 to 6.1.14
  • Priority changed from 30 to 32
Actions #42

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.14 to 6.1.15
Actions #43

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.15 to 6.1.16
Actions #44

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.16 to 6.1.17
Actions #45

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.17 to 6.1.18
Actions #46

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.18 to 6.1.19
Actions #47

Updated by François ARMAND about 2 years ago

  • Tracker changed from Bug to Architecture
  • Severity deleted (Minor - inconvenience | misleading | easy workaround)
  • User visibility deleted (Operational - other Techniques | Rudder settings | Plugins)
  • Priority deleted (32)
Actions #48

Updated by Vincent MEMBRÉ almost 2 years ago

  • Target version changed from 6.1.19 to 6.1.20
Actions #49

Updated by Vincent MEMBRÉ almost 2 years ago

  • Target version changed from 6.1.20 to 6.1.21
Actions #50

Updated by Vincent MEMBRÉ almost 2 years ago

  • Target version changed from 6.1.21 to old 6.1 issues to relocate
Actions #51

Updated by Alexis Mousset about 1 year ago

  • Target version changed from old 6.1 issues to relocate to 8.0.0~alpha1
Actions #52

Updated by Vincent MEMBRÉ 9 months ago

  • Target version changed from 8.0.0~alpha1 to 8.0.0~beta1
Actions #53

Updated by Alexis Mousset 8 months ago

  • Target version changed from 8.0.0~beta1 to 8.1.0~alpha1
Actions #54

Updated by Vincent MEMBRÉ 2 months ago

  • Target version changed from 8.1.0~alpha1 to 8.1.0~beta1
Actions #55

Updated by Vincent MEMBRÉ 25 days ago

  • Target version changed from 8.1.0~beta1 to 8.1.0~beta2
Actions #56

Updated by Vincent MEMBRÉ 7 days ago

  • Target version changed from 8.1.0~beta2 to 8.1.0~beta3
Actions

Also available in: Atom PDF