Project

General

Profile

Actions
Copy link

Architecture #9582

open

Query to get config of last run can be pretty expensive when there are a lot of new nodes runs

Added by Nicolas CHARLES about 8 years ago. Updated 15 days ago.

Status:
New
Priority:
N/A
Assignee:
-
Category:
Performance and scalability
Target version:
8.1.9
Pull Request:
Effort required:
Name check:
Fix check:
Regression:

Description

The query to get last nodes runs configuration is ok, but when we have a significant number of nodes to filter on, it gets pretty slow (from 50ms to 2s with 900 nodes)
There is a threashold where it is cheaper to simply take all nodes, and filter in scala (as taking all nodes is more about 1s on my test system)

(note: this is also true in 3.1 implementation to get expected reports)

This needs more testing, obviously, but it could improve dashboard loading at web interface start, or when all nodes are answering at the same time

Actions
Copy link
 #1

Updated by Nicolas CHARLES about 8 years ago

limit seems to be at about 400 nodes in the query out of 900 total node (a bit less than 50%) with a very large DB (8 days worth of reports and config)

with smaller (16 GB) dataset, difference seems even bigger

more test should be done with more and less nodes than 900, but it seems we can assume than more than 500 nodes in the query=> postfilter the whole list of nodes

Actions
Copy link
 #2

Updated by Alexis Mousset about 8 years ago

  • Target version changed from 321 to 4.0.1
Actions
Copy link
 #3

Updated by Vincent MEMBRÉ almost 8 years ago

  • Target version changed from 4.0.1 to 4.0.2
Actions
Copy link
 #4

Updated by Vincent MEMBRÉ almost 8 years ago

  • Target version changed from 4.0.2 to 4.0.3
Actions
Copy link
 #5

Updated by Vincent MEMBRÉ almost 8 years ago

  • Target version changed from 4.0.3 to 4.0.4
Actions
Copy link
 #6

Updated by Benoît PECCATTE over 7 years ago

  • Severity set to Minor - inconvenience | misleading | easy workaround
  • User visibility set to Operational - other Techniques | Technique editor | Rudder settings
  • Priority set to 16
Actions
Copy link
 #7

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 4.0.4 to 4.0.5
Actions
Copy link
 #8

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 4.0.5 to 4.0.6
  • Priority changed from 16 to 15
Actions
Copy link
 #9

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 4.0.6 to 4.0.7
Actions
Copy link
 #10

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 4.0.7 to 357
Actions
Copy link
 #11

Updated by Benoît PECCATTE over 7 years ago

  • Priority changed from 15 to 29
Actions
Copy link
 #12

Updated by Alexis Mousset over 7 years ago

  • Target version changed from 357 to 4.1.6
Actions
Copy link
 #13

Updated by Vincent MEMBRÉ over 7 years ago

  • Target version changed from 4.1.6 to 4.1.7
  • Priority changed from 29 to 28
Actions
Copy link
 #14

Updated by Vincent MEMBRÉ about 7 years ago

  • Target version changed from 4.1.7 to 4.1.8
Actions
Copy link
 #15

Updated by Vincent MEMBRÉ about 7 years ago

  • Target version changed from 4.1.8 to 4.1.9
  • Priority changed from 28 to 27
Actions
Copy link
 #16

Updated by Vincent MEMBRÉ almost 7 years ago

  • Target version changed from 4.1.9 to 4.1.10
Actions
Copy link
 #17

Updated by Vincent MEMBRÉ almost 7 years ago

  • Target version changed from 4.1.10 to 4.1.11
Actions
Copy link
 #18

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 4.1.11 to 4.1.12
Actions
Copy link
 #19

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 4.1.12 to 4.1.13
Actions
Copy link
 #20

Updated by Vincent MEMBRÉ over 6 years ago

  • Target version changed from 4.1.13 to 4.1.14
Actions
Copy link
 #21

Updated by Benoît PECCATTE over 6 years ago

  • Target version changed from 4.1.14 to 4.1.15
Actions
Copy link
 #22

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 4.1.15 to 4.1.16
Actions
Copy link
 #23

Updated by Vincent MEMBRÉ about 6 years ago

  • Target version changed from 4.1.16 to 4.1.17
Actions
Copy link
 #24

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 4.1.17 to 4.1.18
  • Priority changed from 27 to 0
Actions
Copy link
 #25

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 4.1.18 to 4.1.19
Actions
Copy link
 #26

Updated by Alexis Mousset almost 6 years ago

  • Target version changed from 4.1.19 to 4.1.20
Actions
Copy link
 #27

Updated by François ARMAND over 5 years ago

  • Target version changed from 4.1.20 to 4.1.21
Actions
Copy link
 #28

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.21 to 4.1.22
Actions
Copy link
 #29

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.22 to 4.1.23
Actions
Copy link
 #30

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.23 to 4.1.24
Actions
Copy link
 #31

Updated by Vincent MEMBRÉ over 5 years ago

  • Target version changed from 4.1.24 to 588
Actions
Copy link
 #32

Updated by François ARMAND over 5 years ago

  • Category changed from Web - Compliance & node report to Performance and scalability
  • Target version changed from 588 to 5.0.13
  • User visibility changed from Operational - other Techniques | Technique editor | Rudder settings to Operational - other Techniques | Rudder settings | Plugins
  • Priority changed from 0 to 27

@ncharles: it would be interesting to benchmark that on the load testing env.

Actions
Copy link
 #33

Updated by Vincent MEMBRÉ about 5 years ago

  • Target version changed from 5.0.13 to 5.0.14
Actions
Copy link
 #34

Updated by Vincent MEMBRÉ about 5 years ago

  • Target version changed from 5.0.14 to 5.0.15
Actions
Copy link
 #35

Updated by Vincent MEMBRÉ almost 5 years ago

  • Target version changed from 5.0.15 to 5.0.16
Actions
Copy link
 #36

Updated by Alexis Mousset almost 5 years ago

  • Target version changed from 5.0.16 to 5.0.17
  • Priority changed from 27 to 28
Actions
Copy link
 #37

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 5.0.17 to 5.0.18
Actions
Copy link
 #38

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 5.0.18 to 5.0.19
  • Priority changed from 28 to 29
Actions
Copy link
 #39

Updated by Vincent MEMBRÉ about 4 years ago

  • Target version changed from 5.0.19 to 5.0.20
Actions
Copy link
 #40

Updated by Vincent MEMBRÉ about 4 years ago

  • Target version changed from 5.0.20 to 797
  • Priority changed from 29 to 30
Actions
Copy link
 #41

Updated by Benoît PECCATTE over 3 years ago

  • Target version changed from 797 to 6.1.14
  • Priority changed from 30 to 32
Actions
Copy link
 #42

Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 6.1.14 to 6.1.15
Actions
Copy link
 #43

Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 6.1.15 to 6.1.16
Actions
Copy link
 #44

Updated by Vincent MEMBRÉ about 3 years ago

  • Target version changed from 6.1.16 to 6.1.17
Actions
Copy link
 #45

Updated by Vincent MEMBRÉ about 3 years ago

  • Target version changed from 6.1.17 to 6.1.18
Actions
Copy link
 #46

Updated by Vincent MEMBRÉ almost 3 years ago

  • Target version changed from 6.1.18 to 6.1.19
Actions
Copy link
 #47

Updated by François ARMAND almost 3 years ago

  • Tracker changed from Bug to Architecture
  • Severity deleted (Minor - inconvenience | misleading | easy workaround)
  • User visibility deleted (Operational - other Techniques | Rudder settings | Plugins)
  • Priority deleted (32)
Actions
Copy link
 #48

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.19 to 6.1.20
Actions
Copy link
 #49

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.20 to 6.1.21
Actions
Copy link
 #50

Updated by Vincent MEMBRÉ over 2 years ago

  • Target version changed from 6.1.21 to old 6.1 issues to relocate
Actions
Copy link
 #51

Updated by Alexis Mousset over 1 year ago

  • Target version changed from old 6.1 issues to relocate to 8.0.0~alpha1
Actions
Copy link
 #52

Updated by Vincent MEMBRÉ over 1 year ago

  • Target version changed from 8.0.0~alpha1 to 8.0.0~beta1
Actions
Copy link
 #53

Updated by Alexis Mousset over 1 year ago

  • Target version changed from 8.0.0~beta1 to 8.1.0~alpha1
Actions
Copy link
 #54

Updated by Vincent MEMBRÉ 10 months ago

  • Target version changed from 8.1.0~alpha1 to 8.1.0~beta1
Actions
Copy link
 #55

Updated by Vincent MEMBRÉ 9 months ago

  • Target version changed from 8.1.0~beta1 to 8.1.0~beta2
Actions
Copy link
 #56

Updated by Vincent MEMBRÉ 8 months ago

  • Target version changed from 8.1.0~beta2 to 8.1.0~rc1
Actions
Copy link
 #57

Updated by Vincent MEMBRÉ 8 months ago

  • Target version changed from 8.1.0~rc1 to 8.1.0
Actions
Copy link
 #58

Updated by Vincent MEMBRÉ 8 months ago

  • Target version changed from 8.1.0 to 8.1.1
Actions
Copy link
 #59

Updated by Vincent MEMBRÉ 7 months ago

  • Target version changed from 8.1.1 to 8.1.2
Actions
Copy link
 #60

Updated by Vincent MEMBRÉ 7 months ago

  • Target version changed from 8.1.2 to 8.1.3
Actions
Copy link
 #61

Updated by Vincent MEMBRÉ 6 months ago

  • Target version changed from 8.1.3 to 8.1.4
Actions
Copy link
 #62

Updated by Vincent MEMBRÉ 5 months ago

  • Target version changed from 8.1.4 to 8.1.5
Actions
Copy link
 #63

Updated by Vincent MEMBRÉ 5 months ago

  • Target version changed from 8.1.5 to 8.1.6
Actions
Copy link
 #64

Updated by Vincent MEMBRÉ 4 months ago

  • Target version changed from 8.1.6 to 8.1.7
Actions
Copy link
 #65

Updated by Vincent MEMBRÉ 2 months ago

  • Target version changed from 8.1.7 to 8.1.8
Actions
Copy link
 #66

Updated by Vincent MEMBRÉ 15 days ago

  • Target version changed from 8.1.8 to 8.1.9
Actions
Copy link

Also available in: Atom PDF