Project

General

Profile

Actions

Architecture #4782

closed

Architecture #4778: Optimize promise generation algorithm: Rudder variables lookup

Split rudder variable parsing and variable binding

Added by François ARMAND almost 10 years ago. Updated almost 9 years ago.

Status:
Released
Priority:
1
Category:
Performance and scalability
Target version:
Effort required:
Name check:
Fix check:
Regression:

Description

Today, we parse Rudder variable in Directives FOR EACH NODE. That does not make the least sense. We only need to bind them for each node, and even that is two wide: we can bind them as soon as the full context is qualified, or even remove free variable until there is no more.

So, we have to:

- create a new data structure for directive variable (Parametrized Variables ?) which allows to build the variable from a context (node infos, etc): a kind of Context => Variable type
- create a new parsing phase in the promise generation process to parse Directive parameters and extracts Rudder variables
- and it could (should ?) even be done at Directive update-time, so that we are able to give the user some feed-back on it.
- bind Parametrized Variable when their context is known


Related issues 1 (0 open1 closed)

Related to Rudder - Bug #4242: Promise generation takes too long when getting more and more nodesReleasedFrançois ARMAND2014-03-11Actions
Actions #1

Updated by François ARMAND almost 10 years ago

  • Status changed from 8 to In progress
  • Pull Request set to https://github.com/Normation/rudder/pull/507

A first pull request is available here: https://github.com/Normation/rudder/pull/507

Everything mostly works, but:

- a lot of tests have to be changed, because they are now mostly irrelevant (test copy on case classes), and other part are missing (only the parsing is really well tested for now)
- interpolation in Rudder Parameters is not implemented (but it should be quite straight forward).

Actions #2

Updated by François ARMAND almost 10 years ago

Ho, I missed the intersting part: node configuration building step of promise generation for ~250 nodes, ~20 rules, ~50 directives is down from ~25s to <5 secondes.

Actions #3

Updated by François ARMAND almost 10 years ago

So, now everything is implemented, and timings (for ~250 nodes, 20 rules, some params and interpolation to actually solve) are:

Old:

Build current rule values :        162 ms, 23 ms, 24 ms
Expand rule parameters    :        223 ms, 64 ms, 41 ms
Build target configuration:      29709 ms, 27838 ms, 27555 ms
Update rule vals          :       1643 ms, 111 ms, 88 ms

Sum: 31737, 28036, 27708

New:

Build current rule values :         12 ms, 22 ms, 12 ms
Build target configuration:       2788 ms, 2751 ms, 2969 ms
Update rule vals          :        719 ms, 100 ms, 78 ms

Sum: 3519, 2873, 3059

So that's roughtly a x10 improvment.

Actions #4

Updated by François ARMAND almost 10 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from François ARMAND to Nicolas CHARLES

There is some tests that don't seem to test anything anymore, but I didn't succeed in identifying what we wanted to test on them.
A lot of other tests were added (more on the interpolation variable interpreation code, thought).

Actions #5

Updated by François ARMAND almost 10 years ago

  • Status changed from Pending technical review to Pending release
  • % Done changed from 0 to 100
Actions #6

Updated by Vincent MEMBRÉ over 9 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 2.11.0~beta1 that was released yesterday.

Actions #7

Updated by Benoît PECCATTE almost 9 years ago

  • Tracker changed from Enhancement to Architecture
Actions

Also available in: Atom PDF