Rudder

There is also one modification that could be worth doing.
Now, when we add a node to a rule/group, then we consider that the rule changed, and regenerate the promises for all the nodes of that rule.

We could, for node addition only, generate only the promises for the new nodes (== increasing the serial of a rule only if there there are nodes in the old target of rules that are not in the new target of rules)

It is clearly not exclusive with the solution stated above

What do you think of it?

Nicolas CHARLES wrote:

There is also one modification that could be worth doing.
Now, when we add a node to a rule/group, then we consider that the rule changed, and regenerate the promises for all the nodes of that rule.

We could, for node addition only, generate only the promises for the new nodes (== increasing the serial of a rule only if there there are nodes in the old target of rules that are not in the new target of rules)

It is clearly not exclusive with the solution stated above

What do you think of it?

Any and all optimisations that are easy to describe, understand and provide potential gain are good for me!

Also, a quick and easy optimisation here would be to modify the code that runs the verification command (usually cf-promises) and check if it is set to "/bin/true", then avoid running any systems calls for that case (forking to run an external command, even just /bin/true, 1000 or 5000 times has got to have a cost).

Jonathan CLARKE wrote:

Also, a quick and easy optimisation here would be to modify the code that runs the verification command (usually cf-promises) and check if it is set to "/bin/true", then avoid running any systems calls for that case (forking to run an external command, even just /bin/true, 1000 or 5000 times has got to have a cost).

This is addressed in #4475

Just wanted to put in a "this feels good" vote for 1)
I think those need to be turned into async:

Policy gen
Policy test
Policy application (if tested)

Not a simple task if one considers changes on top of changes and the need to respect validation.
You'll probably end up inventing filesystem snapshots ;)

Well, actually if you thing to the whole process as a stream of events each one leading to an immutable version of THE IT configuration (internal to Rudder: the one data structure that allows to know the state of all the config), it is quite easy to model: we have the chance to have really few events to take care of, and it is rather easy to know what are incompatibles changes.

So imagine something like that:

Incomming events are: group/technique/directive/rule/parameters changes, new inventory from a node, system parameter (like authorized network or run interval) changes
If the event is not legal (bad parameters, apply to something not existing, etc), give an immediate feedback to the event producer and ignore it.
On legal event:
=> create a new version of the global configuration model . That version is the new reference one, new modif are related to that one and all folowing steps are asynchrone. Here, we also keep (at least) the last good configuration model. Note that here, we can have several events applied at the same time, we just don't have promise generated for them. New event are not starting a new generation, but are waiting for the asynchrone steps to finish.

So we end up with somethin like:

T0 -> config0, status: fully generated, CURRENT, path to root directory of promises: ...
T1 -> config1, status: not generated
T2 -> config2, status: not generated
T3 -> config3, status: processing async steps
T4 -> config4, status: waiting

=> create promises, write them, test them, and why not post-hook are part of the processing.

Until a config reach end of processing, we don't change the CURRENT flag to the new root directory. Changing the flag must be something atomic (change a symlink, change a snapshot number of the fs, change a branch on git...)

So, what happens on error, because that's the intersting part ?

When a generation fails, nothing is broken yet, because old ok promises are still the one available.
So we mark the config "bad", display in a status page the reason, and start building the most recent config not builded yet (either one not generated, or one waiting).

We could even have that flow at the node granularity, but we need to make the information clearly available for the user, so that he can knows with one sight what node are in what config version (of course, the important part is being to know easily what nodes are not is the last version and so may miss important updates).

T0 -> config0, status: fully generated, CURRENT, path to root directory of promises: ...
T1 -> config1, status: not generated
T2 -> config2, status: not generated
T3 -> config3, status: bad: error message(s) explaining the problems
T4 -> config4, status: waiting
T5 -> config5, status: waiting
T5 -> config4, status: processing async steps

Effectively, it looks like some kind of fs snapshotting, but now that we have git, it's almost a given ;)

We are working on a major speed improvement of cf-promise: https://github.com/cfengine/core/pull/2818

1. policy generation is much much faster
2. policy validation still takes a lot of time (about 1s per node)
3. We have an option in Settings to disable the policy validation (see #14331 )

Plan is to outsource to the nodes themselves the policy validation (i cannot find the ticket for the moment on the bugtracker thought)