Rudder

On very large system, dynamic group computation can be fairly slow.
There are some hotspot in the code where time is spent, but it seems most of time is wasted by the fact that we are querying OpenLDAP with a single thread - whereas we could hammer it with queries.

A proof of concept is available here https://github.com/Normation/rudder/pull/3523/files , and on my laptop (yeah, not great test plateform):

• 1 thread is 1.2s
• 2 threads is 0.8s
• 4 threads is 0.8s
• 8 threads is 0.75s (probably in the error bar)

Note that this implementation is not correct yet (as of https://github.com/Normation/rudder/pull/3523/commits/261d2900275c988aa36e5b641c23b9e150226133 ) as group depending on group must be treated afterward (ideally, using https://issues.rudder.io/issues/12060#note-6 algorithm, but simply sequentially on one thread if not possible)

we need also to add a configuration option for the number of threads

Testing it¶

- set rudder_compute_dyngroups_max_parallelism to for ex 4:

curl -k --request POST -H "X-API-TOKEN: J903Y6Yj8NrqZ9a..." -H "Content-Type: application/json" 'https://192.168.25.2/rudder/api/latest/settings/rudder_compute_dyngroups_max_parallelism' --data '{"value":"4"}'

- create several dyngroups, with some being composed of other ones (that's the only complex part)

Set logger scheduled.job to debug, reload dynamic groups, and check that they are recomputed in parallel but with group with dependencies after group without.