Bug #18981
Updated by François ARMAND almost 3 years ago
On very large system, dynamic group computation can be fairly slow.
There are some hotspot in the code where time is spent, but it seems most of time is wasted by the fact that we are querying OpenLDAP with a single thread - whereas we could hammer it with queries.
A proof of concept is available here https://github.com/Normation/rudder/pull/3523/files , and on my laptop (yeah, not great test plateform):
* 1 thread is 1.2s
* 2 threads is 0.8s
* 4 threads is 0.8s
* 8 threads is 0.75s (probably in the error bar)
Note that this implementation is *not* correct yet (as of https://github.com/Normation/rudder/pull/3523/commits/261d2900275c988aa36e5b641c23b9e150226133 ) as group depending on group must be treated afterward (ideally, using https://issues.rudder.io/issues/12060#note-6 algorithm, but simply sequentially on one thread if not possible)
we need also to add a configuration option for the number of threads
h3. Testing it
- set @rudder_compute_dyngroups_max_parallelism@ to for ex 4:
<pre>
curl -k --request POST -H "X-API-TOKEN: J903Y6Yj8NrqZ9a..." -H "Content-Type: application/json" 'https://192.168.25.2/rudder/api/latest/settings/rudder_compute_dyngroups_max_parallelism' --data '{"value":"4"}'
</pre>
- create several dyngroups, with some being composed of other ones (that's the only complex part)
Set logger @scheduled.job@ to debug, reload dynamic groups, and check that they are recomputed in parallel but with group with dependencies after group without.