Project

General

Profile

Bug #18981

Updated by François ARMAND over 3 years ago

On very large system, dynamic group computation can be fairly slow. 
 There are some hotspot in the code where time is spent, but it seems most of time is wasted by the fact that we are querying OpenLDAP with a single thread - whereas we could hammer it with queries. 

 A proof of concept is available here https://github.com/Normation/rudder/pull/3523/files , and on my laptop (yeah, not great test plateform): 
 * 1 thread is 1.2s 
 * 2 threads is 0.8s 
 * 4 threads is 0.8s 
 * 8 threads is 0.75s (probably in the error bar) 

 Note that this implementation is *not* correct yet (as of https://github.com/Normation/rudder/pull/3523/commits/261d2900275c988aa36e5b641c23b9e150226133 ) as group depending on group must be treated afterward (ideally, using https://issues.rudder.io/issues/12060#note-6 algorithm, but simply sequentially on one thread if not possible) 

 we need also to add a configuration option for the number of threads 

 h3. Testing it 

 - set @rudder_compute_dyngroups_max_parallelism@ to for ex 4: 

 <pre> 
 curl -k --request POST -H "X-API-TOKEN: J903Y6Yj8NrqZ9a..." -H "Content-Type: application/json" 'https://192.168.25.2/rudder/api/latest/settings/rudder_compute_dyngroups_max_parallelism' --data '{"value":"4"}' 
 </pre> 

 - create several dyngroups, with some being composed of other ones (that's the only complex part) 

 Set logger @scheduled.job@ to debug, reload dynamic groups, and check that they are recomputed in parallel but with group with dependencies after group without.

Back