Architecture #6087
closed
Architecture to handle thousands of nodes
Description
Since 3.0, Rudder manage correctly hundreds of nodes. When we reach the couple of thousand of nodes, Rudder starts to be not nice to use (at all).
The main pain points are:
- the UI is slow;
- generation of policies takes several minutes (with cf-promises check disabled)
The main reason for that are (and are linked):
- we have (almost) no cache
- we don't handle diff/gradual update
Tipically, we should be able to only write promise files that are actually modified, not all file for a given node - and ideally, we should only write parameters-like file, all other being shared in a common place (and ln -s to).
Example of cache that could be set: we preciselly know when the list of directive change.
And the major cache that we could set-up is on nodes (inventories).
Note the in #5452, we introduced a cache for nodeInfo, but the underlying architecture was not adapted. Ideally, we want to have some event generated when a node/inventory/config is modified, and use these events to invalidate caches.
Updated by 
Updated by 
Updated by 
Updated by