Rudder

I would like to suggest that we make use of the available methods a system has to limit the agents runtime usage.

If cgroups are found operational, they could be used
Otherwise ionice and nice could be used, as could ulimits.

On systemd based systems, it would probably just be a matter of putting the right stuff in the unit file.

The idea would be to set all limits to a point where even a large agent / policy would be fine, but runaway issues would be automatically caught.
For CPU and memory limits we should put our expectations in the cgroup config, my main concern being that the agent is still allowed to use a lot of CPU but will do so at a low priority compared to applications on the system, and ideally, that would go for its disk and net IO too.

My classic example being that I never want to see a hickup on a near-realtime application because we suddenly used 100% for 3 seconds - where we also could have used 80% for 3.2s and not have had any visible impact.

For large setups this would be very helpful and reduce the length of test phases one needs to have.