Bug #8424
closedWhen updating runs, hooks should really be async
Description
We have a task in Rudder that periodically looks to new ruddersysevents and from that, marks agent run as completed. That task also calls several hooks (for now, two: one to update cache for recent changes, one other to update compliance).
These hooks were asynchrone until http://www.rudder-project.org/redmine/issues/8051. But with that ticket, we completelly removed the async calls, because we weren't sure what was the cause of the strange thread lock we were seing.
But that lead to very predictible problems: if one of the hooks, for example update change caches (see #8423), is taking a lot of time, then runs are not correctly marked as completed, and apocalypse ensues.
We will most likelly have case were one of the hook will take a lot of time.
So we need to build a robust, consistant, correct async hook executor that is not subject to the problem in #8051 but let hooks take their time if needed, without breaking everything.