Rudder

An ideal solution here would be to have a proxy in front of one or two cf-serverd processes that can phase out an existing cf-serverd when we know it's running an old config, and direct all new connections to the new cf-serverd.

haproxy?

haproxy is a little bit overkill, don't you think?
You could use iptable rules and have them simply redirect to a newer version after doing the update.
That would also keep existing connections alive, since only new incoming connections would be affected.
Just make sure you'll have cf-serverd listen on localhost (bindtointerface), not to have additional ports open.
And by making this change, you'd also have to put the port information (port => "&COMMUNITYPORT&";) outside of cf-served.st, and the external script would need to manage the ports of the running cf-serverd instances, since you could have already n running instances of it, all having still open connections that they are serving. how do you know a cf-serverd is not having any clients any more, and can be killed? How do you handle the checking of the numbers of processes that are allowed to run? So many question when you go from 1 to n... :)

Options are:

• restart instead of reload + two-steps policy update in the agent
• implementing graceful restart properly in cf-serverd
• switch between two instances using iptables
• prevent new connection using iptables until reload is done (with a timeout)

Is it an architectural thing to keep using cf-serverd? Does it benefit in long term?
Any benefits on long term to switch over to some kind of https-based file serving?

We were discussing that idea not two weeks ago. It has a side bonus point to allow consistant handling of cfengine/DSC agent.

And it allows to clearly defined our own protocol and its elements (for ex: something different for remote run, with clearly defined limited commands and identical for cfengine/dsc, again).

There is anther solution to do that : use systemd to do graceful reload.
We can use systemd's socket activation to pass sockets to cf-serverd, this is easy to implement : https://github.com/puma/puma/blob/master/docs/systemd.md and https://insanity.industries/post/socket-activation-all-the-things/

From there, it should be possible to let systemd handle port opening and incoming connections and let it pass those connection to the right cf-server instance.
There is a blog post by Lennart that says systemd can do this kind graceful restart, but systemd waits for the process to fully stop before restarting a new one, which fails to provide the feature.

However, there are workarounds to make it work : this post in french https://vincent.bernat.ch/fr/blog/2018-systemd-golang-socket-activation explains those workarounds, they all go around making systemd ignoring that the process has not finished stopping.

To sum things up, what happens is:

1. We add a new node to Rudder, we generate policies for it
2. cf-serverd ACLs are updated to allow it to connect
3. cf-serverd policies are updated (on root or relay), with these new ACLs
4. Then the cf-serverd process is supposed to detect the configuration change within a minute, and reload it. The problem is that cf-serverd cannot reload config with open connections (due to technical limitations that would probably be hard to overcome), so it does not even try to check if reload is needed when there is at least one connected node. On moderately loaded relays it may work eventually, but as new connections are not prevented, it is easy to completely skip configuration reload indefinitely.
   • A service reload (with SIGHUP) has exactly the same limit, and will be ignored if there are open connections.
   • A service restart fixes the problem, but will break existing connections, potentially leading to policy update and file copy errors on the connected nodes.

So we actually need two things:

• A way to properly reload config.
  • It could use systemd socket activation, as we have systemd on all relays and root servers. In this case we would spawn a new cf-serverd with the new config when a configuration reload is required, and let the old process handle existing connections. We would need to ask it to terminate when all connections are closed (a feature that does not exist for now) to avoid piling up.

• A way to detect a reload is needed, from outside of cf-serverd.
  • It is already done on root server using a policy generation hook.
  • On relays we might need to rely on policy update repairs, to trigger a reload by systemd

Benoît PECCATTE is working on implementing the systemd socket activation.

Upstream PR https://github.com/cfengine/core/pull/4499.