Rudder

It does not seems to correct itself (even when running by hand rudder agent run -u -i)

Just reloading cf-serverd does not work, as "failsafe.cf" is used:

root@server:/var/rudder/configuration-repository# /opt/rudder/etc/hooks.d/policy-generation-finished/50-reload-policy-file-server
# in journalctl:
Mar 20 10:28:21 server cf-serverd[12321]:   notice: Rereading policy file '/var/rudder/cfengine-community/inputs/failsafe.cf'
Mar 20 10:28:21 server cf-serverd[12321]: CFEngine(server) rudder Rereading policy file '/var/rudder/cfengine-community/inputs/failsafe.cf'

Forcing full policy regeneration does not heal the problem, cf-serverd is still reading failsafe.cf.

Running "rudder server debug xxxx" makes "xxxx" able to get its policies, but as soon as the debug server is stoppend, we fall back in bad state with impossible update.

Bahaviour is not the same in 4.1.10
Breaking promises.cf results in

Mar 20 11:45:12 orchestrateur-4 [25127]: CFEngine(common)  There are syntax errors in policy files
Mar 20 11:45:12 orchestrateur-4 cf-serverd[32336]: CFEngine(server) rudder Policy failed validation with command '"/var/rudder/cfengine-community/bin/cf-promises" -c "/var/rudder/cfengine-community/inputs/promises.cf"'

in 4.2, I have

Mar 20 11:52:05 orchestrateur-6 [10077]: CFEngine(common)  There are syntax errors in policy files
Mar 20 11:51:06 orchestrateur-6 cf-serverd[16990]: CFEngine(server) rudder Policy failed validation with command '"/var/rudder/cfengine-community/bin/cf-promises" -c "/var/rudder/cfengine-community/inputs/promises.cf"'

actually, in 4.3, it behaves the same
If I manually corrupt promises.cf and reload cf-serverd policies, I get

Mar 20 05:55:01 server cf-serverd[8667]: CFEngine(server) rudder Policy failed validation with command '"/var/rudder/cfengine-community/bin/cf-promises" -c "/var/rudder/cfengine-community/inputs/promises
Mar 20 05:55:01 server cf-serverd[8667]: /var/rudder/cfengine-community/inputs/promises.cf:24:3: error: syntax error
Mar 20 05:55:01 server cf-serverd[8667]: iii
Mar 20 05:55:01 server cf-serverd[8667]: ^
Mar 20 05:55:01 server cf-serverd[8667]: /var/rudder/cfengine-community/inputs/promises.cf:24:3: error: Expected 'bundle' or 'body' keyword, wrong input 'iii'
Mar 20 05:55:01 server cf-serverd[8667]: iii
Mar 20 05:55:01 server cf-serverd[8667]: ^
Mar 20 05:55:01 server cf-serverd[8667]: error: There are syntax errors in policy files
Mar 20 05:55:01 server cf-serverd[8667]: error: Policy failed validation with command '"/var/rudder/cfengine-community/bin/cf-promises" -c "/var/rudder/cfengine-community/inputs/promises.cf"'

There should be an error log saying:

Failsafe condition triggered. Interactive session detected, skipping failsafe.cf execution.

do you see it?

Restarting rudder-agent (service rudder-agent restart) repairs the problem:

Mar 20 11:05:40 server systemd[1]: Stopped Rudder agent umbrella service.
Mar 20 11:05:40 server systemd[1]: Stopping Rudder agent umbrella service...
Mar 20 11:05:40 server systemd[1]: Starting Rudder agent umbrella service...
Mar 20 11:05:40 server systemd[1]: Stopping CFEngine Execution Scheduler...
Mar 20 11:05:40 server cf-serverd[12321]:   notice: Cleaning up and exiting...
Mar 20 11:05:40 server cf-serverd[12321]: CFEngine(server) rudder Cleaning up and exiting...
Mar 20 11:05:40 server systemd[1]: Stopping CFEngine file server...
Mar 20 11:05:40 server systemd[1]: Stopped CFEngine Execution Scheduler.
Mar 20 11:05:40 server systemd[1]: Stopped CFEngine file server.
Mar 20 11:05:40 server systemd[1]: Started Rudder agent umbrella service.
Mar 20 11:05:40 server systemd[1]: Started CFEngine file server.
Mar 20 11:05:40 server systemd[1]: Started CFEngine Execution Scheduler.
Mar 20 11:05:41 server cf-serverd[12037]:   notice: Server is starting...
Mar 20 11:05:41 server cf-serverd[12037]: CFEngine(server) rudder Server is starting...

And then policy updates from nodes are possible again.

breaking intentionnaly generated promises in 4.1 does not trigger the issue

Mar 20 13:43:56 orchestrateur-4 cf-serverd[32336]: CFEngine(server) rudder Policy failed validation with command '"/var/rudder/cfengine-community/bin/cf-promises" -c "/var/rudder/cfengine-community/inputs/promises" -c "/var/rudder/cfengine-community/inputs/promises.cf"'

This is not related to systemd
I stopped cf-serverd, and started it manually, with broken promises
it went

root@server:/home/vagrant# /var/rudder/cfengine-community/bin/cf-serverd  -v --no-fork
 verbose: Could not open extension plugin 'cfengine-enterprise.so' from '/var/rudder/cfengine-community/lib/cfengine-enterprise.so': (not installed)
 verbose:  CFEngine Core 3.10.3
 verbose: ----------------------------------------------------------------
 verbose:  Initialization preamble 
 verbose: ----------------------------------------------------------------
 verbose: Default port for cfengine is 5308
 verbose: Work directory is /var/rudder/cfengine-community
 verbose: Making sure that internal directories are private...
 verbose: Checking integrity of the trusted workdir
 verbose: Checking integrity of the state database
 verbose: Checking integrity of the module directory
 verbose: Checking integrity of the PKI directory
 verbose: Loaded private key at '/var/rudder/cfengine-community/ppkeys/localhost.priv'
 verbose: Loaded public key '/var/rudder/cfengine-community/ppkeys/localhost.pub'
 verbose: Reference time set to 'Tue Mar 20 09:01:28 2018'
 verbose: CFEngine Core 3.10.3 - ready
 verbose: ----------------------------------------------------------------
 verbose:  Environment discovery 
 verbose: ----------------------------------------------------------------
 verbose: Host name is: server
 verbose: Operating System Type is linux
 verbose: Operating System Release is 4.4.21-69-default
 verbose: Architecture = x86_64
 verbose: CFEngine detected operating system description is linux
 verbose: The time is now Tue Mar 20 09:01:28 2018
 verbose: Additional hard class defined as: 64_bit
 verbose: Additional hard class defined as: linux_4_4_21_69_default
 verbose: Additional hard class defined as: linux_x86_64
 verbose: Additional hard class defined as: linux_x86_64_4_4_21_69_default
 verbose: GNU autoconf class from compile time: compiled_on_linux_gnu
 verbose: Address given by nameserver: 192.168.42.2
 verbose: No interface exception file /var/rudder/cfengine-community/inputs/ignore_interfaces.rx
 verbose: Interface 1: lo
 verbose: Interface 2: eth0
 verbose: IP address of host set to 10.0.2.15
 verbose: Interface 3: eth1
 verbose: Trying to locate my IPv6 address
 verbose: Found IPv6 address fe80::a00:27ff:fe9f:3206
 verbose: Found IPv6 address fe80::a00:27ff:fe88:bef0
 verbose: Reading netstat info from /proc/net/netstat
 verbose: Reading (null) info from /proc/net/route
 verbose: Reading (null) info from /proc/net/snmp6
 verbose: Reading (null) info from /proc/net/ipv6_route
 verbose: Reading (null) info from /proc/net/if_inet6
 verbose: Reading interfaces_data info from /proc/net/dev
 verbose: Looking for environment from cf-monitord...
 verbose: Unable to detect environment from cf-monitord
 verbose: This appears to be a SUSE system.
 verbose: Looking for SUSE enterprise info in 'SUSE Linux Enterprise Server 12 (x86_64)'
 verbose: Discovered SUSE version SuSE_12
 verbose: Found 1 processor
 verbose: Loading persistent classes
 verbose: Persistent class 'heartbeat_sent' for 3 more minutes
 verbose: Adding persistent class 'heartbeat_sent' to heap
 verbose: Persistent class 'inventory_sent' for 233 more minutes
 verbose: Adding persistent class 'inventory_sent' to heap
 verbose: Persistent class 'rudder_ncf_common_updated_ok' for 3 more minutes
 verbose: Adding persistent class 'rudder_ncf_common_updated_ok' to heap
 verbose: Persistent class 'rudder_ncf_local_updated_ok' for 3 more minutes
 verbose: Adding persistent class 'rudder_ncf_local_updated_ok' to heap
 verbose: Persistent class 'site_ok' for 9 more minutes
 verbose: Adding persistent class 'site_ok' to heap
 verbose: Loading JSON augments from '/var/rudder/cfengine-community/inputs/def.json' (input dir '/var/rudder/cfengine-community/inputs', input file '/var/rudder/cfengine-community/inputs/promises.cf'
 verbose: could not load JSON augments from '/var/rudder/cfengine-community/inputs/def.json'
 verbose: This agent is bootstrapped to: 127.0.0.1
 verbose: Input file '/var/rudder/cfengine-community/inputs/promises.cf' has changed since the last policy read attempt (file is newer than previous)
 verbose: Input file is changed since last validation, validating it
 verbose: Verifying the syntax of the inputs...
 verbose: Checking policy with command '"/var/rudder/cfengine-community/bin/cf-promises" -c "/var/rudder/cfengine-community/inputs/promises.cf"'
/var/rudder/cfengine-community/inputs/promises.cf:24:3: error: syntax error
iii
  ^
/var/rudder/cfengine-community/inputs/promises.cf:24:3: error: Expected 'bundle' or 'body' keyword, wrong input 'iii'
iii
  ^
   error: There are syntax errors in policy files
   error: Policy failed validation with command '"/var/rudder/cfengine-community/bin/cf-promises" -c "/var/rudder/cfengine-community/inputs/promises.cf"'
   error: Failsafe condition triggered. Interactive session detected, skipping failsafe.cf execution.
   error: Error reading CFEngine policy. Exiting...

reloading with /opt/rudder/etc/hooks.d/policy-generation-finished/50-reload-policy-file-server resulted in the following log

Mar 20 09:12:11 server cf-serverd[21809]: CFEngine(server) rudder Rereading policy file '/var/rudder/cfengine-community/inputs/failsafe.cf'
Mar 20 09:12:11 server cf-serverd[21809]: notice: Rereading policy file '/var/rudder/cfengine-community/inputs/failsafe.cf'

Please note that running again the cf-serverd tells me:

Mar 20 09:11:46 server cf-promises[22300]: CFEngine(common)  There are syntax errors in policy files
Mar 20 09:11:46 server cf-serverd[22298]: CFEngine(server)  Policy failed validation with command '"/var/rudder/cfengine-community/bin/cf-promises" -c "/var/rudder/cfengine-community/inputs/promises.cf"'
Mar 20 09:11:46 server cf-serverd[22298]: CFEngine(server)  Failsafe condition triggered. Interactive session detected, skipping failsafe.cf execution.
Mar 20 09:11:46 server cf-serverd[22298]: CFEngine(server)  Error reading CFEngine policy. Exiting...

Interestingly, the bad cf-serverd is listening on port 5308 rather than 5309

What happens here:

• This behavior always existed but was never visible because agent restart with broken policies were done only during installation/upgrade, hence very likely in a tty, and CFEngine does not switch to failsafe in this case
• Since we are using systemd units, "service rudder agent restart" uses systemctl, CFEngine sees no tty and reloads failsafe.

This can be reproduced on any previous Rudder version by restarting the service (with broken policies) with nohup (for example).

We could:

• Patch CFEngine to disable the switch to failsafe (but that won't probably be acceptable upstream, and requires improving "rudder agent check" to trigger a failsafe run when promises are broken)
• ?

Doesn't ruder agent check already run failsafe when promises are broken ?

No, there are two cases:

• failsafe is broken -> reset initial policy + failsafe + run
• promises are broken -> clear update flag to force update at next run

Target 4.1, this should fix strange bugs where node couldn't connect while the server was running.