Rudder

This is transcient and starting an other full regeneration manually corrected the problem.

We have the problem on 2 different centos 7. The failing part is not always the same. Asking for a new regeneration (or a full new generation) clean the problem (most of the time).
Asking for a full regeneration have a hight probability (35% ?) of chance to lead to a variant of the error.
It is not always the same node, nor the same file which have the error. Below, some example:

#### two nodes failing, root and one behing a relay: 

 Exit code=1 for hook: '/opt/rudder/etc/hooks.d/policy-generation-node-ready/10-cf-promise-check' with environment variables: [PATH:/sbin:/usr/sbin:/bin:/usr/bin] [SYSTEMCTL_IGNORE_DEPENDENCIES:] [NLSPATH:/usr/dt/lib/nls/msg/%L/%N.cat] [SYSTEMCTL_SKIP_REDIRECT:] [OLDPWD:/opt/rudder/jetty7] [TERM:vt100] [XFILESEARCHPATH:/usr/dt/app-defaults/%L/Dt] [PWD:/opt/rudder/jetty7] [SHLVL:2] [_:/bin/java] [RUDDER_GENERATION_DATETIME:2017-03-29T14:16:00.537Z] [RUDDER_NODEID:root] [RUDDER_NEXT_POLICIES_DIRECTORY:/var/rudder/cfengine-community/inputs.new] [RUDDER_AGENT_TYPE:cfengine-community]. 
Stdout: ' error: Can't stat file '/var/rudder/ncf/find: '/var/rudder/cfengine-community/state/ncf-exclude-cache-3.10.0/_var_rudder_ncf_common_30_generic_methods': No such file or directory' for parsing. (stat: No such file or directory)
' 
Stderr: '' ; Exit code=1 for hook: '/opt/rudder/etc/hooks.d/policy-generation-node-ready/10-cf-promise-check' with environment variables: [PATH:/sbin:/usr/sbin:/bin:/usr/bin] [SYSTEMCTL_IGNORE_DEPENDENCIES:] [NLSPATH:/usr/dt/lib/nls/msg/%L/%N.cat] [SYSTEMCTL_SKIP_REDIRECT:] [OLDPWD:/opt/rudder/jetty7] [TERM:vt100] [XFILESEARCHPATH:/usr/dt/app-defaults/%L/Dt] [PWD:/opt/rudder/jetty7] [SHLVL:2] [_:/bin/java] [RUDDER_GENERATION_DATETIME:2017-03-29T14:16:00.537Z] [RUDDER_NODEID:26acd240-5347-4e0c-91e9-d6281341cf2b] [RUDDER_NEXT_POLICIES_DIRECTORY:/var/rudder/share/19a0eaaf-7d56-4251-a697-83942b18df20/share/26acd240-5347-4e0c-91e9-d6281341cf2b/rules.new/cfengine-community] [RUDDER_AGENT_TYPE:cfengine-community]. 
Stdout: ' error: Can't stat file '/var/rudder/ncf/find: '/var/rudder/cfengine-community/state/ncf-exclude-cache-3.10.0/_var_rudder_ncf_common_10_ncf_internals': No such file or directory' for parsing. (stat: No such file or directory)
' 
Stderr: ''

#### one other example:

⇨ Policy update error for process '17' at 2017-03-29 14:22:29 
⇨ Cannot write configuration node 
⇨ Exit code=1 for hook: '/opt/rudder/etc/hooks.d/policy-generation-node-ready/10-cf-promise-check' with environment variables: [PATH:/sbin:/usr/sbin:/bin:/usr/bin] [SYSTEMCTL_IGNORE_DEPENDENCIES:] [NLSPATH:/usr/dt/lib/nls/msg/%L/%N.cat] [SYSTEMCTL_SKIP_REDIRECT:] [OLDPWD:/opt/rudder/jetty7] [TERM:vt100] [XFILESEARCHPATH:/usr/dt/app-defaults/%L/Dt] [PWD:/opt/rudder/jetty7] [SHLVL:2] [_:/bin/java] [RUDDER_GENERATION_DATETIME:2017-03-29T14:22:23.982Z] [RUDDER_NODEID:19a0eaaf-7d56-4251-a697-83942b18df20] [RUDDER_NEXT_POLICIES_DIRECTORY:/var/rudder/share/19a0eaaf-7d56-4251-a697-83942b18df20/rules.new/cfengine-community] [RUDDER_AGENT_TYPE:cfengine-community]. 
Stdout: ' error: Can't stat file '/var/rudder/ncf/find: '/var/rudder/cfengine-community/state/ncf-exclude-cache-3.10.0/_var_rudder_ncf_common_60_services': No such file or directory' for parsing. (stat: No such file or directory)
' 
Stderr: ''

#### an other example:

⇨ Policy update error for process '26' at 2017-03-29 14:27:30 
⇨ Cannot write configuration node 
⇨ Exit code=1 for hook: '/opt/rudder/etc/hooks.d/policy-generation-node-ready/10-cf-promise-check' with environment variables: [PATH:/sbin:/usr/sbin:/bin:/usr/bin] [SYSTEMCTL_IGNORE_DEPENDENCIES:] [NLSPATH:/usr/dt/lib/nls/msg/%L/%N.cat] [SYSTEMCTL_SKIP_REDIRECT:] [OLDPWD:/opt/rudder/jetty7] [TERM:vt100] [XFILESEARCHPATH:/usr/dt/app-defaults/%L/Dt] [PWD:/opt/rudder/jetty7] [SHLVL:2] [_:/bin/java] [RUDDER_GENERATION_DATETIME:2017-03-29T14:27:23.982Z] [RUDDER_NODEID:26acd240-5347-4e0c-91e9-d6281341cf2b] [RUDDER_NEXT_POLICIES_DIRECTORY:/var/rudder/share/19a0eaaf-7d56-4251-a697-83942b18df20/share/26acd240-5347-4e0c-91e9-d6281341cf2b/rules.new/cfengine-community] [RUDDER_AGENT_TYPE:cfengine-community]. 
Stdout: ' error: Can't stat file '/var/rudder/ncf/find: '/var/rudder/cfengine-community/state/ncf-exclude-cache-3.10.0/_var_rudder_ncf_common_20_cfe_basics': No such file or directory' for parsing. (stat: No such file or directory)
' 
Stderr: ''

And so on.

This is a race condition in list-compatible-inputs that seems to only happen on CentOS7:

root@server:/opt/rudder/jetty7# /var/rudder/cfengine-community/bin/cf-promises -f /var/rudder/cfengine-community/inputs.new/promises.cf &
[6] 22088
root@server:/opt/rudder/jetty7# /var/rudder/cfengine-community/bin/cf-promises -f /var/rudder/cfengine-community/inputs.new/promises.cf &
[7] 22645
root@server:/opt/rudder/jetty7#    error: Can't stat file '/var/rudder/ncf/find: '/var/rudder/cfengine-community/state/ncf-exclude-cache-3.10.0/_var_rudder_ncf_common_30_generic_methods': No such file or directory' for parsing. (stat: No such file or directory)

This only happen when SELinux is enabled. If we "setenforce 0", the problem completelly disapears.

So, it may not be linked to selinux after all - it was just less frequent when testing that track.

The problem may be that the file "/opt/rudder/etc/agent-capabilities" was not viewed as newer than the cache (because for some reason, the cache was modified in the future), and so one of the parallel process delete the cache, and the next one try to do comparison on deleted files.