Rudder

Hello,

Since maybe 7.2.0 version, I'm witnessing that rudder_relayd daemon is constantly writing tons of "Invalid report" lines in /var/log/daemon.log, though we did not change anything is our setup or rules. We just followed the server and clients upgrades.

Exemples of such lines are : [1]

Our setup is very basic, with one only hub server and around 80 clients (Linux only). This rudder_relayd is running on this same host.

I'd be more than happy to get help to dig into the reasons why it's happening.
This is leading to a complete filling of the /var partition thus stopping the server.

Thank you.

Nicolas ECARNOT

[1]

...
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:52+00:00 R: @@copyGitFile@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@878f24ab-07f3-4db4-aa78-492edef40c42@@878f24ab-07f3-4db4-aa78-492edef40c4232377fd7-02fd-43d0-aab7-28460a91347b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /etc/profile.d is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:52+00:00 R: @@copyGitFile@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@c6d97129-2f5d-46cc-bb8a-4cf62dcc5209@@c6d97129-2f5d-46cc-bb8a-4cf62dcc520932377fd7-02fd-43d0-aab7-28460a91347b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /var/rudder/hooks.d is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:52+00:00 R: @@copyGitFile@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@18fde49f-1416-4bab-ba23-08a670480560@@18fde49f-1416-4bab-ba23-08a67048056032377fd7-02fd-43d0-aab7-28460a91347b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /etc/logrotate.d is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@copyGitFile@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@2322942d-2d2a-4f47-8a4a-4ca6ac898f38@@2322942d-2d2a-4f47-8a4a-4ca6ac898f3832377fd7-02fd-43d0-aab7-28460a91347b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /root is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Insert content into /etc/snmp/snmpd.conf was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Insert content into /etc/snmp/snmpd.conf was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Insert content into /etc/snmp/snmpd.conf was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Insert content into /etc/snmp/snmpd.conf was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Create directory /etc/snmp/scripts was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Create directory /etc/snmp/scripts was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Copying /etc/snmp/scripts/rudder_health.sh from /var/rudder/cfengine-community/inputs/rudderSNMP/1.0/rudder_health.sh was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Copying /etc/snmp/scripts/rudder_health.sh from local /var/rudder/cfengine-community/inputs/rudderSNMP/1.0/rudder_health.sh, recursion level 0 was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Copying /etc/snmp/scripts/rudder_health.sh from local /var/rudder/cfengine-community/inputs/rudderSNMP/1.0/rudder_health.sh was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Ensure permissions mode 755, owner root and group root on /etc/snmp/scripts/rudder_health.sh on type all with 0 recursion level was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Ensure permissions mode 755, owner root and group root on /etc/snmp/scripts/rudder_health.sh was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@copyGitFile@@log_info@@0c595089-0cb4-4cc8-8334-a5ab844d105d@@ac5bb59d-bddc-473b-96b5-988b8f51a730@@ac5bb59d-bddc-473b-96b5-988b8f51a7300c595089-0cb4-4cc8-8334-a5ab844d105d@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /etc/sudoers.d is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@copyGitFile@@log_info@@65485923-9e4b-4b42-88c1-3b2b1039822b@@7a136103-9313-4d9e-9b43-aa57e3ebbfa2@@7a136103-9313-4d9e-9b43-aa57e3ebbfa265485923-9e4b-4b42-88c1-3b2b1039822b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /etc/systemd is a symlink could not be repaired
...