Actions
Bug #22029
closed
rudder_relayd filling /var/log/daemon.log (Invalid report)
Pull Request:
Severity:
Major - prevents use of part of Rudder | no simple workaround
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
To do
Fix check:
Checked
Regression:
No
Description
Hello,
Since maybe 7.2.0 version, I'm witnessing that rudder_relayd daemon is constantly writing tons of "Invalid report" lines in /var/log/daemon.log, though we did not change anything is our setup or rules. We just followed the server and clients upgrades.
Exemples of such lines are : [1]
Our setup is very basic, with one only hub server and around 80 clients (Linux only). This rudder_relayd is running on this same host.
I'd be more than happy to get help to dig into the reasons why it's happening.
This is leading to a complete filling of the /var partition thus stopping the server.
Thank you.
Nicolas ECARNOT
[1]
...
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:52+00:00 R: @@copyGitFile@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@878f24ab-07f3-4db4-aa78-492edef40c42@@878f24ab-07f3-4db4-aa78-492edef40c4232377fd7-02fd-43d0-aab7-28460a91347b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /etc/profile.d is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:52+00:00 R: @@copyGitFile@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@c6d97129-2f5d-46cc-bb8a-4cf62dcc5209@@c6d97129-2f5d-46cc-bb8a-4cf62dcc520932377fd7-02fd-43d0-aab7-28460a91347b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /var/rudder/hooks.d is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:52+00:00 R: @@copyGitFile@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@18fde49f-1416-4bab-ba23-08a670480560@@18fde49f-1416-4bab-ba23-08a67048056032377fd7-02fd-43d0-aab7-28460a91347b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /etc/logrotate.d is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@copyGitFile@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@2322942d-2d2a-4f47-8a4a-4ca6ac898f38@@2322942d-2d2a-4f47-8a4a-4ca6ac898f3832377fd7-02fd-43d0-aab7-28460a91347b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /root is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Insert content into /etc/snmp/snmpd.conf was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Insert content into /etc/snmp/snmpd.conf was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Insert content into /etc/snmp/snmpd.conf was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Insert content into /etc/snmp/snmpd.conf was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Create directory /etc/snmp/scripts was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Create directory /etc/snmp/scripts was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Copying /etc/snmp/scripts/rudder_health.sh from /var/rudder/cfengine-community/inputs/rudderSNMP/1.0/rudder_health.sh was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Copying /etc/snmp/scripts/rudder_health.sh from local /var/rudder/cfengine-community/inputs/rudderSNMP/1.0/rudder_health.sh, recursion level 0 was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Copying /etc/snmp/scripts/rudder_health.sh from local /var/rudder/cfengine-community/inputs/rudderSNMP/1.0/rudder_health.sh was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Ensure permissions mode 755, owner root and group root on /etc/snmp/scripts/rudder_health.sh on type all with 0 recursion level was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Ensure permissions mode 755, owner root and group root on /etc/snmp/scripts/rudder_health.sh was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@copyGitFile@@log_info@@0c595089-0cb4-4cc8-8334-a5ab844d105d@@ac5bb59d-bddc-473b-96b5-988b8f51a730@@ac5bb59d-bddc-473b-96b5-988b8f51a7300c595089-0cb4-4cc8-8334-a5ab844d105d@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /etc/sudoers.d is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]: WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@copyGitFile@@log_info@@65485923-9e4b-4b42-88c1-3b2b1039822b@@7a136103-9313-4d9e-9b43-aa57e3ebbfa2@@7a136103-9313-4d9e-9b43-aa57e3ebbfa265485923-9e4b-4b42-88c1-3b2b1039822b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /etc/systemd is a symlink could not be repaired
...
Updated by Alexis Mousset over 1 year ago
- Assignee set to Alexis Mousset
- Target version set to 7.2.2
Updated by Alexis Mousset over 1 year ago
- Status changed from New to In progress
Updated by Alexis Mousset over 1 year ago
- Status changed from In progress to Pending technical review
- Assignee changed from Alexis Mousset to Nicolas CHARLES
- Pull Request set to https://github.com/Normation/rudder/pull/4553
Updated by Alexis Mousset over 1 year ago
There are two problems:
- The @@@@ should actually be @@None@@, the
Nonebeing the key_value. It is incorrect because of old bug in the way we set reporting metadata, that were surfaced by #21736 - You should allow empty fields to avoid parsing errors in the meantime. It is what is done by the change in this ticket.
Updated by Alexis Mousset over 1 year ago
- Target version changed from 7.2.2 to 7.1.8
Updated by Alexis Mousset over 1 year ago
- Status changed from Pending technical review to In progress
- Assignee changed from Nicolas CHARLES to Alexis Mousset
Updated by Alexis Mousset over 1 year ago
- Status changed from In progress to Pending release
Applied in changeset rudder|423a833a6f43667d29a6177ba88c226aa48ef3f8.
Updated by Nicolas Ecarnot over 1 year ago
Alexis Mousset wrote in #note-7:
Applied in changeset rudder|423a833a6f43667d29a6177ba88c226aa48ef3f8.
Not being comfortable with the CI workflow, I don't know in which version this patch will be available.
At present, in 7.2.1, this hasn't been patched, has it?
Updated by Alexis Mousset over 1 year ago
No, not released yet. The ticket status, here "Pending release", gives the current state.
As a workaround you can take the /opt/rudder/bin/rudder-relayd file from a nightly package in https://repository.rudder.io/apt/nightly/pool/main/r/rudder-server-relay/ and replce it on your server.
Updated by Nicolas Ecarnot over 1 year ago
Alexis Mousset wrote in #note-9:
No, not released yet. The ticket status, here "Pending release", gives the current state.
As a workaround you can take the
/opt/rudder/bin/rudder-relaydfile from a nightly package in https://repository.rudder.io/apt/nightly/pool/main/r/rudder-server-relay/ and replce it on your server.
OK I tried your advice. Thank you Alexis.
So far, it is looking good.
(Strange that I did activate the "Watch" button on this Redmine web page, though I received no notification on your answer...)
Updated by Alexis Mousset over 1 year ago
- Fix check changed from To do to Checked
Updated by Vincent MEMBRÉ over 1 year ago
- Status changed from Pending release to Released
This bug has been fixed in Rudder 7.1.8 and 7.2.2 which were released today.
Actions