Project

General

Profile

Actions

Bug #22029

closed

rudder_relayd filling /var/log/daemon.log (Invalid report)

Added by Nicolas Ecarnot 3 months ago. Updated 2 months ago.

Status:
Released
Priority:
N/A
Category:
Relay server or API
Target version:
Severity:
Major - prevents use of part of Rudder | no simple workaround
UX impact:
User visibility:
Effort required:
Priority:
0
Regression:
No

Description

Hello,

Since maybe 7.2.0 version, I'm witnessing that rudder_relayd daemon is constantly writing tons of "Invalid report" lines in /var/log/daemon.log, though we did not change anything is our setup or rules. We just followed the server and clients upgrades.

Exemples of such lines are : [1]

Our setup is very basic, with one only hub server and around 80 clients (Linux only). This rudder_relayd is running on this same host.

I'd be more than happy to get help to dig into the reasons why it's happening.
This is leading to a complete filling of the /var partition thus stopping the server.

Thank you.

Nicolas ECARNOT

[1]

...
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:52+00:00 R: @@copyGitFile@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@878f24ab-07f3-4db4-aa78-492edef40c42@@878f24ab-07f3-4db4-aa78-492edef40c4232377fd7-02fd-43d0-aab7-28460a91347b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /etc/profile.d is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:52+00:00 R: @@copyGitFile@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@c6d97129-2f5d-46cc-bb8a-4cf62dcc5209@@c6d97129-2f5d-46cc-bb8a-4cf62dcc520932377fd7-02fd-43d0-aab7-28460a91347b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /var/rudder/hooks.d is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:52+00:00 R: @@copyGitFile@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@18fde49f-1416-4bab-ba23-08a670480560@@18fde49f-1416-4bab-ba23-08a67048056032377fd7-02fd-43d0-aab7-28460a91347b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /etc/logrotate.d is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@copyGitFile@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@2322942d-2d2a-4f47-8a4a-4ca6ac898f38@@2322942d-2d2a-4f47-8a4a-4ca6ac898f3832377fd7-02fd-43d0-aab7-28460a91347b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /root is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Insert content into /etc/snmp/snmpd.conf was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Insert content into /etc/snmp/snmpd.conf was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Insert content into /etc/snmp/snmpd.conf was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Insert content into /etc/snmp/snmpd.conf was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Create directory /etc/snmp/scripts was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Create directory /etc/snmp/scripts was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Copying /etc/snmp/scripts/rudder_health.sh from /var/rudder/cfengine-community/inputs/rudderSNMP/1.0/rudder_health.sh was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Copying /etc/snmp/scripts/rudder_health.sh from local /var/rudder/cfengine-community/inputs/rudderSNMP/1.0/rudder_health.sh, recursion level 0 was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Copying /etc/snmp/scripts/rudder_health.sh from local /var/rudder/cfengine-community/inputs/rudderSNMP/1.0/rudder_health.sh was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Ensure permissions mode 755, owner root and group root on /etc/snmp/scripts/rudder_health.sh on type all with 0 recursion level was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@rudderSNMP@@log_info@@01f16456-4658-4990-a197-4fb53c3ca329@@648f52b4-0733-4e1d-a090-459431dc5af6@@648f52b4-0733-4e1d-a090-459431dc5af601f16456-4658-4990-a197-4fb53c3ca329@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Ensure permissions mode 755, owner root and group root on /etc/snmp/scripts/rudder_health.sh was correct
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@copyGitFile@@log_info@@0c595089-0cb4-4cc8-8334-a5ab844d105d@@ac5bb59d-bddc-473b-96b5-988b8f51a730@@ac5bb59d-bddc-473b-96b5-988b8f51a7300c595089-0cb4-4cc8-8334-a5ab844d105d@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /etc/sudoers.d is a symlink could not be repaired
Oct 27 09:50:54 sit-conf-prd01 rudder-relayd[548]:  WARN report{queue_id=4EBCB89D1B48569033321E5E30C5BF35}: rudder_relayd::data::runlog: Invalid report: 2022-10-27T07:50:53+00:00 R: @@copyGitFile@@log_info@@65485923-9e4b-4b42-88c1-3b2b1039822b@@7a136103-9313-4d9e-9b43-aa57e3ebbfa2@@7a136103-9313-4d9e-9b43-aa57e3ebbfa265485923-9e4b-4b42-88c1-3b2b1039822b@@None@@@@2022-10-27 07:50:51+00:00##9bb6927f-9219-43b0-ad23-947525f3f482@#Check if /etc/systemd is a symlink could not be repaired
...

Actions #1

Updated by Alexis Mousset 3 months ago

  • Assignee set to Alexis Mousset
  • Target version set to 7.2.2
Actions #2

Updated by Alexis Mousset 3 months ago

  • Status changed from New to In progress
Actions #3

Updated by Alexis Mousset 3 months ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Alexis Mousset to Nicolas CHARLES
  • Pull Request set to https://github.com/Normation/rudder/pull/4553
Actions #4

Updated by Alexis Mousset 3 months ago

There are two problems:

  • The @@@@ should actually be @@None@@, the None being the key_value. It is incorrect because of old bug in the way we set reporting metadata, that were surfaced by #21736
  • You should allow empty fields to avoid parsing errors in the meantime. It is what is done by the change in this ticket.
Actions #5

Updated by Alexis Mousset 3 months ago

  • Target version changed from 7.2.2 to 7.1.8
Actions #6

Updated by Alexis Mousset 3 months ago

  • Status changed from Pending technical review to In progress
  • Assignee changed from Nicolas CHARLES to Alexis Mousset
Actions #7

Updated by Alexis Mousset 3 months ago

  • Status changed from In progress to Pending release
Actions #8

Updated by Nicolas Ecarnot 3 months ago

Alexis Mousset wrote in #note-7:

Applied in changeset rudder|423a833a6f43667d29a6177ba88c226aa48ef3f8.

Not being comfortable with the CI workflow, I don't know in which version this patch will be available.
At present, in 7.2.1, this hasn't been patched, has it?

Actions #9

Updated by Alexis Mousset 3 months ago

No, not released yet. The ticket status, here "Pending release", gives the current state.

As a workaround you can take the /opt/rudder/bin/rudder-relayd file from a nightly package in https://repository.rudder.io/apt/nightly/pool/main/r/rudder-server-relay/ and replce it on your server.

Actions #10

Updated by Nicolas Ecarnot 3 months ago

Alexis Mousset wrote in #note-9:

No, not released yet. The ticket status, here "Pending release", gives the current state.

As a workaround you can take the /opt/rudder/bin/rudder-relayd file from a nightly package in https://repository.rudder.io/apt/nightly/pool/main/r/rudder-server-relay/ and replce it on your server.

OK I tried your advice. Thank you Alexis.
So far, it is looking good.

(Strange that I did activate the "Watch" button on this Redmine web page, though I received no notification on your answer...)

Actions #12

Updated by Vincent MEMBRÉ 2 months ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 7.1.8 and 7.2.2 which were released today.

Actions

Also available in: Atom PDF