Bug #26867: Repeated transient / random errors with the sshKeyDistribution standard technique - Rudder - Issue Tracker

Actions

Copy link

Bug #26867

open

Repeated transient / random errors with the sshKeyDistribution standard technique

Added by Michel BOUISSOU 27 days ago. Updated 9 days ago.

Status:

New

Priority:

Assignee:

Michel BOUISSOU

Category:

Techniques

Target version:

8.2.7

Pull Request:

Severity:

UX impact:

User visibility:

Effort required:

Priority:

Name check:

To do

Fix check:

To do

Regression:

Description

On many of my nodes using a single, simple directive that uses the standard sshKeyDistribution technique :
- Setting SSH authoriszed keys for 2 users : myself and root
- With the option “Remove other keys“ = Yes

Nothing else AFAIK changing any thing with the authorized_keys on those machines.

- Most of the times the Rudder agent runs good with everything compliant.
- But every so often, the Rudder agents reports that :
-- The SSH key for each user was “repaired”
-- The SSH keys for each user “could not be flushed”

I have no explanation for such behaviour.

See attached screenshots.

Files

Download all files

sshKeyDistrbution_errors_250504d.png (228 KB) sshKeyDistrbution_errors_250504d.png	Used directive parameters	Michel BOUISSOU, 2025-05-04 19:57
sshKeyDistrbution_errors_250504a.png (465 KB) sshKeyDistrbution_errors_250504a.png	Agent run with errors	Michel BOUISSOU, 2025-05-04 19:57
sshKeyDistrbution_errors_250504b.png (185 KB) sshKeyDistrbution_errors_250504b.png	Compliance report with errors	Michel BOUISSOU, 2025-05-04 19:57
sshKeyDistrbution_errors_250504c.png (489 KB) sshKeyDistrbution_errors_250504c.png	Agent run without errors	Michel BOUISSOU, 2025-05-04 19:57

Actions

Copy link

Updated by Vincent MEMBRÉ 25 days ago

Target version changed from 8.3.1 to 8.3.2

Actions

Copy link

Updated by François ARMAND 16 days ago

Assignee set to Michel BOUISSOU
Target version changed from 8.3.2 to 8.2.7

We need more info with rudder agent history

Actions

Copy link

Updated by Michel BOUISSOU 16 days ago

Assignee changed from Michel BOUISSOU to François ARMAND

“rudder agent history” doesn't tell much.

Most of the times the agent runs with “Compliant” for SSH keys, but every so often :

It shows 2 repairs for “sshKeyDistribution” where it seems to have modified the files contents (where it shouldn't as nothing has changed) then the 2 following errors :

E| error         sshKeyDistribution        Flush SSH file            michel michel RSA  The keys for user michel could not be flushed
E| error         sshKeyDistribution        Flush SSH file            root michel RSA    The keys for user root could not be flushed

The relevant parts of cfengine-community/outputs/cf_vpn1__1747286851_Thu_May_15_07_27_31_2025_0 (on considered node) seems to be :

Close to the beginning :

rudder     info: Deleted the promised line 1 'ssh-rsa <some_SSH_public_key>' from /var/rudder/tmp/check_ssh_key_distribution//michel.authorized_keys.tmp
rudder     info: delete_lines promise '.*' repaired
rudder     info: Moved '/var/rudder/tmp/check_ssh_key_distribution//michel.authorized_keys.tmp.cf-before-edit' to repository location '/var/rudder/modified-files/_var_rudder_tmp_check_ssh_key_distribution__michel_authorized_keys_tmp_cf_before_edit'
rudder     info: Edited file '/var/rudder/tmp/check_ssh_key_distribution//michel.authorized_keys.tmp'
rudder     info: Deleted the promised line 1 'ssh-rsa <some_SSH_public_key>' from /var/rudder/tmp/check_ssh_key_distribution//root.authorized_keys.tmp
rudder     info: delete_lines promise '.*' repaired
rudder     info: Moved '/var/rudder/tmp/check_ssh_key_distribution//root.authorized_keys.tmp.cf-before-edit' to repository location '/var/rudder/modified-files/_var_rudder_tmp_check_ssh_key_distribution__root_authorized_keys_tmp_cf_before_edit'
rudder     info: Edited file '/var/rudder/tmp/check_ssh_key_distribution//root.authorized_keys.tmp'

Then :

rudder     info: insert_lines promise 'ssh-rsa <some_SSH_public_key>' repaired
rudder     info: Moved '/home/michel/.ssh/authorized_keys.cf-before-edit' to repository location '/var/rudder/modified-files/_home_michel__ssh_authorized_keys_cf_before_edit'
rudder     info: Edited file '/home/michel/.ssh/authorized_keys'
rudder     info: Inserted the promised line 'ssh-rsa <some_SSH_public_key>' into '/root/.ssh/authorized_keys' after locator
rudder     info: insert_lines promise 'ssh-rsa <some_SSH_public_key>' repaired
rudder     info: Moved '/root/.ssh/authorized_keys.cf-before-edit' to repository location '/var/rudder/modified-files/_root__ssh_authorized_keys_cf_before_edit'
rudder     info: Edited file '/root/.ssh/authorized_keys'
   error: File '/var/rudder/tmp/check_ssh_key_distribution//michel.authorized_keys.tmp' was marked for editing but could not be opened
   error: Errors encountered when actuating files promise '/var/rudder/tmp/check_ssh_key_distribution//michel.authorized_keys.tmp'
   error: File '/var/rudder/tmp/check_ssh_key_distribution//root.authorized_keys.tmp' was marked for editing but could not be opened
   error: Errors encountered when actuating files promise '/var/rudder/tmp/check_ssh_key_distribution//root.authorized_keys.tmp'
R: @@sshKeyDistribution@@result_repaired@@0caa2ab2-67a0-441e-80b5-be02e602b678@@2df3bf3a-d2d5-4c3d-97df-d8df68d2bcf2@@0@@SSH key@@michel michel RSA@@2025-05-15 05:27:33+00:00##ddb6665c-35d4-48c4-af44-f7d9cc0a6f29@#SSH key "michel michel RSA" for user michel was repaired
R: @@sshKeyDistribution@@result_repaired@@0caa2ab2-67a0-441e-80b5-be02e602b678@@2df3bf3a-d2d5-4c3d-97df-d8df68d2bcf2@@0@@SSH key@@root michel RSA@@2025-05-15 05:27:33+00:00##ddb6665c-35d4-48c4-af44-f7d9cc0a6f29@#SSH key "root michel RSA" for user root was repaired

Then close to the end:

   error: Can't stat file '/var/rudder/tmp/check_ssh_key_distribution//michel.authorized_keys.tmp' on 'localhost' in files.copy_from promise, it may be missing or access may not be authorized
   error: Errors encountered when actuating files promise '/home/michel/.ssh/authorized_keys'
   error: Can't stat file '/var/rudder/tmp/check_ssh_key_distribution//root.authorized_keys.tmp' on 'localhost' in files.copy_from promise, it may be missing or access may not be authorized
   error: Errors encountered when actuating files promise '/root/.ssh/authorized_keys'
R: @@sshKeyDistribution@@result_error@@0caa2ab2-67a0-441e-80b5-be02e602b678@@2df3bf3a-d2d5-4c3d-97df-d8df68d2bcf2@@0@@Flush SSH file@@michel michel RSA@@2025-05-15 05:27:33+00:00##ddb6665c-35d4-48c4-af44-f7d9cc0a6f29@#The keys for user michel could not be flushed
R: @@sshKeyDistribution@@result_error@@0caa2ab2-67a0-441e-80b5-be02e602b678@@2df3bf3a-d2d5-4c3d-97df-d8df68d2bcf2@@0@@Flush SSH file@@root michel RSA@@2025-05-15 05:27:33+00:00##ddb6665c-35d4-48c4-af44-f7d9cc0a6f29@#The keys for user root could not be flushed

Actions

Copy link