Project

General

Profile

Actions

Bug #17250

closed

Webapp does not regerate policies when webdav password is changed, breaking inventories after 5.0 -> 6.0 upgrade

Added by Victor Héry over 4 years ago. Updated over 3 years ago.

Status:
Released
Priority:
N/A
Category:
System techniques
Target version:
Severity:
Major - prevents use of part of Rudder | no simple workaround
UX impact:
User visibility:
Getting started - demo | first install | Technique editor and level 1 Techniques
Effort required:
Priority:
58
Name check:
Reviewed
Fix check:
Checked
Regression:

Description

Hello,

I have upgraded my rudder server from 5.0.16 to 6.0.5, works fine.

After that, I have enabled the "HTTPS and Syslog" report in settings, and started to upgrade my agent.

But, after upgrading agent from 5.0.16 to 6.0.5, it appears something is broken in the inventory system.

rudder agent run works, and repair any non-compliant rules, but the inventory reporting failed:

# rudder agent inventory
Rudder agent 6.0.5-debian9
Node uuid: c91fc62e-339a-4746-b233-6b47349d4d86
Start execution with config [20200426-111351-77443358]

   error: Finished command related to promiser '/var/rudder/inventories/xmpp-c91fc62e-339a-4746-b233-6b47349d4d86.ocs\..*' -- an error occurred, returned 22
   error: Transformer '/var/rudder/inventories/xmpp-c91fc62e-339a-4746-b233-6b47349d4d86.ocs.sign' => '/opt/rudder/bin/curl --tlsv1.2 --location --insecure --fail --silent --proxy '' --user rudder:rudder --upload-file /var/rudder/inventories/xmpp-c91fc62e-339a-4746-b233-6b47349d4d86.ocs.sign https://RUDDER_SERVER/inventory-updates/' returned error
   error: Finished command related to promiser '/var/rudder/inventories/xmpp-c91fc62e-339a-4746-b233-6b47349d4d86.ocs\..*' -- an error occurred, returned 22
   error: Transformer '/var/rudder/inventories/xmpp-c91fc62e-339a-4746-b233-6b47349d4d86.ocs.gz' => '/opt/rudder/bin/curl --tlsv1.2 --location --insecure --fail --silent --proxy '' --user rudder:rudder --upload-file /var/rudder/inventories/xmpp-c91fc62e-339a-4746-b233-6b47349d4d86.ocs.gz https://RUDDER_SERVER/inventory-updates/' returned error
M| State         Technique                 Component                 Key                Message
E| error         Inventory                 inventory                                    Could not send the inventory
   error: Method 'sendInventory' failed in some repairs
   error: Method 'doInventory_always' failed in some repairs
info     Rudder agent was run on a subset of policies - not all policies were checked

If I run the curl manually, without the --silent, it reports a 401:

# /opt/rudder/bin/curl --tlsv1.2 --location --insecure --fail --proxy '' --user rudder:rudder --upload-file /var/rudder/inventories/xmpp-c91fc62e-339a-4746-b233-6b47349d4d86.ocs.gz https://RUDDER_SERVER/inventory-updates/
curl: (22) The requested URL returned error: 401

And indeed, if I connect manually to https://RUDDER_SERVER/inventory-updates using rudder/rudder as credentials, I get a 401.

I found nowhere how to configure this credential nor on server webui or agent configuration files, but I found (by looking in apache vhost) that the htpasswd file is here:

/opt/rudder/etc/htpasswd-webdav

And indeed, the password stored inside is not rudder:

# htpasswd -v /opt/rudder/etc/htpasswd-webdav rudder
Enter password: rudder
password verification failed

If I backup the file, and change it to match rudder/rudder, the inventory works fine again:

## On rudder server
# htpasswd /opt/rudder/etc/htpasswd-webdav rudder
New password: 
Re-type new password: 
Updating password for user rudder

## On rudder agent
# rudder agent inventory
Rudder agent 6.0.5-debian9
Node uuid: c91fc62e-339a-4746-b233-6b47349d4d86
Start execution with config [20200426-111351-77443358]

M| State         Technique                 Component                 Key                Message
E| compliant     Inventory                 inventory                                    The inventory has been successfully sent

But I am not sure that changing this password will not break somethin else, and by the way after some minutes, the password in "/opt/rudder/etc/htpasswd-webdav" is updated automatically (possibly by rudder itself?) and the inventory fails again.

At the moment, I do not know how to change this password, either on the server or on the agent, as it seems this user is outside the common "user" system: https://docs.rudder.io/reference/6.0/administration/users.html

In addition to that, as the password is hashed in htpasswd file, I do not know how to get it on the server to configure agent accordingly :-/

Do you know if there is something to do to configure the newly upgraded agent with the correct credential, and where to find these credentials on the server?

Do not hesitate to tell if you need any other test or details :)

Thanks a lot!


Subtasks 1 (0 open1 closed)

Bug #18226: remove unused /opt/rudder/etc/policy-update-running during upgradeReleasedBenoît PECCATTEActions

Related issues 2 (1 open1 closed)

Related to Rudder - Bug #18217: /opt/rudder/etc/htpasswd-webdav incorrect permissions after updateResolvedActions
Related to Rudder - Architecture #18221: deprecate flag files /opt/rudder/etc/policy-update-running and /opt/rudder/etc/trigger-policy-generation"NewActions
Actions #1

Updated by Victor Héry over 4 years ago

  • Subject changed from After upload from rudder 5.0.16 to 6.0.5, https inventory does not work with 401 to After upload from rudder 5.0.17 to 6.0.5, https inventory does not work with 401
Actions #2

Updated by Victor Héry over 4 years ago

  • Subject changed from After upload from rudder 5.0.17 to 6.0.5, https inventory does not work with 401 to After upgrade from rudder 5.0.17 to 6.0.5, https inventory does not work with 401
  • Severity set to Major - prevents use of part of Rudder | no simple workaround
Actions #3

Updated by Alexis Mousset over 4 years ago

The password should be distributed as part of the policies (and is in clear text on the server in /opt/rudder/etc/rudder-passwords.conf, named "RUDDER_WEBDAV_PASSWORD").

This may happen because no policy update happened since server upgrade, is it possible? Does rudder agent update succeed on the nodes?

Actions #4

Updated by Alexis Mousset over 4 years ago

Could you also try restarting the apache service on the server and test again? Could be a missing restart in postinst.

Actions #5

Updated by Victor Héry over 4 years ago

Hello,

Restarting apache does not help unfortunately, the problem is still here.

I think your point about policies could be a good idea, I do not have the passwords file on the node, and agent update does nothing about it:

# ls /opt/rudder/etc/rudder-passwords.conf
/opt/rudder/etc/rudder-passwords.conf: Aucun fichier ou dossier de ce type
# rudder agent update
ok: Rudder agent promises were updated.
# ls /opt/rudder/etc/rudder-passwords.conf
/opt/rudder/etc/rudder-passwords.conf: Aucun fichier ou dossier de ce type

However, I have indeed the file on the rudder server, but the agent on the server node does not work with the same error, even if the file is here :-/

The agent still use the rudder:rudder credentials:

# ls /opt/rudder/etc/rudder-passwords.conf
/opt/rudder/etc/rudder-passwords.conf
# rudder agent inventory
Rudder agent 6.0.5-debian9
Node uuid: root
Start execution with config [20200426-114025-f46d2b21]

   error: Finished command related to promiser '/var/rudder/inventories/rudder-root.ocs\..*' -- an error occurred, returned 22
   error: Transformer '/var/rudder/inventories/rudder-root.ocs.gz' => '/opt/rudder/bin/curl --tlsv1.2 --location --insecure --fail --silent --proxy '' --user rudder:rudder --upload-file /var/rudder/inventories/rudder-root.ocs.gz https://192.168.201.173/inventory-updates/' returned error
   error: Finished command related to promiser '/var/rudder/inventories/rudder-root.ocs\..*' -- an error occurred, returned 22

Just to be sure I have fully rebooted the server (in case some other restart could have been missed), but it does not help.

I have copy/paste the passwords file manually to the node and restarted rudder services (rudder-agent and rudder-cf-serverd.service) but the inventory still use rudder:rudder

There is something odd 😅

Actions #6

Updated by Alexis Mousset over 4 years ago

/opt/rudder/etc/rudder-passwords.conf is only supposed to exist on the root server.

Agents have the password in their policies:

grep DAVPASSWORD /var/rudder/cfengine-community/inputs/rudder.json

And it should match the passord in /opt/rudder/etc/rudder-passwords.conf (that should also be the one in htpasswd file).

What do you have in the node's policies?

Actions #7

Updated by Alexis Mousset over 4 years ago

  • Category set to System techniques
  • Assignee set to Alexis Mousset
  • Target version set to 6.0.6
  • User visibility set to Getting started - demo | first install | Technique editor and level 1 Techniques
  • Priority changed from 0 to 70
Actions #8

Updated by Victor Héry over 4 years ago

Thanks for the help!

Indeed, I have this file, and it contains "rudder" as password:

# grep DAVPASSWORD /var/rudder/cfengine-community/inputs/rudder.json
  "DAVPASSWORD":"rudder",

This is the same on the agent node, the server node, and even on old 5.0.17 nodes 🤔

Not upgraded nodes have this file with rudder as password too, and they indeed have the error to send inventory, I hadn'nt noticed that Oo

I am sure it worked before the server update to 6.0, as my compliance reports for all nodes were complete.

So it's perhaps a problem with the server and not with the agent as I first guess.

I have followed the doc from https://docs.rudder.io/reference/6.0/installation/upgrade/debian.html

And ran "rudder server upgrade-techniques -o" on the server. Not sure if it impacts policies thought.

If I run "rudder agent update" then "rudder agent run" on node, there is a repair for the policy:

E| repaired      Common                    Update                                       Policy or configuration library were updated

But the webdav password is still "rudder".

In fact, each time I run "rudder agent update" (with or without -f), this promise is Repaired, is this normal?

Actions #9

Updated by Alexis Mousset over 4 years ago

Victor Héry wrote in #note-8:

But the webdav password is still "rudder".

Ok, so it's the problem here. The following commands:

  • grep "rudder.webdav.password" /opt/rudder/etc/rudder-web.properties
  • grep "DAV_PASSWORD" /opt/rudder/etc/rudder-passwords.conf

should return the same password value. I think they are different and rudder-web.properties contains rudder instead of the actual password, which cases the problem. In this case you can manually update rudder-web.properties to check if it fixes the problem.

In fact, each time I run "rudder agent update" (with or without -f), this promise is Repaired, is this normal?

The "repaired" state persists for 5 minutes after the actual repair, so it's probably normal.

Actions #10

Updated by Victor Héry over 4 years ago

I confirm that both command return the same password, and it's the good one.

I have tried to set manually this password on the node in /var/rudder/cfengine-community/inputs/rudder.json to see if it works, and restart rudder services.

But, when I run rudder agent update then rudder agent run, the file is modified back to "rudder" as password.

So it seems there is something somewhere that think this password is rudder, even if the server files are corrects 😅

I am sorry, this seems a tricky one :-/

Not sure if it could help, but all these nodes came from debian 6 and rudder 4.
I have regularly updated them to now debian 9 and rudder 5.0.17/6.0.3, perhaps a post-inst script on one previous version could have been missed?

Actions #11

Updated by Alexis Mousset over 4 years ago

Does

grep davpw /var/rudder/cfengine-community/inputs/common/1.0/common.cf

(on the node) return "rudder" too?

Silly question, but is the generation status in the webapp green?

Actions #12

Updated by Victor Héry over 4 years ago

Yes, it returns rudder on all nodes (5 and 6 nodes):

# grep davpw /var/rudder/cfengine-community/inputs/common/1.0/common.cf
    "davpw"                 string => "rudder";

In the web interface, all regarding rudder policy are green:

However, there is an error, I think regarding the inventory problem, but not sure:

All nodes have this error message, but the compliance are green

Actions #13

Updated by Alexis Mousset over 4 years ago

Could you try the "Regenrate all policies" in the generation status menu? Maybe a cache problem in the web application.

Actions #14

Updated by Victor Héry over 4 years ago

Wow, it works!

After regeneration of the policies, the password is ok!

And after running rudder agent update on nodes, the password is also ok in /var/rudder/cfengine-community/inputs/common/1.0/common.cf and /var/rudder/cfengine-community/inputs/rudder.json

And, rudder agent inventory works!

Do you know if there is a way to do that regeneration with the cli, when upgrading from 5 to 6?

Anyway, thanks a lot for the help 🙏

Actions #15

Updated by Alexis Mousset over 4 years ago

  • Subject changed from After upgrade from rudder 5.0.17 to 6.0.5, https inventory does not work with 401 to Webapp does not regerate policies when webdav password is changed, breaking inventories after 5.0 -> 6.0 upgrade

Good!

Renaming the ticket then, we need to understand why it did not detect the change.

You can force regeneration with the API: https://docs.rudder.io/api/#operation/regeneratePolicies

Something like:

curl --header "X-API-Token: yourToken" --request POST 'https://rudder.example.com/rudder/api/latest/system/regenerate/policies'

Actions #16

Updated by Victor Héry over 4 years ago

Good I will keep this trick somewhere :-D

Regarding the cause of the problem, I have still a backup (while not rotated yet) of the 5.0.17 server, if you want I may restore and do some tests, do not hesitate to tell me if you have commands or verification to tests!

Actions #17

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 6.0.6 to 6.0.7
Actions #18

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 6.0.7 to 6.0.8
  • Priority changed from 70 to 69
Actions #19

Updated by François ARMAND over 4 years ago

  • Priority changed from 69 to 68
Actions #20

Updated by Vincent MEMBRÉ over 4 years ago

  • Target version changed from 6.0.8 to 6.0.9
Actions #21

Updated by Vincent MEMBRÉ about 4 years ago

  • Target version changed from 6.0.9 to 6.0.10
  • Priority changed from 68 to 66
Actions #22

Updated by Nicolas CHARLES about 4 years ago

  • Related to Bug #18217: /opt/rudder/etc/htpasswd-webdav incorrect permissions after update added
Actions #23

Updated by Nicolas CHARLES about 4 years ago

Simply triggering a policy generation at webapp start would solve the issue
It would check if any variable changed, and if so regenerate

Actions #24

Updated by Nicolas CHARLES about 4 years ago

  • Status changed from New to In progress
  • Assignee changed from Alexis Mousset to Nicolas CHARLES
Actions #25

Updated by Nicolas CHARLES about 4 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas CHARLES to François ARMAND
  • Pull Request set to https://github.com/Normation/rudder/pull/3217
Actions #26

Updated by Nicolas CHARLES about 4 years ago

  • Assignee changed from François ARMAND to Alexis Mousset
  • Pull Request changed from https://github.com/Normation/rudder/pull/3217 to https://github.com/Normation/rudder/pull/3218
Actions #27

Updated by Nicolas CHARLES about 4 years ago

  • Assignee changed from Alexis Mousset to François ARMAND
Actions #28

Updated by Nicolas CHARLES about 4 years ago

  • Assignee changed from François ARMAND to Alexis Mousset
Actions #29

Updated by Nicolas CHARLES about 4 years ago

  • Status changed from Pending technical review to Pending release
  • Priority changed from 66 to 65
Actions #30

Updated by Nicolas CHARLES about 4 years ago

  • Fix check changed from To do to Checked
Actions #31

Updated by Alexis Mousset about 4 years ago

  • Priority changed from 65 to 64
  • Name check changed from To do to Reviewed
Actions #32

Updated by Vincent MEMBRÉ over 3 years ago

  • Priority changed from 64 to 58

This bug has been fixed in Rudder 6.0.10, 6.1.6, 6.2.0~beta1 which were released by the end of October 2020.

Actions #33

Updated by Vincent MEMBRÉ over 3 years ago

  • Related to Architecture #18221: deprecate flag files /opt/rudder/etc/policy-update-running and /opt/rudder/etc/trigger-policy-generation" added
Actions #34

Updated by Vincent MEMBRÉ over 3 years ago

  • Status changed from Pending release to Released
Actions

Also available in: Atom PDF