Bug #16641: Error 404 when accessing shared-files (relayd) - Rudder - Issue Tracker

Actions

Copy link

Bug #16641

closed

Bug #16549: Shared files are broken in relayd

Error 404 when accessing shared-files (relayd)

Added by Samuel Chesnel almost 5 years ago. Updated over 4 years ago.

Status:

Released

Priority:

N/A

Assignee:

Benoît PECCATTE

Category:

Relay server or API

Target version:

6.0.3

Pull Request:

https://github.com/Normation/rudder/p...

Severity:

Major - prevents use of part of Rudder | no simple workaround

UX impact:

User visibility:

Operational - other Techniques | Rudder settings | Plugins

Effort required:

Priority:

Name check:

Reviewed

Fix check:

Checked

Regression:

Description

Hello,
I'm running Rudder 6.0.2 on a CentOS8 server and I'm using the centreon plugin.
First, I encountered the following error when trying to apply a Centreon template :

E| compliant DeploiementGlobalR Monitoring template Global_R Host follows Global_R monitoring template was correct
error: Command related to promiser 'echo | cat /var/rudder/tmp/rudder_monitoring.csv.sign - /var/rudder/tmp/rudder_monitoring.csv | /opt/rudder/bin/curl' returned code defined as promise failed 22
error: Method 'sharedfile_to_node' failed in some repairs
E| repaired Common Monitoring Share Rudder monitoring information with the server was repaired
error: Method 'monitoring_hook_post' failed in some repairs

Then thanks to your help I got to patch this error by doing the following actions :
I applied the patch showed on https://github.com/Normation/rudder-techniques/pull/1578/files/7a5675ac060c97fd9f548a8f4aea74363104443b on the file /var/rudder/configuration-repository/techniques/system/server-roles/1.0/relayd.conf.tpl, then run cd /var/rudder/configuration-repository, git add techniques/system/server-roles/1.0/relayd.conf.tpl && git commit -m "Fix 16549", and rudder server reload-techniques.

The issue is still present and i got the following error on my relayd service:

Jan 29 16:15:52 hostname rudder-relayd[29531]:  INFO relayd::relay-api: 127.0.0.1:38578 "HEAD /rudder/relay-api/1/shared-files/root/3ce13542-195b-482a-9e2e-7274add46528/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 64.338µs
Jan 29 16:15:52 hostname rudder-relayd[29531]:  INFO relayd::relay-api: 127.0.0.1:38578 "PUT /rudder/relay-api/1/shared-files/root/3ce13542-195b-482a-9e2e-7274add46528/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 41.57µs
Jan 29 16:17:48 hostname rudder-relayd[29531]:  INFO relayd::relay-api: 127.0.0.1:38614 "HEAD /rudder/relay-api/1/shared-files/root/d19c9d81-156d-4bac-8704-8cb519779918/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 72.778µs
Jan 29 16:17:48 hostname rudder-relayd[29531]:  INFO relayd::relay-api: 127.0.0.1:38614 "PUT /rudder/relay-api/1/shared-files/root/d19c9d81-156d-4bac-8704-8cb519779918/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 59.228µs
Jan 29 16:20:09 hostname rudder-relayd[29531]:  INFO relayd::relay-api: 127.0.0.1:38614 "HEAD /rudder/relay-api/1/shared-files/root/3da496fa-40f1-4e66-9018-7d4b2f2f5d03/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 76.745µs
Jan 29 16:20:09 hostname rudder-relayd[29531]:  INFO relayd::relay-api: 127.0.0.1:38614 "PUT /rudder/relay-api/1/shared-files/root/3da496fa-40f1-4e66-9018-7d4b2f2f5d03/rudder_monitoring.csv HTTP/1.1" 500 "-" "curl/7.61.0" 674.054µs
Jan 29 16:20:18 hostname rudder-relayd[29531]:  INFO relayd::relay-api: 127.0.0.1:38578 "HEAD /rudder/relay-api/1/shared-files/root/093896e1-8c96-4d27-81e7-346312da43ac/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 57.967µs
Jan 29 16:20:18 hostname rudder-relayd[29531]:  INFO relayd::relay-api: 127.0.0.1:38614 "PUT /rudder/relay-api/1/shared-files/root/093896e1-8c96-4d27-81e7-346312da43ac/rudder_monitoring.csv HTTP/1.1" 500 "-" "curl/7.61.0" 423.109µs
Jan 29 16:20:33 hostname rudder-relayd[29531]:  INFO relayd::relay-api: 127.0.0.1:38664 "HEAD /rudder/relay-api/1/shared-files/root/3ce13542-195b-482a-9e2e-7274add46528/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 60.554µs
Jan 29 16:20:33 hostname rudder-relayd[29531]:  INFO relayd::relay-api: 127.0.0.1:38666 "PUT /rudder/relay-api/1/shared-files/root/3ce13542-195b-482a-9e2e-7274add46528/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 47.46µs
Jan 29 16:21:33 hostname rudder-relayd[29531]:  INFO relayd::relay-api: 127.0.0.1:38664 "HEAD /rudder/relay-api/1/shared-files/root/093896e1-8c96-4d27-81e7-346312da43ac/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 57.561µs
Jan 29 16:21:33 hostname rudder-relayd[29531]:  INFO relayd::relay-api: 127.0.0.1:38666 "PUT /rudder/relay-api/1/shared-files/root/093896e1-8c96-4d27-81e7-346312da43ac/rudder_monitoring.csv HTTP/1.1" 500 "-" "curl/7.61.0" 414.567µs

Thank you for your help.

Subtasks 1 (0 open — 1 closed)

Actions

Copy link

Updated by Alexis Mousset almost 5 years ago

Description updated (diff)

Actions

Copy link

Updated by Alexis Mousset almost 5 years ago

Severity changed from Minor - inconvenience | misleading | easy workaround to Major - prevents use of part of Rudder | no simple workaround
User visibility set to Operational - other Techniques | Rudder settings | Plugins
Priority changed from 0 to 52

Reproduced the 500 error.

Actions

Copy link

Updated by Alexis Mousset almost 5 years ago

Added logs in #16648, this will help troublehoosting the problem.

Actions

Copy link

Updated by Samuel Chesnel almost 5 years ago

Hello,
Is there a workaround or should I still wait ?
Will it be fixed in the next patch ? When will it be released ?

Thank you for your answer.

Actions

Copy link

Updated by Alexis Mousset almost 5 years ago

No workaround for now, still investigating.

Detailed error:

janv. 31 11:45:05 server rudder-relayd[19078]: ERROR relayd::api: error while processing request: invalid hash type provided header=rudder-signature-v1
janv. 31 11:45:05 server rudder-relayd[19078]: algorithm=sha512
janv. 31 11:45:05 server rudder-relayd[19078]: digest=42cfc842c0c40b6a46c77fb672d52ed1c257181a5f4827fd3ecc2e62ff20a6f8e8e61b14d2e9bd03c7698e97a92141d7b7f01be1eeec548c0730e229a47ec6f9665c6bb3cb65fb6ec48125e724a63adb7218f13cce6811ebd203003af92eb1bdde041586052c7df0025a360397f03c261b37f9f841d08f883fe5ee0f300c535b8d75dac9b896357cbb80ec05e793e2bff971036c74fbcc77d3b532ffff962b275e4b926579790fbda16376cb148a37758b49c8f9b3f9a7f8318dbaad1fbd5f169539e06b6a6a632a0f9bac99cc2e6d0df2d424098a0fb0b1eaad29c320cc97303c857d3b15e1b8fb52eb1b33a46b5ec1ebfa9381dcb12a673289d82f8fc523fb55ce46c3c47774540b5b9d5c1e73b4a0cf5d56630c4e51613403ec0eab0817aee54c4a11043e23ded44604e0fc7e5e8e471140f6e5de12d37b8c5a4bab3cb96699414b092bd1f0b046ba74ab0776b3f20e1bbf5a0b52c9140fd6f87701610df99aa230f3f79a41373e61a8432061fa34b027329be7dd46810f1ea8f14d2ffa51a35167b6bcf5233d36ffba87cde20707d17e7480cd4641870dd3e5c32536ee79495d73444e885a47a18213ed798a3c363f99ce9dd2d36d75e3a2c0e64aa9e31997cb39132588c3ef4f3d3ff03b226d1c7e09814ac63115b83655c5a772c5fd2f0c27d3c1076ffbbdf77ea0ea6724a220166ae11a0cbf013e867dcdee0482480b
janv. 31 11:45:05 server rudder-relayd[19078]: hash_value=adb7647d077f8e3389a6c7eebdff15ffabc1b94d65f0ea94784e720d787779c96e1c50e83e775e62862d85aff1374d0f2b366ce3673a33a7cf366223b7104a71
janv. 31 11:45:05 server rudder-relayd[19078]: short_pubkey=MIICCgKCAgEAvGbc/WWATeyZDSlJaVIGHY957TG4hrSzt0zv4mXs/k/MdJZqVbIMHWlrQmnIOYACXqtuWQZ/6//+fptKaY/CmQKZbIvNWZusF1qZs3Px4aw201vwT8aQoCXun+Cf2WxKEag8GZZiBMoog3+rF1mr0raM64TDLMdnHzfYG7b/X/g5Kv0jkTKLKFGZ9tGjJU387pANgnAvSCD83K8F+WEDrofCbJ/SxxN3AxoNFINLtVMXAR41Xe4DY9lFlQs9W5VEMSILZM31q9ghlvtnVLe3UE16oM4GoHzXNanSKEl2UxxiBYMVFd7zAXrJo4Z3vwOAp23a+c24GQDv3JjxjNDOyVLTfX4UCF2Yjq3M5Ia+ay67UIFLlG2PXbjDxwIGQQ70QQsPeX1fD8m+fLkQNQEvwC+OE6Xmv8qJHDZ/vxd5+Mnnokq2bmgqLg2t4EbLeX4aTPa5I4R+fGdBLTpvs3snTqan53jgAvAYDd3jgXFMnt0O1sYp2FRrAlploH3VMimmE1ics2CeF6O+nShNbpARq7+ml+0b3BbLQqR5T1CSo2GlOKi3uVt5TO1J4rJ0zxLGhnUsGRyroSlET94MV7yC2RQktBuB6EHB6T/rySCg7EruB7mRsfN6Er5Eq3REq6jVA+mImx7957DFKZANSaOWr6yHrEHw21eo6tg5gapwmo0CAwEAAQ==
janv. 31 11:45:05 server rudder-relayd[19078]: hostname=server
janv. 31 11:45:05 server rudder-relayd[19078]: keydate=2020-01-28 15:36:56.704254842 +0000
janv. 31 11:45:05 server rudder-relayd[19078]: keyid=AA709A8D
janv. 31 11:45:05 server rudder-relayd[19078]: (available hash types: sha256, sha512)

This is a problem in metadata parsing.

Actions

Copy link

Updated by Alexis Mousset almost 5 years ago

Fixed #16657, will continue testing (and adding tests fot this specific call).

Actions

Copy link

Updated by Alexis Mousset almost 5 years ago

Now getting:

janv. 31 15:05:31 server rudder-relayd[17590]: WARN shared_files_put{target_id=root source_id=root file_id=blob}: relayd::api::shared_files: hash of public key (f4b977d94d60d2f7eb4f55bafb63ce30c546071fe7f59f623a445dc500d89bddbf9aee3487558436ad537851854fcc67ff34207b1d361a76afe914729b32a11d) does not match metadata (42cfc842c0c40b6a46c77fb672d52ed1c257181a5f4827fd3ecc2e62ff20a6f8e8e61b14d2e9bd03c7698e97a92141d7b7f01be1eeec548c0730e229a47ec6f9665c6bb3cb65fb6ec48125e724a63adb7218f13cce6811ebd203003af92eb1bdde041586052c7df0025a360397f03c261b37f9f841d08f883fe5ee0f300c535b8d75dac9b896357cbb80ec05e793e2bff971036c74fbcc77d3b532ffff962b275e4b926579790fbda16376cb148a37758b49c8f9b3f9a7f8318dbaad1fbd5f169539e06b6a6a632a0f9bac99cc2e6d0df2d424098a0fb0b1eaad29c320cc97303c857d3b15e1b8fb52eb1b33a46b5ec1ebfa9381dcb12a673289d82f8fc523fb55ce46c3c47774540b5b9d5c1e73b4a0cf5d56630c4e51613403ec0eab0817aee54c4a11043e23ded44604e0fc7e5e8e471140f6e5de12d37b8c5a4bab3cb96699414b092bd1f0b046ba74ab0776b3f20e1bbf5a0b52c9140fd6f87701610df99aa230f3f79a41373e61a8432061fa34b027329be7dd46810f1ea8f14d2ffa51a35167b6bcf5233d36ffba87cde20707d17e7480cd4641870dd3e5c32536ee79495d73444e885a47a18213ed798a3c363f99ce9dd2d36d75e3a2c0e64aa9e31997cb39132588c3ef4f3d3ff03b226d1c7e09814ac63115b83655c5a772c5fd2f0c27d3c1076ffbbdf77ea0ea6724a220166ae11a0cbf013e867dcdee0482480b)
janv. 31 15:05:31 server rudder-relayd[17590]: INFO relayd::relay-api: 127.0.0.1:56202 "PUT /rudder/relay-api/1/shared-files/root/root/blob HTTP/1.1" 404 "-" "curl/7.61.0" 994.721µs

Actions

Copy link

Updated by Alexis Mousset almost 5 years ago

Status changed from New to In progress

Actions

Copy link

Updated by Samuel Chesnel almost 5 years ago

Do you have any idea about when it will be patched ?
This is blocking us at the moment.

Actions

Copy link

#10

Updated by Alexis Mousset almost 5 years ago

Status changed from In progress to Pending technical review
Assignee changed from Alexis Mousset to Benoît PECCATTE
Pull Request set to https://github.com/Normation/rudder/pull/2769

Actions

Copy link

#11

Updated by Alexis Mousset almost 5 years ago

Finally this required a lot of changes.

The problem were that there was some confusion between sent file hash and public key hash, and that consequently the key-hash from the nodeslist file was ignored. Support for forwarding to parent policy server was also missing.

General changes

Added a proper representation of the key hash in nodes list (to parse the algorithm and the hash value separately). This is the most impacting change as it is shared with the reporting code, but it's very small.

For shared-folder API

Refactored hashing code to use an Hash struct containing the algorithm and the hash value. This improves the consistency with shared-files.

For shared-files API:

Replaced the shared-files test data with smaller files (it was an image which is was hard to compare).
Added a simplified version of rudder-sign to the repo to allow easily adding other test cases (it also has more parameters to allow using custom keys, and does not support 1.0 format which is not usable with shared-files)
The shared-files implementation is now now split between a file describing the base data structures (Metadata and SharedFile) in src/data/shared_file.rs and the implementation the the api in src/api/shared_files.rs.
Added the support to forward calls to parent policy server (it was previously missing)
Added 3 tests cases for shared-files PUT API. It was the only call missing tests.
Added more validation of the API parameters (check that things supposed to be hexadecimal really are, check the node id and file id only use alphanumeric chars, check that the public key is parseable by openssl)