Bug #16641
closed
Bug #16549: Shared files are broken in relayd
Error 404 when accessing shared-files (relayd)
Added by Samuel Chesnel over 4 years ago.
Updated about 4 years ago.
Category:
Relay server or API
Severity:
Major - prevents use of part of Rudder | no simple workaround
User visibility:
Operational - other Techniques | Rudder settings | Plugins
Description
Hello,
I'm running Rudder 6.0.2 on a CentOS8 server and I'm using the centreon plugin.
First, I encountered the following error when trying to apply a Centreon template :
E| compliant DeploiementGlobalR Monitoring template Global_R Host follows Global_R monitoring template was correct
error: Command related to promiser 'echo | cat /var/rudder/tmp/rudder_monitoring.csv.sign - /var/rudder/tmp/rudder_monitoring.csv | /opt/rudder/bin/curl' returned code defined as promise failed 22
error: Method 'sharedfile_to_node' failed in some repairs
E| repaired Common Monitoring Share Rudder monitoring information with the server was repaired
error: Method 'monitoring_hook_post' failed in some repairs
Then thanks to your help I got to patch this error by doing the following actions :
I applied the patch showed on https://github.com/Normation/rudder-techniques/pull/1578/files/7a5675ac060c97fd9f548a8f4aea74363104443b on the file /var/rudder/configuration-repository/techniques/system/server-roles/1.0/relayd.conf.tpl, then run cd /var/rudder/configuration-repository, git add techniques/system/server-roles/1.0/relayd.conf.tpl && git commit -m "Fix 16549", and rudder server reload-techniques.
The issue is still present and i got the following error on my relayd service:
Jan 29 16:15:52 hostname rudder-relayd[29531]: INFO relayd::relay-api: 127.0.0.1:38578 "HEAD /rudder/relay-api/1/shared-files/root/3ce13542-195b-482a-9e2e-7274add46528/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 64.338µs
Jan 29 16:15:52 hostname rudder-relayd[29531]: INFO relayd::relay-api: 127.0.0.1:38578 "PUT /rudder/relay-api/1/shared-files/root/3ce13542-195b-482a-9e2e-7274add46528/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 41.57µs
Jan 29 16:17:48 hostname rudder-relayd[29531]: INFO relayd::relay-api: 127.0.0.1:38614 "HEAD /rudder/relay-api/1/shared-files/root/d19c9d81-156d-4bac-8704-8cb519779918/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 72.778µs
Jan 29 16:17:48 hostname rudder-relayd[29531]: INFO relayd::relay-api: 127.0.0.1:38614 "PUT /rudder/relay-api/1/shared-files/root/d19c9d81-156d-4bac-8704-8cb519779918/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 59.228µs
Jan 29 16:20:09 hostname rudder-relayd[29531]: INFO relayd::relay-api: 127.0.0.1:38614 "HEAD /rudder/relay-api/1/shared-files/root/3da496fa-40f1-4e66-9018-7d4b2f2f5d03/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 76.745µs
Jan 29 16:20:09 hostname rudder-relayd[29531]: INFO relayd::relay-api: 127.0.0.1:38614 "PUT /rudder/relay-api/1/shared-files/root/3da496fa-40f1-4e66-9018-7d4b2f2f5d03/rudder_monitoring.csv HTTP/1.1" 500 "-" "curl/7.61.0" 674.054µs
Jan 29 16:20:18 hostname rudder-relayd[29531]: INFO relayd::relay-api: 127.0.0.1:38578 "HEAD /rudder/relay-api/1/shared-files/root/093896e1-8c96-4d27-81e7-346312da43ac/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 57.967µs
Jan 29 16:20:18 hostname rudder-relayd[29531]: INFO relayd::relay-api: 127.0.0.1:38614 "PUT /rudder/relay-api/1/shared-files/root/093896e1-8c96-4d27-81e7-346312da43ac/rudder_monitoring.csv HTTP/1.1" 500 "-" "curl/7.61.0" 423.109µs
Jan 29 16:20:33 hostname rudder-relayd[29531]: INFO relayd::relay-api: 127.0.0.1:38664 "HEAD /rudder/relay-api/1/shared-files/root/3ce13542-195b-482a-9e2e-7274add46528/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 60.554µs
Jan 29 16:20:33 hostname rudder-relayd[29531]: INFO relayd::relay-api: 127.0.0.1:38666 "PUT /rudder/relay-api/1/shared-files/root/3ce13542-195b-482a-9e2e-7274add46528/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 47.46µs
Jan 29 16:21:33 hostname rudder-relayd[29531]: INFO relayd::relay-api: 127.0.0.1:38664 "HEAD /rudder/relay-api/1/shared-files/root/093896e1-8c96-4d27-81e7-346312da43ac/rudder_monitoring.csv HTTP/1.1" 404 "-" "curl/7.61.0" 57.561µs
Jan 29 16:21:33 hostname rudder-relayd[29531]: INFO relayd::relay-api: 127.0.0.1:38666 "PUT /rudder/relay-api/1/shared-files/root/093896e1-8c96-4d27-81e7-346312da43ac/rudder_monitoring.csv HTTP/1.1" 500 "-" "curl/7.61.0" 414.567µs
Thank you for your help.
- Description updated (diff)
- Severity changed from Minor - inconvenience | misleading | easy workaround to Major - prevents use of part of Rudder | no simple workaround
- User visibility set to Operational - other Techniques | Rudder settings | Plugins
- Priority changed from 0 to 52
Reproduced the 500 error.
Added logs in #16648, this will help troublehoosting the problem.
Hello,
Is there a workaround or should I still wait ?
Will it be fixed in the next patch ? When will it be released ?
Thank you for your answer.
No workaround for now, still investigating.
Detailed error:
janv. 31 11:45:05 server rudder-relayd[19078]: ERROR relayd::api: error while processing request: invalid hash type provided header=rudder-signature-v1
janv. 31 11:45:05 server rudder-relayd[19078]: algorithm=sha512
janv. 31 11:45:05 server rudder-relayd[19078]: digest=42cfc842c0c40b6a46c77fb672d52ed1c257181a5f4827fd3ecc2e62ff20a6f8e8e61b14d2e9bd03c7698e97a92141d7b7f01be1eeec548c0730e229a47ec6f9665c6bb3cb65fb6ec48125e724a63adb7218f13cce6811ebd203003af92eb1bdde041586052c7df0025a360397f03c261b37f9f841d08f883fe5ee0f300c535b8d75dac9b896357cbb80ec05e793e2bff971036c74fbcc77d3b532ffff962b275e4b926579790fbda16376cb148a37758b49c8f9b3f9a7f8318dbaad1fbd5f169539e06b6a6a632a0f9bac99cc2e6d0df2d424098a0fb0b1eaad29c320cc97303c857d3b15e1b8fb52eb1b33a46b5ec1ebfa9381dcb12a673289d82f8fc523fb55ce46c3c47774540b5b9d5c1e73b4a0cf5d56630c4e51613403ec0eab0817aee54c4a11043e23ded44604e0fc7e5e8e471140f6e5de12d37b8c5a4bab3cb96699414b092bd1f0b046ba74ab0776b3f20e1bbf5a0b52c9140fd6f87701610df99aa230f3f79a41373e61a8432061fa34b027329be7dd46810f1ea8f14d2ffa51a35167b6bcf5233d36ffba87cde20707d17e7480cd4641870dd3e5c32536ee79495d73444e885a47a18213ed798a3c363f99ce9dd2d36d75e3a2c0e64aa9e31997cb39132588c3ef4f3d3ff03b226d1c7e09814ac63115b83655c5a772c5fd2f0c27d3c1076ffbbdf77ea0ea6724a220166ae11a0cbf013e867dcdee0482480b
janv. 31 11:45:05 server rudder-relayd[19078]: hash_value=adb7647d077f8e3389a6c7eebdff15ffabc1b94d65f0ea94784e720d787779c96e1c50e83e775e62862d85aff1374d0f2b366ce3673a33a7cf366223b7104a71
janv. 31 11:45:05 server rudder-relayd[19078]: short_pubkey=MIICCgKCAgEAvGbc/WWATeyZDSlJaVIGHY957TG4hrSzt0zv4mXs/k/MdJZqVbIMHWlrQmnIOYACXqtuWQZ/6//+fptKaY/CmQKZbIvNWZusF1qZs3Px4aw201vwT8aQoCXun+Cf2WxKEag8GZZiBMoog3+rF1mr0raM64TDLMdnHzfYG7b/X/g5Kv0jkTKLKFGZ9tGjJU387pANgnAvSCD83K8F+WEDrofCbJ/SxxN3AxoNFINLtVMXAR41Xe4DY9lFlQs9W5VEMSILZM31q9ghlvtnVLe3UE16oM4GoHzXNanSKEl2UxxiBYMVFd7zAXrJo4Z3vwOAp23a+c24GQDv3JjxjNDOyVLTfX4UCF2Yjq3M5Ia+ay67UIFLlG2PXbjDxwIGQQ70QQsPeX1fD8m+fLkQNQEvwC+OE6Xmv8qJHDZ/vxd5+Mnnokq2bmgqLg2t4EbLeX4aTPa5I4R+fGdBLTpvs3snTqan53jgAvAYDd3jgXFMnt0O1sYp2FRrAlploH3VMimmE1ics2CeF6O+nShNbpARq7+ml+0b3BbLQqR5T1CSo2GlOKi3uVt5TO1J4rJ0zxLGhnUsGRyroSlET94MV7yC2RQktBuB6EHB6T/rySCg7EruB7mRsfN6Er5Eq3REq6jVA+mImx7957DFKZANSaOWr6yHrEHw21eo6tg5gapwmo0CAwEAAQ==
janv. 31 11:45:05 server rudder-relayd[19078]: hostname=server
janv. 31 11:45:05 server rudder-relayd[19078]: keydate=2020-01-28 15:36:56.704254842 +0000
janv. 31 11:45:05 server rudder-relayd[19078]: keyid=AA709A8D
janv. 31 11:45:05 server rudder-relayd[19078]: (available hash types: sha256, sha512)
This is a problem in metadata parsing.
Fixed #16657, will continue testing (and adding tests fot this specific call).
Now getting:
janv. 31 15:05:31 server rudder-relayd[17590]: WARN shared_files_put{target_id=root source_id=root file_id=blob}: relayd::api::shared_files: hash of public key (f4b977d94d60d2f7eb4f55bafb63ce30c546071fe7f59f623a445dc500d89bddbf9aee3487558436ad537851854fcc67ff34207b1d361a76afe914729b32a11d) does not match metadata (42cfc842c0c40b6a46c77fb672d52ed1c257181a5f4827fd3ecc2e62ff20a6f8e8e61b14d2e9bd03c7698e97a92141d7b7f01be1eeec548c0730e229a47ec6f9665c6bb3cb65fb6ec48125e724a63adb7218f13cce6811ebd203003af92eb1bdde041586052c7df0025a360397f03c261b37f9f841d08f883fe5ee0f300c535b8d75dac9b896357cbb80ec05e793e2bff971036c74fbcc77d3b532ffff962b275e4b926579790fbda16376cb148a37758b49c8f9b3f9a7f8318dbaad1fbd5f169539e06b6a6a632a0f9bac99cc2e6d0df2d424098a0fb0b1eaad29c320cc97303c857d3b15e1b8fb52eb1b33a46b5ec1ebfa9381dcb12a673289d82f8fc523fb55ce46c3c47774540b5b9d5c1e73b4a0cf5d56630c4e51613403ec0eab0817aee54c4a11043e23ded44604e0fc7e5e8e471140f6e5de12d37b8c5a4bab3cb96699414b092bd1f0b046ba74ab0776b3f20e1bbf5a0b52c9140fd6f87701610df99aa230f3f79a41373e61a8432061fa34b027329be7dd46810f1ea8f14d2ffa51a35167b6bcf5233d36ffba87cde20707d17e7480cd4641870dd3e5c32536ee79495d73444e885a47a18213ed798a3c363f99ce9dd2d36d75e3a2c0e64aa9e31997cb39132588c3ef4f3d3ff03b226d1c7e09814ac63115b83655c5a772c5fd2f0c27d3c1076ffbbdf77ea0ea6724a220166ae11a0cbf013e867dcdee0482480b)
janv. 31 15:05:31 server rudder-relayd[17590]: INFO relayd::relay-api: 127.0.0.1:56202 "PUT /rudder/relay-api/1/shared-files/root/root/blob HTTP/1.1" 404 "-" "curl/7.61.0" 994.721µs
- Status changed from New to In progress
Do you have any idea about when it will be patched ?
This is blocking us at the moment.
- Status changed from In progress to Pending technical review
- Assignee changed from Alexis Mousset to Benoît PECCATTE
- Pull Request set to https://github.com/Normation/rudder/pull/2769
Finally this required a lot of changes.
The problem were that there was some confusion between sent file hash and public key hash, and that consequently the key-hash from the nodeslist file was ignored. Support for forwarding to parent policy server was also missing.
General changes
- Added a proper representation of the key hash in nodes list (to parse the algorithm and the hash value separately). This is the most impacting change as it is shared with the reporting code, but it's very small.
For shared-folder API
- Refactored hashing code to use an
Hash struct containing the algorithm and the hash value. This improves the consistency with shared-files.
For shared-files API:
- Replaced the
shared-files test data with smaller files (it was an image which is was hard to compare).
- Added a simplified version of rudder-sign to the repo to allow easily adding other test cases (it also has more parameters to allow using custom keys, and does not support 1.0 format which is not usable with shared-files)
- The shared-files implementation is now now split between a file describing the base data structures (
Metadata and SharedFile) in src/data/shared_file.rs and the implementation the the api in src/api/shared_files.rs.
- Added the support to forward calls to parent policy server (it was previously missing)
- Added 3 tests cases for shared-files PUT API. It was the only call missing tests.
- Added more validation of the API parameters (check that things supposed to be hexadecimal really are, check the node id and file id only use alphanumeric chars, check that the public key is parseable by openssl)
- Status changed from Pending technical review to Pending release
- Parent task set to #16549
- Fix check changed from To do to Checked
- Name check changed from To do to Reviewed
- Status changed from Pending release to Released
This bug has been fixed in Rudder 6.0.3 which was released today.
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by