User story #14275
closedRudder server agent does not submit new inventory
Description
when calling "rudder agent inventory" rudder policy server the inventory always lands in /var/rudder/inventories/failed/
Output from "rudder agent inventory -i -w -R -f":
Rudder agent 5.0.6-stretch0 Node uuid: root rudder info: Using command line specified bundlesequence rudder info: Executing 'no timeout' ... '/opt/rudder/bin/curl -L -k -1 -s -f --proxy '' -o "/var/rudder/cfengine-community/rudder-server-uuid.txt" https://rudder/uuid' rudder info: Completed execution of '/opt/rudder/bin/curl -L -k -1 -s -f --proxy '' -o "/var/rudder/cfengine-community/rudder-server-uuid.txt" https://rudder/uuid' rudder info: Executing 'no timeout' ... '/opt/rudder/bin/run-inventory --local=/var/rudder/tmp/inventory/px-10001-root.ocs' notice: Q: "...-inventory --lo": [info] FusionInventory instance: builtin - Perl instance: system Q: "...-inventory --lo": [info] running task Inventory Q: "...-inventory --lo": [info] Inventory saved in /var/rudder/tmp/inventory/px-10001-root.ocs rudder info: Last 3 quoted lines were generated by promiser '/opt/rudder/bin/run-inventory --local=/var/rudder/tmp/inventory/px-10001-root.ocs' rudder info: Completed execution of '/opt/rudder/bin/run-inventory --local=/var/rudder/tmp/inventory/px-10001-root.ocs' rudder info: Copying from 'localhost:/var/rudder/tmp/inventory/px-10001-root.ocs' rudder info: Transforming '/opt/rudder/bin/rudder-sign "/var/rudder/inventories/px-10001-root.ocs"' rudder info: Transformer '/var/rudder/inventories/px-10001-root.ocs' => '/opt/rudder/bin/rudder-sign "/var/rudder/inventories/px-10001-root.ocs"' seemed to work ok rudder info: Transforming '/opt/rudder/bin/rudder-sign "/var/rudder/inventories/px-10001-root.ocs"' rudder info: Transforming '/bin/gzip -fq /var/rudder/inventories/px-10001-root.ocs' rudder info: Transformer '/var/rudder/inventories/px-10001-root.ocs' => '/bin/gzip -fq /var/rudder/inventories/px-10001-root.ocs' seemed to work ok rudder info: Transforming '/opt/rudder/bin/curl -L -k -1 -f -s --proxy '' --user rudder:rudder -T /var/rudder/inventories/px-10001-root.ocs.gz https://rudder/inventory-updates/' rudder info: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> rudder info: <html><head> rudder info: <title>201 Created</title> rudder info: </head><body> rudder info: <h1>Created</h1> rudder info: <p>Resource /inventory-updates/px-10001-root.ocs.gz has been created.</p> rudder info: <hr /> rudder info: <address>Apache/2.4.25 (Debian) Server at rudder Port 443</address> rudder info: </body></html> rudder info: Automatically promoting context scope for 'inventory_sent' to namespace visibility, due to persistence rudder info: Transformer '/var/rudder/inventories/px-10001-root.ocs.gz' => '/opt/rudder/bin/curl -L -k -1 -f -s --proxy '' --user rudder:rudder -T /var/rudder/inventories/px-10001-root.ocs.gz https://rudder/inventory-updates/' seemed to work ok rudder info: Transforming '/opt/rudder/bin/curl -L -k -1 -f -s --proxy '' --user rudder:rudder -T /var/rudder/inventories/px-10001-root.ocs.gz https://rudder/inventory-updates/' rudder info: Transforming '/opt/rudder/bin/curl -L -k -1 -f -s --proxy '' --user rudder:rudder -T /var/rudder/inventories/px-10001-root.ocs.sign https://rudder/inventory-updates/' rudder info: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> rudder info: <html><head> rudder info: <title>201 Created</title> rudder info: </head><body> rudder info: <h1>Created</h1> rudder info: <p>Resource /inventory-updates/px-10001-root.ocs.sign has been created.</p> rudder info: <hr /> rudder info: <address>Apache/2.4.25 (Debian) Server at rudder Port 443</address> rudder info: </body></html> rudder info: Automatically promoting context scope for 'inventory_sent' to namespace visibility, due to persistence rudder info: Transformer '/var/rudder/inventories/px-10001-root.ocs.sign' => '/opt/rudder/bin/curl -L -k -1 -f -s --proxy '' --user rudder:rudder -T /var/rudder/inventories/px-10001-root.ocs.sign https://rudder/inventory-updates/' seemed to work ok rudder info: Transforming '/opt/rudder/bin/curl -L -k -1 -f -s --proxy '' --user rudder:rudder -T /var/rudder/inventories/px-10001-root.ocs.sign https://rudder/inventory-updates/' rudder info: Transforming '/bin/rm -f /var/rudder/inventories/px-10001-root.ocs.gz' rudder info: Transformer '/var/rudder/inventories/px-10001-root.ocs.gz' => '/bin/rm -f /var/rudder/inventories/px-10001-root.ocs.gz' seemed to work ok rudder info: Transforming '/bin/rm -f /var/rudder/inventories/px-10001-root.ocs.sign' rudder info: Transformer '/var/rudder/inventories/px-10001-root.ocs.sign' => '/bin/rm -f /var/rudder/inventories/px-10001-root.ocs.sign' seemed to work ok rudder info: Touched (updated time stamps) for path '/var/rudder/tmp/inventory_sent'
Apache2 - access.log:
10.11.0.130 - - [05/Feb/2019:10:26:17 +0100] "GET /uuid HTTP/1.1" 200 1776 "-" "curl/7.61.0" 10.11.0.130 - - [05/Feb/2019:10:26:18 +0100] "PUT /inventory-updates/px-10001-root.ocs.gz HTTP/1.1" 201 2135 "-" "curl/7.61.0" 10.11.0.130 - - [05/Feb/2019:10:26:18 +0100] "PUT /inventory-updates/px-10001-root.ocs.sign HTTP/1.1" 201 2139 "-" "curl/7.61.0"
Files exists but don't get updated (since 3 weeks):
/var/rudder/inventories/received/px-10001-root.ocs [root:root] [0660] /var/rudder/inventories/received/px-10001-root.ocs.sign [www-data:www-data] [0644]
Only these files are getting updated:
/var/rudder/inventories/failed/px-10001-root.ocs [root:root] [0660] /var/rudder/inventories/failed/px-10001-root.ocs.sign [www-data:www-data] [0644]
Others clients managed with this rudder server have no problems at all - their inventory-submissions work perfectly.
Any ideas?
Updated by François ARMAND almost 6 years ago
Hello,
Thanks for the details information. Given that, we can tell that the problem is most lidely in the inventory content interpretation by rudder (parsing or something).
The problem may be explained in `/var/log/rudder/webapp/2019_02_05.stderrout.log`.
You can change verbosity of inventory processing by adding the line:
<logger name="inventory-processing" level="trace" />
in /opt/rudder/etc/logback.xml whereever in the file after the `<root level="info">...</root>` declaration, for example towards the end, before `</configuration>`.
If you don't see anything strange, can you send us the inventory file+signature for analysis? Parhaps it will need some anonymisation, though.
Updated by Stefan Schmitt almost 6 years ago
- Tracker changed from Bug to User story
- Priority deleted (
0)
Hello François!
thx for the good pointing to -> /var/log/rudder/webapp/2019_02_05.stderrout.log
And yes - there have been a good explenation:
[2019-02-05 10:22:48] ERROR inventory-processing - Rejecting Inventory 'px-10001-root.ocs' for Node 'root' because the Inventory signature is not valid: the Inventory was not signed with the same agent key as the one saved within Rudder for that Node. If you updated the agent key on this node, you can update the key stored within Rudder with the following command on the Rudder Server: '/opt/rudder/bin/rudder-keys change-key root <your new public key>'. If you did not change the key, please ensure that the node sending that inventory is actually the node registered within Rudder
We had some keys regenerated... so there was the problem.
The rudder policy server didn't knew the new public key -> changed it with the following command:
/opt/rudder/bin/rudder-keys change-key root /var/rudder/cfengine-community/ppkeys/localhost.pub
=> now the inventory is beeing accepted! -> thx!
=> ticket can be closed.