Rudder

So, it's even stranger:

- the node inventory is correct. It has a certificate with what is expected
- the agent is not init, as if NodeFact.defaultRudderAgent was used: in ldap, we have:

{"agentType":"cfengine-community","version":"unknown","securityToken":{"value":"-----BEGIN RSA PUBLIC KEY-----\nnot initialized\n-----END RSA PUBLIC KEY-----","type":"publicKey"},"capabilities":[]}

But all other values are correct: software, env vars, etc.

And I'm never able to reproduce it localy, even if I try to accept the same inventory, also by API.

It doesn't look like it's a "only one core + race condition" thing either

I got an other instance on 8.1.2:

2024-05-16 21:22:10+0000 INFO  inventory-processing - Received new inventory file 'e13386a3-643d-48f5-9c2a-eab1b8662052_2024-05-16T21:22:03+00:00.ocs' with signature available: process.
2024-05-16 21:22:10+0000 WARN  inventory-processing - Error when parsing date for last user loggin. Awaited format is EEE MMM dd HH:mm, found: Thu May 16 21:19:47
2024-05-16 21:22:10+0000 WARN  nodes.security - only certificate are supported for agent security token since Rudder 7.0
2024-05-16 21:22:10+0000 INFO  nodes - New pending node: 'node1.rudder.local' [e13386a3-643d-48f5-9c2a-eab1b8662052]'
2024-05-16 21:22:10+0000 INFO  inventory-processing - Inventory 'e13386a3-643d-48f5-9c2a-eab1b8662052_2024-05-16T21:22:03+00:00.ocs' for node 'node1.rudder.local' [e13386a3-643d-48f5-9c2a-eab1b8662052] (signature:certified) processed in 201 milliseconds
2024-05-16 21:22:26+0000 INFO  inventory-processing - Received new inventory file 'e13386a3-643d-48f5-9c2a-eab1b8662052_2024-05-16T21:22:14+00:00.ocs' with signature available: process.
2024-05-16 21:22:26+0000 WARN  inventory-processing - Error when parsing date for last user loggin. Awaited format is EEE MMM dd HH:mm, found: Thu May 16 21:19:47
2024-05-16 21:22:26+0000 ERROR inventory-processing - Error when trying to process inventory 'e13386a3-643d-48f5-9c2a-eab1b8662052_2024-05-16T21:22:14+00:00.ocs'; cause was: CryptoEx: Key '-----BEGIN RSA PUBLIC KEY-----
not initialized
-----END RSA PUBLIC KEY-----' cannot be parsed as a public key; root exception was: unable to decode base64 string: String index out of range: 15
2024-05-16 21:22:26+0000 ERROR inventory-processing - Inventory 'e13386a3-643d-48f5-9c2a-eab1b8662052_2024-05-16T21:22:14+00:00.ocs' for Node 'unknown' failed to be saved in Rudder. Cause was: Error when trying to process inventory 'e13386a3-643d-48f5-9c2a-eab1b8662052_2024-05-16T21:22:14+00:00.ocs'; cause was: CryptoEx: Key '-----BEGIN RSA PUBLIC KEY-----
not initialized
-----END RSA PUBLIC KEY-----' cannot be parsed as a public key; root exception was: unable to decode base64 string: String index out of range: 15
2024-05-16 21:22:35+0000 INFO  nodes - New accepted node: 'node1.rudder.local' [e13386a3-643d-48f5-9c2a-eab1b8662052]'

What is strange is that there is two lines Received new inventory file 'e13386a3-643d-48f5-9c2a-eab1b8662052_2024-05-16T21:22:03+00:00.ocs' with signature available: process.

The first doesn't have the signature error, while the second has. So perhaps the first inventory is incorrect in some way, perhaps it has both a certificate and signature but they don't match ? I don't have access to that inventory, it is erased by the second one just after.

The lines:

2024-05-16 21:22:10+0000 WARN  nodes.security - only certificate are supported for agent security token since Rudder 7.0
2024-05-16 21:22:10+0000 INFO  nodes - New pending node: 'node1.rudder.local' [e13386a3-643d-48f5-9c2a-eab1b8662052]'
2024-05-16 21:22:10+0000 INFO  inventory-processing - Inventory 'e13386a3-643d-48f5-9c2a-eab1b8662052_2024-05-16T21:22:03+00:00.ocs' for node 'node1.rudder.local' [e13386a3-643d-48f5-9c2a-eab1b8662052] (signature:certified) processed in 201 milliseconds

So, it looks like things happen like that:

- we get a first inventory. That inventory does not have a certificate or at least, for some reason, it contains a "publickey" entry that is not valid, or something else BUT something that leads to the message "public key are not supported anymore", but still with a valid signature somehow (HOW?)
- that certificate get to pending, with its (invalid) key getting the certified status.
- a second, correct inventory comes, with everything good.
- rudder looks and find the other inventory in pending, and get it from there. This is a regression in 8.1 where we don't look only on accepted nodes in that case
- so we use the incorrect key with its certified status and everything else are correct values from the second inventory
- the node is accepted with the incorrect key.

I don't get how the first step happens. If the signature is there, there must be a valid cert/public key to use. If it's not there, the inventory is ignored.
I think that maybe both certificate and public key attribute are present, and the certificate is ok but not the public key, and we check with the first but store the second one.
I was not able to see that in the code, but in any case, the problem happens because we still parse the public key part, what we stopped supporting in 7.0, so we really need to get ride of it now. Ignoring the public key at root will make the whole scenario much simpler to understand.