Rudder

Sorry, Update of rudder-server was from 8.2.3 to 8.2.4, OS is Ubuntu 22.04

Hello, thanks for reporting.

As a first step, we will check that the entry is here, and then if it's the case (which is likely, given what you tested with snapshot restauration), we will try to understand what is going on. It's a very basic and ancient integrity check, so understanding why it triggered will be intersting.

So, step 1: check that the LDAP content is correct.
Can you please exec on the root server:

ldapsearch -o ldif-wrap=no -H "ldap://localhost:389" -x -D "cn=Manager,cn=rudder-configuration" -W -b "groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration" -s one '1.1'

In the command output, you should see several entries, among them the one that is seen as missing (dn: nodeGroupId=hasPolicyServer-root,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration)

Hi Francois,

thank you very much for you assisstance.

I'have checked it, unfortunately there is no entry:

root@system: ldapsearch -o ldif-wrap=no -H "ldap://localhost:389" -x -D "cn=Manager,cn=rudder-configuration" -W -b "groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration" -s one '1.1'
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration> with scope oneLevel
# filter: (objectclass=*)
# requesting: 1.1
#

# search result
search: 2
result: 0 Success

# numResponses: 1

Do you know how this can happen?

To make things clear: After everything failed, I turned back to a 7 days old snapshot (the snapshot is of the whole VM, not only the LXD-Container), and there the problem is still there. After having fiddeled around a lot, I again returned to that snapshot, so the snapshot is the actual state.

There is also a somewhat old snapshot of only the container directly in LXD (from 08/01/2025). I just now returned to this snapshot and excuted your mentioned command:

ldapsearch -o ldif-wrap=no -H "ldap://localhost:389" -x -D "cn=Manager,cn=rudder-configuration" -W -b "groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration" -s one '1.1'
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration> with scope oneLevel
# filter: (objectclass=*)
# requesting: 1.1
#

# special:all, SystemGroups, GroupRoot, Rudder, rudder-configuration
dn: ruleTarget=special:all,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration

# policyServer:root, SystemGroups, GroupRoot, Rudder, rudder-configuration
dn: ruleTarget=policyServer:root,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration

# hasPolicyServer-root, SystemGroups, GroupRoot, Rudder, rudder-configuration
dn: nodeGroupId=hasPolicyServer-root,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration

# special:all_policyServers, SystemGroups, GroupRoot, Rudder, rudder-configuration
dn: ruleTarget=special:all_policyServers,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration

# all-nodes-with-cfengine-agent, SystemGroups, GroupRoot, Rudder, rudder-configuration
dn: nodeGroupId=all-nodes-with-cfengine-agent,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration

# special:all_exceptPolicyServers, SystemGroups, GroupRoot, Rudder, rudder-configuration
dn: ruleTarget=special:all_exceptPolicyServers,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration

There is even more output and Jetty is working fine.

I think, those entries got lost in LDAP. Do you know this could happen?

Thank you in advance!

Bruno

OK, so nice that you find them back. The only reason I see that could lead to that is some error during the upgrade process that lead to note keeping the LDAP configuration and loosing the data archive. I'm not sure how it can happen, but perhaps if you answer "yes" to dpkg question regarding getting the package new conf, you can reach that state.
Still, you should at least have the backups done before upgrade in /var/rudder/ldap/backup/ - you could look for the most recent and see if it has the entry, and if not, iteratively look in older one to see if the date help to correlate with some other odities

Thank you! No, I dont's think that the issue was caused by the dpkq-question. As mentioned, older snapshots had the same issues and apt dist-upgrade did not run there.

Ok, seems like this cannot be repaired ad hoc. Getting back to LXD snapshot works as mentioned. But: Since 01/08/2025 there weren't modified directives, techniques etc. pp. but many computers (that have been reinstalled mostly) were joined to Rudder and they don't come in to the pending nodes.

That's why, please allow me this one question: How to cope with this? Is it a way, to just delete all computers in Rudders and let them join automatically by themselves?
Or has a factory reset of the agent to be made on all nodes?

Thank you!

Hi everybody,

I think we achieved to reconstruct the problem:

When adding system groups into another category, the ldap db crashes and jetty doesn't start anymore. Maybe because the system group can't be found anymore. We think, there shouldn't be a possibility
to move system groups (as it seems that this isn't intended at all).

Maybe this is a bug. Can you please fix the issue, so that other people don't get caught into it?

Thank you very much!

Hi !

I'm confused as to how you changed the group category, as the UI doesn't allow it for system groups
Did you change it using the API?

Thank you, Nicolas, good move for helping other users not to get in this issue :)

This has some link with #25348 : the system groups were able to be deleted, but they shouldn't, neither in the UI nor in the API.
Here there can be updated, in the UI and the API, but they shouldn't.

So the fix is to disallow it as in #25348, and for existing broken system groups, they should be moved to the "Root of the group and group categories" (parent ID : "GroupRoot").

• disallowing update of system categories in the API for POST /groups/categories/{groupCategoryId}
• removing the "Update" button and disable the selector for the group category