Bug #3526
closed
Add a promise to remove Git lock file on configuration-repository if older than 5 minutes
Description
When trying to add a node on Rudder 2.5.2 on CentOS , we get the error:
Error when trying to accept node f58dab50-6f5b-4463-b44e-5a5be0fce562 <- Error when executing accept node process named accept_new_server:add_system_configuration_rules <- Could not update the technical group with ID 'NodeGroupId(hasPolicyServer-root)' <- Exception caught during execution of add command
This was from a fresh install from Rudder-vagrant (branch 2.5, vagrant up centos6).
It does not works any better when upgrading to Rudder 2.6 nigtlies.
It does work for a fresh install on Debian (from Rudder-vagrant, branch 2.5, vagrant up server).
It does work on dev machine, or in other test environment (orch-3).
In fact, this is due to a bad initialisation of the .git in /var/rudder/configuration-repository that let the '/var/rudder/configuration-repository/.git/index.lock' file.
Updated by François ARMAND over 11 years ago
In fact, the problem is due to Git!
In debug mode, we can see:
[2013-04-24 18:39:45] DEBUG com.normation.rudder.repository.xml.GitNodeGroupArchiverImpl - Archived node group: /var/rudder/configuration-repository/groups/SystemGroups/hasPolicyServer-root.xml [2013-04-24 18:39:45] DEBUG com.normation.rudder.repository.xml.GitNodeGroupArchiverImpl - Add file groups/SystemGroups/hasPolicyServer-root.xml from configuration repository [2013-04-24 18:39:45] ERROR com.normation.rudder.services.servers.NewNodeManagerImpl - Error when trying to accept node f58dab50-6f5b-4463-b44e-5a5be0fce562. Rollbaking. <- Error when executing accept node process named accept_new_server:add_system_configuration_rules <- Could not update the technical group with ID 'NodeGroupId(hasPolicyServer-root)' <- Exception caught during execution of add command
So it's clearly when archiving in Git file groups/SystemGroups/hasPolicyServer-root.xml that the problem leading to Rollback occures.
After deleting the directory "/var/rudder/configuration-repositoy/.git" in the server, and then adding it back (git init && git add rules etc... && git commit -m "plop"), everything works again.
Updated by François ARMAND over 11 years ago
- Subject changed from Can't add a node on CentOS to A bad initialisation configuration-repository/.git leads to a lot of problems (can't add nodes, can't archive, ...)
- Description updated (diff)
Updated by Nicolas PERRON over 11 years ago
After a new installation of rudder 2.5, the git repository will not works:
server:/var/rudder/configuration-repository # git status fatal: unable to create '.git/index.lock': File exists
Simply deleting the file should suffice:
server:/var/rudder/configuration-repository # rm .git/index.lock server:/var/rudder/configuration-repository # git status # On branch master nothing to commit (working directory clean)
Updated by François ARMAND over 11 years ago
Confirmed, I also had the lock file:
fatal: Unable to create '/var/rudder/configuration-repository/.git/index.lock': File exists.
(message displayed when trying to git add a file)
Updated by François ARMAND over 11 years ago
This bug was not reproducible on other installations, so let's say I wasn't lucky and something bad happens.
To prevent that to complettly block a Rudder server, we are going to add a CFEngine policy that check the presence of the lock file, and delete it if it is older than 5 minutes (a Git lock should never be that old, most Git operation takes only seconds even on heavy I/O loaded machine).
Updated by Matthieu CERDA over 11 years ago
- Status changed from New to In progress
- Assignee set to Matthieu CERDA
Updated by François ARMAND over 11 years ago
- Subject changed from A bad initialisation configuration-repository/.git leads to a lot of problems (can't add nodes, can't archive, ...) to Add a promise to remove Git lock file on configuration-repository if older than 5 minutes
- Description updated (diff)
- Assignee deleted (
Matthieu CERDA)
Updated by Matthieu CERDA over 11 years ago
- Status changed from In progress to Pending technical review
- % Done changed from 0 to 100
- Pull Request set to https://github.com/Normation/rudder-techniques/pull/102
PR available.
Updated by Matthieu CERDA over 11 years ago
- Status changed from Pending technical review to Pending release
Applied in changeset policy-templates:commit:081574496aa9812d1197cf95e386b507d8622bc9.
Updated by Matthieu CERDA over 11 years ago
Applied in changeset policy-templates:commit:bb9d9201c671077ba3a7f831d08cfb774661d8c4.
Updated by Matthieu CERDA over 11 years ago
Applied in changeset policy-templates:commit:d6aa21ea7565811afc6303c1d21a08833c9c401e.
Updated by Jonathan CLARKE over 11 years ago
Applied in changeset policy-templates:commit:1458db9bb803be1691d0d092b94581e4fc8e97ca.
Updated by Matthieu CERDA over 11 years ago
Applied in changeset policy-templates:commit:c1083ced4c6e30912b67bb1399946469ca795dc1.
Updated by Jonathan CLARKE over 11 years ago
Applied in changeset policy-templates:commit:73187a7af6a3e84c71c293287ab74871bc8f6533.
Updated by Matthieu CERDA over 11 years ago
- Pull Request changed from https://github.com/Normation/rudder-techniques/pull/102 to https://github.com/Normation/rudder-techniques/pull/103
PR updated
Updated by Matthieu CERDA over 11 years ago
Updated by Matthieu CERDA over 11 years ago
Applied in changeset policy-templates:commit:b562fb5897c02a0978a2d6f3bcad59741b2de93b.
Updated by Jonathan CLARKE over 11 years ago
Applied in changeset policy-templates:commit:d64139d955a9dfcafe82bba746221f7745a734c5.
Updated by Nicolas PERRON over 11 years ago
- Status changed from Pending release to Released