Bug #3526
closed
Add a promise to remove Git lock file on configuration-repository if older than 5 minutes
Added by François ARMAND over 11 years ago.
Updated over 11 years ago.
Category:
Web - Nodes & inventories
Description
When trying to add a node on Rudder 2.5.2 on CentOS , we get the error:
Error when trying to accept node f58dab50-6f5b-4463-b44e-5a5be0fce562
<- Error when executing accept node process named accept_new_server:add_system_configuration_rules
<- Could not update the technical group with ID 'NodeGroupId(hasPolicyServer-root)'
<- Exception caught during execution of add command
This was from a fresh install from Rudder-vagrant (branch 2.5, vagrant up centos6).
It does not works any better when upgrading to Rudder 2.6 nigtlies.
It does work for a fresh install on Debian (from Rudder-vagrant, branch 2.5, vagrant up server).
It does work on dev machine, or in other test environment (orch-3).
In fact, this is due to a bad initialisation of the .git in /var/rudder/configuration-repository that let the '/var/rudder/configuration-repository/.git/index.lock' file.
In fact, the problem is due to Git!
In debug mode, we can see:
[2013-04-24 18:39:45] DEBUG com.normation.rudder.repository.xml.GitNodeGroupArchiverImpl - Archived node group: /var/rudder/configuration-repository/groups/SystemGroups/hasPolicyServer-root.xml
[2013-04-24 18:39:45] DEBUG com.normation.rudder.repository.xml.GitNodeGroupArchiverImpl - Add file groups/SystemGroups/hasPolicyServer-root.xml from configuration repository
[2013-04-24 18:39:45] ERROR com.normation.rudder.services.servers.NewNodeManagerImpl - Error when trying to accept node f58dab50-6f5b-4463-b44e-5a5be0fce562. Rollbaking. <- Error when executing accept node process named accept_new_server:add_system_configuration_rules <- Could not update the technical group with ID 'NodeGroupId(hasPolicyServer-root)' <- Exception caught during execution of add command
So it's clearly when archiving in Git file groups/SystemGroups/hasPolicyServer-root.xml that the problem leading to Rollback occures.
After deleting the directory "/var/rudder/configuration-repositoy/.git" in the server, and then adding it back (git init && git add rules etc... && git commit -m "plop"), everything works again.
- Subject changed from Can't add a node on CentOS to A bad initialisation configuration-repository/.git leads to a lot of problems (can't add nodes, can't archive, ...)
- Description updated (diff)
After a new installation of rudder 2.5, the git repository will not works:
server:/var/rudder/configuration-repository # git status
fatal: unable to create '.git/index.lock': File exists
Simply deleting the file should suffice:
server:/var/rudder/configuration-repository # rm .git/index.lock
server:/var/rudder/configuration-repository # git status
# On branch master
nothing to commit (working directory clean)
Confirmed, I also had the lock file:
fatal: Unable to create '/var/rudder/configuration-repository/.git/index.lock': File exists.
(message displayed when trying to git add a file)
This bug was not reproducible on other installations, so let's say I wasn't lucky and something bad happens.
To prevent that to complettly block a Rudder server, we are going to add a CFEngine policy that check the presence of the lock file, and delete it if it is older than 5 minutes (a Git lock should never be that old, most Git operation takes only seconds even on heavy I/O loaded machine).
- Status changed from New to In progress
- Assignee set to Matthieu CERDA
- Subject changed from A bad initialisation configuration-repository/.git leads to a lot of problems (can't add nodes, can't archive, ...) to Add a promise to remove Git lock file on configuration-repository if older than 5 minutes
- Description updated (diff)
- Assignee deleted (
Matthieu CERDA)
- Assignee set to Matthieu CERDA
- Status changed from In progress to Pending technical review
- % Done changed from 0 to 100
- Pull Request set to https://github.com/Normation/rudder-techniques/pull/102
- Status changed from Pending technical review to Pending release
Applied in changeset policy-templates:commit:081574496aa9812d1197cf95e386b507d8622bc9.
Applied in changeset policy-templates:commit:bb9d9201c671077ba3a7f831d08cfb774661d8c4.
Applied in changeset policy-templates:commit:d6aa21ea7565811afc6303c1d21a08833c9c401e.
Applied in changeset policy-templates:commit:1458db9bb803be1691d0d092b94581e4fc8e97ca.
Applied in changeset policy-templates:commit:c1083ced4c6e30912b67bb1399946469ca795dc1.
Applied in changeset policy-templates:commit:73187a7af6a3e84c71c293287ab74871bc8f6533.
- Pull Request changed from https://github.com/Normation/rudder-techniques/pull/102 to https://github.com/Normation/rudder-techniques/pull/103
Applied in changeset policy-templates:commit:b562fb5897c02a0978a2d6f3bcad59741b2de93b.
Applied in changeset policy-templates:commit:d64139d955a9dfcafe82bba746221f7745a734c5.
- Status changed from Pending release to Released
Also available in: Atom
PDF
Updated by 
Updated by Nicolas PERRON 
Updated by Matthieu CERDA 
Updated by Jonathan CLARKE