Bug #3526: Add a promise to remove Git lock file on configuration-repository if older than 5 minutes - Rudder - Issue Tracker

Actions

Copy link

Bug #3526

closed

Add a promise to remove Git lock file on configuration-repository if older than 5 minutes

Added by François ARMAND almost 11 years ago. Updated almost 11 years ago.

Status:

Released

Priority:

Assignee:

Matthieu CERDA

Category:

Web - Nodes & inventories

Target version:

2.5.3

Pull Request:

https://github.com/Normation/rudder-t...

Severity:

UX impact:

User visibility:

Effort required:

Priority:

Name check:

Fix check:

Regression:

Description

When trying to add a node on Rudder 2.5.2 on CentOS , we get the error:

   Error when trying to accept node f58dab50-6f5b-4463-b44e-5a5be0fce562 
<- Error when executing accept node process named accept_new_server:add_system_configuration_rules 
<- Could not update the technical group with ID 'NodeGroupId(hasPolicyServer-root)' 
<- Exception caught during execution of add command

This was from a fresh install from Rudder-vagrant (branch 2.5, vagrant up centos6).
It does not works any better when upgrading to Rudder 2.6 nigtlies.

It does work for a fresh install on Debian (from Rudder-vagrant, branch 2.5, vagrant up server).
It does work on dev machine, or in other test environment (orch-3).

In fact, this is due to a bad initialisation of the .git in /var/rudder/configuration-repository that let the '/var/rudder/configuration-repository/.git/index.lock' file.

Related issues 2 (0 open — 2 closed)

Actions

Copy link

Updated by François ARMAND almost 11 years ago

In fact, the problem is due to Git!

In debug mode, we can see:

[2013-04-24 18:39:45] DEBUG com.normation.rudder.repository.xml.GitNodeGroupArchiverImpl - Archived node group: /var/rudder/configuration-repository/groups/SystemGroups/hasPolicyServer-root.xml
[2013-04-24 18:39:45] DEBUG com.normation.rudder.repository.xml.GitNodeGroupArchiverImpl - Add file groups/SystemGroups/hasPolicyServer-root.xml from configuration repository
[2013-04-24 18:39:45] ERROR com.normation.rudder.services.servers.NewNodeManagerImpl - Error when trying to accept node f58dab50-6f5b-4463-b44e-5a5be0fce562. Rollbaking. <- Error when executing accept node process named accept_new_server:add_system_configuration_rules <- Could not update the technical group with ID 'NodeGroupId(hasPolicyServer-root)' <- Exception caught during execution of add command

So it's clearly when archiving in Git file groups/SystemGroups/hasPolicyServer-root.xml that the problem leading to Rollback occures.

After deleting the directory "/var/rudder/configuration-repositoy/.git" in the server, and then adding it back (git init && git add rules etc... && git commit -m "plop"), everything works again.

Actions

Copy link

Updated by François ARMAND almost 11 years ago

Subject changed from Can't add a node on CentOS to A bad initialisation configuration-repository/.git leads to a lot of problems (can't add nodes, can't archive, ...)
Description updated (diff)

Actions

Copy link

Updated by Nicolas PERRON almost 11 years ago

After a new installation of rudder 2.5, the git repository will not works:

server:/var/rudder/configuration-repository # git status
fatal: unable to create '.git/index.lock': File exists

Simply deleting the file should suffice:

server:/var/rudder/configuration-repository # rm .git/index.lock

server:/var/rudder/configuration-repository # git status
# On branch master
nothing to commit (working directory clean)

Actions

Copy link

Updated by François ARMAND almost 11 years ago

Confirmed, I also had the lock file:

fatal: Unable to create '/var/rudder/configuration-repository/.git/index.lock': File exists.

(message displayed when trying to git add a file)

Actions

Copy link

Updated by François ARMAND almost 11 years ago

This bug was not reproducible on other installations, so let's say I wasn't lucky and something bad happens.
To prevent that to complettly block a Rudder server, we are going to add a CFEngine policy that check the presence of the lock file, and delete it if it is older than 5 minutes (a Git lock should never be that old, most Git operation takes only seconds even on heavy I/O loaded machine).

Actions

Copy link

Updated by Matthieu CERDA almost 11 years ago

Status changed from New to In progress
Assignee set to Matthieu CERDA

Actions

Copy link

Updated by François ARMAND almost 11 years ago

Subject changed from A bad initialisation configuration-repository/.git leads to a lot of problems (can't add nodes, can't archive, ...) to Add a promise to remove Git lock file on configuration-repository if older than 5 minutes
Description updated (diff)
Assignee deleted (~~Matthieu CERDA~~)

Actions

Copy link

Updated by Matthieu CERDA almost 11 years ago

Assignee set to Matthieu CERDA

Actions

Copy link

Updated by Matthieu CERDA almost 11 years ago

Status changed from In progress to Pending technical review
% Done changed from 0 to 100
Pull Request set to https://github.com/Normation/rudder-techniques/pull/102

PR available.

Actions

Copy link

#10