Actions
Bug #19398
closed
Git error when deleting a node or archiving everything, and very slow git
Pull Request:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
To do
Fix check:
Error - Next version
Regression:
Description
Error is
[2021-06-04 14:04:50] ERROR com.normation.rudder.web.services.DisplayNode - There was an error while deleting Node '2f324fa3-f20d-41aa-80c7-81b5af04bfea' <- Accumulated: Error when deleting a node; cause was: Could not remove the node '2f324fa3-f20d-41aa-80c7-81b5af04bfea' from some groups; cause was: Could not update group '799cb91a-ebf4-4fbe-ab5e-c25daa0342db' to remove node '2f324fa3-f20d-41aa-80c7-81b5af04bfea'; cause was: Error when trying to archive automatically nodegroup change; cause was: SystemError: An error occured; cause was: org.eclipse.jgit.api.errors.JGitInternalException: Exception caught during execution of add command -> com.normation.rudder.repository.xml.GitArchiverUtils.$anonfun$commitAddFile$3(GitArchiverUtils.scala:129)
[2021-06-04 14:11:52] ERROR com.normation.rudder.web.snippet.administration.Archives - Error when exporting groups, parameters, directive library and rules. <- An error occured. Cause was: JGitInternalException: Exception caught during execution of add command [2021-06-04 14:11:52] ERROR com.normation.rudder.web.snippet.administration.Archives - org.eclipse.jgit.api.errors.JGitInternalException: Exception caught during execution of add command org.eclipse.jgit.errors.LockFailedException: Cannot lock /var/rudder/configuration-repository/.git/index. Ensure that no other process has an open file handle on the lock file /var/rudder/configuration-repository/.git/index.lock, then you may delete the lock file and retry.
lock file exists
-rw-rw----. 1 root rudder 0 27 mai 12:37 /var/rudder/configuration-repository/.git/index.lock
and no process access it
restarting the webapp doesn't solve it
removing the lock file, and retying t archive everything lead to an unending spining wheel, but it does finish to archive, and deletion of node also succeed
it's probably that the git is really slow
# git --version git version 1.8.3.1
The /var/rudder/configuration-repository/.git is 262MB large
# time git gc Counting objects: 16091, done. Delta compression using up to 10 threads. Compressing objects: 100% (4004/4004), done. Writing objects: 100% (16091/16091), done. Total 16091 (delta 13485), reused 14094 (delta 11780) real 0m4.368s user 0m2.343s sys 0m1.542s
git gc doesn't improve perf of git
git repack -d doesn't improve neither
Workaround: removing lock file /var/rudder/configuration-repository/.git/index.lock
Updated by Nicolas CHARLES almost 3 years ago
- Subject changed from Git error when deleting a node or archiving everything to Git error when deleting a node or archiving everything, and very slow git
Updated by Vincent MEMBRÉ almost 3 years ago
- Target version changed from 6.1.14 to 6.1.15
Updated by Vincent MEMBRÉ almost 3 years ago
- Target version changed from 6.1.15 to 6.1.16
Updated by François ARMAND over 2 years ago
- Related to Bug #18892: Technique Upgrade failed with a JGit error after upgrading from 5.0.20 to 6.1.9 added
Updated by François ARMAND over 2 years ago
- Related to Bug #18983: A writeLock must never be in a read lock for LDAP repo added
Updated by François ARMAND over 2 years ago
I think it could be due to the fact that we have several semaphore for write operation, which miss the point: each git repo need one semaphore for any write operation, else chaos ensues.
Updated by François ARMAND over 2 years ago
- Status changed from New to In progress
- Assignee set to François ARMAND
Updated by François ARMAND over 2 years ago
(and the related action are using several write operation, so if anything else tried a write operation during them, it could lead to the error).
Updated by François ARMAND over 2 years ago
- Status changed from In progress to Pending technical review
- Assignee changed from François ARMAND to Nicolas CHARLES
- Pull Request set to https://github.com/Normation/rudder/pull/3863
Updated by François ARMAND over 2 years ago
- Status changed from Pending technical review to Pending release
Applied in changeset rudder|e5367978529a42ac0bee6a247ad7976fc9b3a74d.
Updated by Nicolas CHARLES over 2 years ago
- Fix check changed from To do to Checked
Updated by Nicolas CHARLES over 2 years ago
- Fix check changed from Checked to Error - Next version
It is fixed when updating rules & directives at the same time, using API (10 parralels calls)
However it fails if, during the same time, I do an archive (and it fails consistently)
[2021-09-02 22:07:54+0200] ERROR com.normation.rudder.web.snippet.administration.Archives - Error when exporting groups, parameters, directive library and rules. <- An error occured. Cause was: JGitInternal Exception: Exception caught during execution of add command [2021-09-02 22:07:54+0200] ERROR com.normation.rudder.web.snippet.administration.Archives - org.eclipse.jgit.api.errors.JGitInternalException: Exception caught during execution of add command org.eclipse.jgit.errors.LockFailedException: Cannot lock /var/rudder/configuration-repository/.git/index. Ensure that no other process has an open file handle on the lock file /var/rudder/configuration-repository/.git/index.lock, then you may delete the lock file and retry. [2021-09-02 22:07:54+0200] ERROR com.normation.rudder.rest.RestUtils - "Could not create Rule <- Consistancy: Cannot create a rule with ID '0c1713ae-cb9d-4f7b-abda-ca38c5d643ea' : there is already a rule with the same id" [2021-09-02 22:07:54+0200] ERROR com.normation.rudder.rest.RestUtils - "Could not create Rule <- Consistancy: Cannot create a rule with ID '0c1713ae-cb9d-4f7b-abda-ca38c5d643ea' : there is already a rule with the same id"
Updated by Vincent MEMBRÉ over 2 years ago
This bug has been fixed in Rudder 6.1.16 and 6.2.10 which were released today.
Updated by François ARMAND about 2 years ago
- Related to Architecture #19910: JGit error when editing in parralel directives, rules & archiving everything added
Updated by François ARMAND about 2 years ago
- Status changed from Pending release to Resolved
Actions