Project

General

Profile

Actions

Bug #19398

closed

Git error when deleting a node or archiving everything, and very slow git

Added by Nicolas CHARLES over 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
N/A
Category:
System integration
Target version:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
To do
Fix check:
Error - Next version
Regression:

Description

Error is

[2021-06-04 14:04:50] ERROR com.normation.rudder.web.services.DisplayNode - There was an error while deleting Node '2f324fa3-f20d-41aa-80c7-81b5af04bfea' <- Accumulated: Error when deleting a node; cause was: Could not remove the node '2f324fa3-f20d-41aa-80c7-81b5af04bfea' from some groups; cause was: Could not update group '799cb91a-ebf4-4fbe-ab5e-c25daa0342db' to remove node '2f324fa3-f20d-41aa-80c7-81b5af04bfea'; cause was: Error when trying to archive automatically nodegroup change; cause was: SystemError: An error occured; cause was: org.eclipse.jgit.api.errors.JGitInternalException: Exception caught during execution of add command 
 -> com.normation.rudder.repository.xml.GitArchiverUtils.$anonfun$commitAddFile$3(GitArchiverUtils.scala:129)
[2021-06-04 14:11:52] ERROR com.normation.rudder.web.snippet.administration.Archives - Error when exporting groups, parameters, directive library and rules. <- An error occured. Cause was: JGitInternalException: Exception caught during execution of add command
[2021-06-04 14:11:52] ERROR com.normation.rudder.web.snippet.administration.Archives - org.eclipse.jgit.api.errors.JGitInternalException: Exception caught during execution of add command
org.eclipse.jgit.errors.LockFailedException: Cannot lock /var/rudder/configuration-repository/.git/index. Ensure that no other process has an open file handle on the lock file /var/rudder/configuration-repository/.git/index.lock, then you may delete the lock file and retry.

lock file exists

-rw-rw----. 1 root rudder 0 27 mai   12:37 /var/rudder/configuration-repository/.git/index.lock

and no process access it

restarting the webapp doesn't solve it

removing the lock file, and retying t archive everything lead to an unending spining wheel, but it does finish to archive, and deletion of node also succeed

it's probably that the git is really slow

# git --version
git version 1.8.3.1

The /var/rudder/configuration-repository/.git is 262MB large

# time git gc
Counting objects: 16091, done.
Delta compression using up to 10 threads.
Compressing objects: 100% (4004/4004), done.
Writing objects: 100% (16091/16091), done.
Total 16091 (delta 13485), reused 14094 (delta 11780)

real    0m4.368s
user    0m2.343s
sys    0m1.542s

git gc doesn't improve perf of git

git repack -d doesn't improve neither

Workaround: removing lock file /var/rudder/configuration-repository/.git/index.lock


Subtasks 1 (0 open1 closed)

Bug #19907: Compilation error after #19398 merge on upper branchesReleasedVincent MEMBRÉActions

Related issues 3 (1 open2 closed)

Related to Rudder - Bug #18892: Technique Upgrade failed with a JGit error after upgrading from 5.0.20 to 6.1.9ResolvedActions
Related to Rudder - Bug #18983: A writeLock must never be in a read lock for LDAP repoReleasedNicolas CHARLESActions
Related to Rudder - Architecture #19910: JGit error when editing in parralel directives, rules & archiving everythingNewActions
Actions #1

Updated by Nicolas CHARLES over 3 years ago

  • Description updated (diff)
Actions #2

Updated by Nicolas CHARLES over 3 years ago

  • Subject changed from Git error when deleting a node or archiving everything to Git error when deleting a node or archiving everything, and very slow git
Actions #3

Updated by Vincent MEMBRÉ over 3 years ago

  • Target version changed from 6.1.14 to 6.1.15
Actions #4

Updated by Vincent MEMBRÉ about 3 years ago

  • Target version changed from 6.1.15 to 6.1.16
Actions #5

Updated by François ARMAND about 3 years ago

  • Related to Bug #18892: Technique Upgrade failed with a JGit error after upgrading from 5.0.20 to 6.1.9 added
Actions #6

Updated by François ARMAND about 3 years ago

  • Related to Bug #18983: A writeLock must never be in a read lock for LDAP repo added
Actions #7

Updated by Nicolas CHARLES about 3 years ago

  • Description updated (diff)
Actions #8

Updated by François ARMAND about 3 years ago

I think it could be due to the fact that we have several semaphore for write operation, which miss the point: each git repo need one semaphore for any write operation, else chaos ensues.

Actions #9

Updated by François ARMAND about 3 years ago

  • Status changed from New to In progress
  • Assignee set to François ARMAND
Actions #10

Updated by François ARMAND about 3 years ago

(and the related action are using several write operation, so if anything else tried a write operation during them, it could lead to the error).

Actions #11

Updated by François ARMAND about 3 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from François ARMAND to Nicolas CHARLES
  • Pull Request set to https://github.com/Normation/rudder/pull/3863
Actions #12

Updated by François ARMAND about 3 years ago

  • Status changed from Pending technical review to Pending release
Actions #13

Updated by Nicolas CHARLES about 3 years ago

  • Fix check changed from To do to Checked
Actions #14

Updated by Nicolas CHARLES about 3 years ago

  • Fix check changed from Checked to Error - Next version

It is fixed when updating rules & directives at the same time, using API (10 parralels calls)

However it fails if, during the same time, I do an archive (and it fails consistently)

[2021-09-02 22:07:54+0200] ERROR com.normation.rudder.web.snippet.administration.Archives - Error when 
exporting groups, parameters, directive library and rules. <- An error occured. Cause was: JGitInternal
Exception: Exception caught during execution of add command
[2021-09-02 22:07:54+0200] ERROR com.normation.rudder.web.snippet.administration.Archives - org.eclipse.jgit.api.errors.JGitInternalException: Exception caught during execution of add command
org.eclipse.jgit.errors.LockFailedException: Cannot lock /var/rudder/configuration-repository/.git/index. Ensure that no other process has an open file handle on the lock file /var/rudder/configuration-repository/.git/index.lock, then you may delete the lock file and retry.
[2021-09-02 22:07:54+0200] ERROR com.normation.rudder.rest.RestUtils - "Could not create Rule <- Consistancy: Cannot create a rule with ID '0c1713ae-cb9d-4f7b-abda-ca38c5d643ea' : there is already a rule with the same id" 
[2021-09-02 22:07:54+0200] ERROR com.normation.rudder.rest.RestUtils - "Could not create Rule <- Consistancy: Cannot create a rule with ID '0c1713ae-cb9d-4f7b-abda-ca38c5d643ea' : there is already a rule with the same id" 

Actions #15

Updated by Vincent MEMBRÉ about 3 years ago

This bug has been fixed in Rudder 6.1.16 and 6.2.10 which were released today.

Actions #16

Updated by François ARMAND over 2 years ago

  • Related to Architecture #19910: JGit error when editing in parralel directives, rules & archiving everything added
Actions #17

Updated by François ARMAND over 2 years ago

  • Status changed from Pending release to Resolved
Actions

Also available in: Atom PDF