Rudder

This may be due to #20350 which was fixed since rc1.

You can try to manually re-run the upgrade script:

/opt/rudder/bin/rudder-upgrade

and then (if it worked without error), check for uncommited files in /var/rudder/configuration-repository/techniques/system. If there are uncommited files, add them and commit.

Then restart the web app with systemctl restart rudder-jetty.

We still should try to handle that error correctly, which could be:

- have a sync wait for the lock, with a timeout.

There is little reason to fail the whole migration because of a lock not available right now.
(of course, it will be complicated, because that error can happen in a multitude of places, git has a very fuzzy concept of atomicity (of course), and there's likely some place where a concurrent change will mean that bad things happened (tm). But here, it's likely just the gitgc that it taking a bit too much time and we got the lock error at the begining of the migration)

Thanks Alexis. Seems there was issue even before the upgrade, there was stale lock in repository.

1. removed /var/rudder/configuration-repository/.git/refs/heads/master.lock and commited all changes
2. checked 6.2, everything is fine
3. run the package upgrade to 7.0 rc1

And now I've another issues. Again there was err during install:

...
  Running scriptlet: rudder-webapp-1398866025:7.0.0.rc1-1.EL.8.x86_64                                                                                                                                                                                            5/11
[]
Job for rudder-jetty.service failed because the control process exited with error code.
See "systemctl status rudder-jetty.service" and "journalctl -xe" for details.
...
  Running scriptlet: rudder-webapp-1398866025:7.0.0.rc1-1.EL.8.x86_64                                                                                                                                                                                           11/11
Job for rudder-jetty.service failed because the control process exited with error code.
See "systemctl status rudder-jetty.service" and "journalctl -xe" for details.
warning: %posttrans(rudder-webapp-1398866025:7.0.0.rc1-1.EL.8.x86_64) scriptlet failed, exit status 1
...

In webapp log I can see:

[2021-12-02 10:31:19+0100] INFO  bootchecks - Check LDAP and PostgreSQL connection
[2021-12-02 10:31:19+0100] INFO  bootchecks - LDAP and PostgreSQL connection are OK
[2021-12-02 10:31:19+0100] INFO  bootchecks - Check if migration of system configuration object and allowed networks to Rudder 7.0 format is needed
[2021-12-02 10:31:19+0100] INFO  git-repository - git-gc performed on fact-repository in PT0.421S
[2021-12-02 10:31:20+0100] INFO  migration - Check if all Rudder 7.0 system technique are correctly loaded in technique library
[2021-12-02 10:31:20+0100] INFO  git-repository - git-gc performed on configuration-repository in PT0.996S
[2021-12-02 10:31:20+0100] ERROR migration - Error during migration of policy server system configuration to Rudder V7.0 data. It is likely that Rudder won't work as expected. Please check the log above. Error was: SystemError: An error occured; cause was: java.nio.file.NoSuchFileException: /var/rudder/configuration-repository/techniques/system/rudder-service-postgresql/1.0/placeholder
 ->
[2021-12-02 10:31:20+0100] INFO  bootchecks - Check mandatory DIT entries
[2021-12-02 10:31:20+0100] ERROR bootchecks - Error when checking for mandatory entries for 'root' server in the DIT. <- Inconsistency: Missing required entry 'directiveId=common-hasPolicyServer-root,activeTechniqueId=common,techniqueCategoryId=Rudder Internal,techniqueCategoryId=Active Techniques,ou=Rudder,cn=rudder-configuration'. This is most likelly because Rudder was not initialized. Please run /opt/rudder/bin/rudder-init to set it up.
[2021-12-02T10:31:20.590+01:00] ERROR FATAL An error happen during Rudder boot. Rudder will stop now. Error: SystemError: An error occured; cause was: javax.servlet.UnavailableException: Error when checking for mandatory entries for 'root' server in the DIT. <- Inconsistency: Missing required entry 'directiveId=common-hasPolicyServer-root,activeTechniqueId=common,techniqueCategoryId=Rudder Internal,techniqueCategoryId=Active Techniques,ou=Rudder,cn=rudder-configuration'. This is most likelly because Rudder was not initialized. Please run /opt/rudder/bin/rudder-init to set it up.
 ->
javax.servlet.UnavailableException: Error when checking for mandatory entries for 'root' server in the DIT. <- Inconsistency: Missing required entry 'directiveId=common-hasPolicyServer-root,activeTechniqueId=common,techniqueCategoryId=Rudder Internal,techniqueCategoryId=Active Techniques,ou=Rudder,cn=rudder-configuration'. This is most likelly because Rudder was not initialized. Please run /opt/rudder/bin/rudder-init to set it up.

# ls -la /var/rudder/configuration-repository/techniques/system/
total 23
drwxrwsr-x 4 root rudder   5 Dec  2 10:20 .
drwxrwsr-x 9 root rudder  11 Jul 31  2020 ..
-rw-rw-r-- 1 root rudder 227 Dec  9  2020 category.xml
drwxr-xr-x 3 root root     3 Nov 25 02:29 common
drwxrwsr-x 3 root rudder   3 Nov 25 02:29 inventory

I tried manual upgrade:

# /opt/rudder/bin/rudder-upgrade
INFO: Checking if rudder-web.properties database access credentials are all right... LDAP OK,  SQL OK
INFO: A Technique library reload is needed and has been scheduled.
Configuration file does not set auto update of Techniques by default (value is false). Skipping
INFO: Checking PostgreSQL service status............ FAILED
PostgreSQL service verification failed after 10 tries.
INFO: Checking LDAP service status... OK
Plugins were not updated, because connection test is failing, please upgrade your plugins manually

ERROR: The migration has failed in some steps. Check previous error messages.
Please restart the failed service(s), and start the migration script again.
(try service rudder-server restart)
Once it is working, run:
# /opt/rudder/bin/rudder-upgrade

This # rudder server upgrade-techniques -o fixed issues with missing files under techniques. But still getting error during upgrade.

# /opt/rudder/bin/rudder-upgrade
INFO: Checking if rudder-web.properties database access credentials are all right... LDAP OK,  SQL OK
INFO: A Technique library reload is needed and has been scheduled.
INFO: A Technique library reload is needed and has been scheduled.
INFO: A Technique library reload is needed and has been scheduled.
INFO: A Technique library reload is needed and has been scheduled.
INFO: A Technique library reload is needed and has been scheduled.
INFO: A Technique library reload is needed and has been scheduled.
Configuration file does not set auto update of Techniques by default (value is false). Skipping
INFO: Checking PostgreSQL service status............ FAILED
PostgreSQL service verification failed after 10 tries.
INFO: Checking LDAP service status... OK
Plugins were not updated, because connection test is failing, please upgrade your plugins manually

ERROR: The migration has failed in some steps. Check previous error messages.
Please restart the failed service(s), and start the migration script again.
(try service rudder-server restart)
Once it is working, run:
# /opt/rudder/bin/rudder-upgrade

But webapp is up and accessible, log:

[2021-12-02 10:45:52+0100] INFO  bootchecks - Check LDAP and PostgreSQL connection
[2021-12-02 10:45:52+0100] INFO  bootchecks - LDAP and PostgreSQL connection are OK
[2021-12-02 10:45:52+0100] INFO  bootchecks - Check if migration of system configuration object and allowed networks to Rudder 7.0 format is needed
[2021-12-02 10:45:52+0100] INFO  git-repository - git-gc performed on fact-repository in PT0.316S
[2021-12-02 10:45:53+0100] INFO  migration - Check if all Rudder 7.0 system technique are correctly loaded in technique library
[2021-12-02 10:45:53+0100] INFO  git-repository - git-gc performed on configuration-repository in PT1.188S
[2021-12-02 10:45:54+0100] INFO  techniques.reader - Reloading technique library, found modified technique(s): ['rudder-service-postgresql': updated (1.0: updated)], ['server-common': updated (1.0: updated)], ['rudder-service-relayd': updated (1.0: updated)], ['rudder-service-webapp': updated (1.0: updated)], ['rudder-service-apache': updated (1.0: updated)], ['rudder-service-slapd': updated (1.0: updated)]
[2021-12-02 10:45:54+0100] INFO  com.normation.rudder.services.policies.DeployOnTechniqueCallback - Reload Technique library to allows migration to 7.x system techniques
[2021-12-02 10:45:54+0100] INFO  migration - Creating new system directives for policy server 'root'
[2021-12-02 10:45:54+0100] INFO  policy.generation - Start policy generation, checking updated rules
[2021-12-02 10:45:54+0100] INFO  migration - Creating new system rules for policy server 'root'
[2021-12-02 10:45:54+0100] ERROR rudder-ldap-entity-mapper - Invalid attribute 'special:all_servers_with_role' for entry ruleTarget.
[2021-12-02 10:45:54+0100] WARN  com.normation.rudder.repository.ldap.RoLDAPNodeGroupRepository - Error when mapping entry with DN 'ruleTarget=special:all_servers_with_role,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration' from node groups library, that entry will be ignored; cause was: UnexpectedObject: Can not unserialize target, 'special:all_servers_with_role' does not match any known target format
[2021-12-02 10:45:54+0100] ERROR com.normation.rudder.domain.policies.RuleTarget - Error when trying to read the following serialized Rule target as a composite target: 'special:all_nodes_without_role'. Reported parsing error cause was: unknown token s
Near: sp
[2021-12-02 10:45:54+0100] WARN  com.normation.rudder.repository.ldap.RoLDAPNodeGroupRepository - Error when mapping entry with DN 'ruleTarget=special:all_nodes_without_role,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration' from node groups library, that entry will be ignored; cause was: UnexpectedObject: Can not unserialize target, 'special:all_nodes_without_role' does not match any known target format
[2021-12-02 10:45:54+0100] INFO  migration - Starting migration for system rules for policy server 'root''
[2021-12-02 10:45:54+0100] INFO  migration - [root techniques] delete old rule 'hasPolicyServer-root'
[2021-12-02 10:45:54+0100] INFO  migration - [root techniques] delete old rule 'root-DP'
[2021-12-02 10:45:54+0100] INFO  migration - [root techniques] delete old directive 'common-root'
[2021-12-02 10:45:54+0100] INFO  migration - Migration of system techniques, directives and rules done for 'root'
[2021-12-02 10:45:54+0100] INFO  migration - Deleting old technique 'distributePolicy' information
[2021-12-02 10:45:54+0100] INFO  migration - Deleting old system object for deprecated server-roles: activeTechniqueId=server-roles,techniqueCategoryId=Rudder Internal,techniqueCategoryId=Active Techniques,ou=Rudder,cn=rudder-configuration
[2021-12-02 10:45:54+0100] INFO  migration - Deleting old system object for deprecated server-roles: ruleId=server-roles,ou=Rules,ou=Rudder,cn=rudder-configuration
[2021-12-02 10:45:54+0100] INFO  migration - Deleting old system object for deprecated server-roles: ruleTarget=special:all_nodes_without_role,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration
[2021-12-02 10:45:54+0100] INFO  migration - Deleting old system object for deprecated server-roles: ruleTarget=special:all_servers_with_role,groupCategoryId=SystemGroups,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration
[2021-12-02 10:45:54+0100] INFO  migration - Migration of all system configuration to Rudder 7.0: done
[2021-12-02 10:45:54+0100] INFO  bootchecks - Check mandatory DIT entries
[2021-12-02 10:45:54+0100] INFO  policy.generation - [metrics] Xmx:991 MB nodes:34 (cached:34) rules:21 (enabled:20) techniques:52 (enabled:52) directives:76 (enabled:76) groups:13 (dynamic:11) parameters:2
[2021-12-02 10:45:54+0100] INFO  bootchecks - All the required DIT entries are present in the LDAP directory
[2021-12-02 10:45:54+0100] INFO  bootchecks - Check initialization of User Technique Library
[2021-12-02 10:45:54+0100] INFO  bootchecks - Check that `rudder` global parameter matches default value
[2021-12-02 10:45:55+0100] INFO  bootchecks - Reseting global parameter 'rudder' from system provider to value: {"packages":{"installed_cache_expire":60,"updates_cache_expire":240},"server":{"cf_serverd_bind_address":"::"}}
[2021-12-02 10:45:55+0100] INFO  policy.generation.timing - Policy generation failed after:
[2021-12-02 10:45:55+0100] ERROR policy.generation.manager - Error when updating policy, reason was: Cannot build Rule vals <- Could not find configuration vals <- Version '1.0' of technique 'distributePolicy' is not available for directive 'Distribute Policy' [root-distributePolicy]
[2021-12-02 10:45:55+0100] ERROR policy.generation.manager - Policy update error for process '9374' at 2021-12-02 10:45:55: Cannot build Rule vals
[2021-12-02 10:45:55+0100] INFO  bootchecks - Check event log migration format 5 -> 6
[2021-12-02 10:45:55+0100] INFO  bootchecks - Check existence of at least one archive of the configuration
[2021-12-02 10:45:55+0100] INFO  migration - No migration detected by migration script (table 'migrationeventlog' is empty or does not exist)
[2021-12-02 10:45:55+0100] INFO  bootchecks - First full archive of configuration-repository items done
[2021-12-02 10:45:55+0100] INFO  bootchecks - Check for force reload of Techniques library
[2021-12-02 10:45:55+0100] INFO  bootchecks - Flag file '/opt/rudder/etc/force_technique_reload' found, reload Technique library now
[2021-12-02 10:45:55+0100] INFO  techniques.reader - Reloading technique library, no modified techniques found
[2021-12-02 10:45:55+0100] INFO  com.normation.rudder.services.policies.DeployOnTechniqueCallback - Reload Technique library at start up
[2021-12-02 10:45:55+0100] INFO  bootchecks - Successfully reloaded Technique library on start up. now deleting flag file '/opt/rudder/etc/force_technique_reload'
[2021-12-02 10:45:55+0100] INFO  bootchecks - Flag file '/opt/rudder/etc/force_technique_reload' successfully removed
[2021-12-02 10:45:55+0100] INFO  bootchecks - Regenerate all ncf techniques
[2021-12-02 10:45:55+0100] INFO  bootchecks - started
[2021-12-02 10:45:56+0100] INFO  bootchecks - techniques - update
[2021-12-02 10:45:56+0100] INFO  bootchecks - methods
[2021-12-02 10:45:56+0100] INFO  bootchecks - techniques - read
[2021-12-02 10:45:57+0100] INFO  techniques.reader - Reloading technique library, no modified techniques found
[2021-12-02 10:45:57+0100] INFO  com.normation.rudder.services.policies.DeployOnTechniqueCallback - Update Technique library after updating all techniques at start up
[2021-12-02 10:45:57+0100] INFO  bootchecks - Trigger policy update automatically at start
[2021-12-02 10:45:57+0100] INFO  bootchecks - Create system api token
[2021-12-02 10:45:57+0100] INFO  bootchecks - System api token file created in /var/rudder/run/api-token
[2021-12-02 10:45:57+0100] INFO  bootchecks - Load node compliance cache
[2021-12-02 10:45:57+0100] INFO  policy.generation - Start policy generation, checking updated rules
2021-12-02 10:45:57.297:INFO:oejshC.rudder:main: Initializing Spring root WebApplicationContext
[2021-12-02 10:45:57+0100] INFO  policy.generation - [metrics] Xmx:991 MB nodes:34 (cached:34) rules:20 (enabled:19) techniques:50 (enabled:50) directives:73 (enabled:73) groups:13 (dynamic:11) parameters:2
[2021-12-02 10:45:58+0100] INFO  application - Configured authentication provider(s): [rootAdmin, file]
[2021-12-02 10:45:58+0100] INFO  application - No master admin account is defined. You can define one with 'rudder.auth.admin.login' and 'rudder.auth.admin.password' properties in the configuration file
[2021-12-02 10:45:58+0100] INFO  application - Add backend providers 'Default authentication backends provider: 'file','rootAdmin'
[2021-12-02 10:45:58+0100] INFO  application - Using configuration file defined by JVM property -Drudder.authFile : /opt/rudder/etc/rudder-users.xml
[2021-12-02 10:45:59+0100] INFO  bootstrap.liftweb.Boot - classpath
[2021-12-02 10:45:59+0100] INFO  policy.generation - Configuration of 34 nodes were updated, their policies are going to be written
[2021-12-02 10:45:59+0100] DEBUG explain_compliance - Updated node configuration ids: [61aa184d-9785-4934-a0b9-e9c24472e472:20211202-101229-2195e1c7->20211202-104557-4199ac28][935de924-6bba-4dd5-86ee-11ec8d5794e8:20211202-101229-c3155752->20211202-104557-ed6aa87d][root:20211202-101229-87e430bd->20211202-104557-17dcafe9][a58e5496-e0d9-4173-a03b-b1b1dae5a3f9:20211119-172123-c587f8e6->20211202-104557-1bba841a][4fd742ed-d997-43ab-951d-67eb93a6eac0:20211202-101229-b9493726->20211202-104557-1a26104f][c4cfc042-ac90-4714-b5f1-9a68a7fa3a51:20211202-101229-e56cb1e8->20211202-104557-689ccc8c][55d72ad7-98eb-46fc-a929-e54f5638d17b:20211119-172123-500b841d->20211202-104557-c68f56ad][7a670f56-85e5-4eb8-b0c3-006ec6bdbb52:20211202-101229-72208bba->20211202-104557-aba3210e][b053570c-b0bb-4aad-9adf-e62d19afae56:20211202-101229-ae3b1484->20211202-104557-a17d89dc][0d5f16ab-a672-4daf-be18-f283dfc42ed7:20211119-172123-5f0aaea1->20211202-104557-1d1d8ed1][c9625141-08b1-43ee-a2bd-052322980bc5:20211202-101229-cdc3e5bd->20211202-104557-6ae67599][72ddae29-749b-4a98-81fd-5040093d8cb7:20211124-020653-e35fee2e->20211202-104557-560a1362][f98b2e2f-c19c-421d-84cc-a14a80214a43:20211202-101229-75e4071a->20211202-104557-fc2dcfaf][aa308ea6-bbea-4a65-801b-818dd86f2588:20211119-172123-b6c60026->20211202-104557-e3b738e0][63f54468-5496-42ae-a485-a9d98febacd3:20211202-101229-7a11cee5->20211202-104557-1c3c78a1][745d10a5-db44-4363-8179-e54606503e9c:20211202-101229-e1f573c7->20211202-104557-35bb488d][4a7d1b3f-4e4f-4a7f-ab34-0f2bb09acdf6:20211130-213710-61f23e0a->20211202-104557-12c85051][8783d4ba-0f27-4f5d-b1dc-2b63b67d72c4:20211202-101229-7be9b136->20211202-104557-4e3889ff][e6d314ac-65fc-4b61-9b3d-00ddf4d9a34b:20211202-101229-6521155a->20211202-104557-e4c4a3e7][232694b6-7008-4afb-84a6-178301536099:20211202-101229-89cdb013->20211202-104557-8044504c][bed910e3-5925-4b31-a77a-1d001a45d7f1:20211202-101229-e4a0d614->20211202-104557-22cca5b][d0a75128-214c-4a62-b792-04a5cf3c4743:20211119-172123-e1997860->20211202-104557-cde2bb35][56bdcb62-a2bf-40d5-b005-50f5015f36dd:20211202-101229-351f26a8->20211202-104557-7fb006f][f4391755-bd77-4fbd-898b-5d74518ab1a9:20211123-092938-6610ce92->20211202-104557-af21d9d0][cd3d15e8-d914-412f-828b-a2d584b51c09:20211202-101229-b876dcec->20211202-104557-a826f5a2][5613ac1f-577e-42ca-9bdd-a30cc1d026f1:20211124-020938-20cef947->20211202-104557-9440e22c][94351ee7-58fb-4d76-8108-7a2c57b8d234:20211119-172123-c59f4a8b->20211202-104557-59713c47][f80bb413-01f5-499e-aaf7-ff44a172d99b:20211202-101229-aefc3587->20211202-104557-e51d001d][013ce4ac-d971-4ca3-8615-1099fbb4ae45:20211202-101229-7a55443->20211202-104557-8e1914be][a9c1d6a0-a0a9-48b3-99a5-a5290f455bcf:20211123-092938-1e58d5d6->20211202-104557-385fdbae][0f262c6c-f17f-48de-a63a-176aa2c58839:20211119-172123-77d43fd2->20211202-104557-a4119583][46ad4ed0-61c1-4a33-bb72-a61fc238868d:20211202-101229-ad93463b->20211202-104557-1ed4d34b][e37c338b-c1e2-43a8-b028-bca4f89fb5bc:20211123-092938-b82d33b->20211202-104557-f373a5f6][63302a19-05b1-4c8f-8943-6dff0f68ecdc:20211123-092938-a022477c->20211202-104557-b13ae997]
[2021-12-02 10:45:59+0100] INFO  org.reflections.Reflections - Reflections took 116 ms to scan 2 urls, producing 15 keys and 53 values
[2021-12-02 10:45:59+0100] INFO  application - INFO Configure inventory processing with parallelism of '2' and queue size of '50'
[2021-12-02 10:45:59+0100] INFO  scheduled.job.metrics - Starting node count historization batch (min:15 m; max:4 h)
[2021-12-02 10:45:59+0100] INFO  inventory-processing - Incoming inventory watcher started - process existing inventories
[2021-12-02 10:45:59+0100] INFO  application - Application Rudder started
2021-12-02 10:45:59.935:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@34c4973{/rudder,file:///var/rudder/tmp/jetty/jetty-rudder.war.dir/webapp/,AVAILABLE}{/opt/rudder/share/webapps/rudder.war}
2021-12-02 10:45:59.968:INFO:oejs.AbstractConnector:main: Started ServerConnector@5cdf221a{HTTP/1.1, (http/1.1)}{127.0.0.1:8080}

Salut François,

thanks a lot for updated info. Seems previous steps fixed all my issues. Feel free to close this ticket.

Thanks
Marek

This issue occurred in 7.2.4 on one of our customer platform while editing an ncf technique.
Webapp log did contain:

[2023-04-07 14:36:17+0200] ERROR com.normation.rudder.rest.RestUtils - "Could not update ncf technique <- error when committing Technique 'my_technique/1.0' <- An error occurred. Cause was: ConcurrentRefUpdateException: Could not lock HEAD. RefUpdate return code was: LOCK_FAILURE" 
[2023-04-07 14:36:43+0200] ERROR com.normation.rudder.rest.RestUtils - "Could not update ncf technique <- error when committing Technique 'my_technique/1.0' <- An error occurred. Cause was: ConcurrentRefUpdateException: Could not lock HEAD. RefUpdate return code was: LOCK_FAILURE" 

We did solve it with the following commands:

cd /var/rudder/configuration-repository
rm .git/refs/heads/master.lock
git add -u
git commit -m "Removing master lock and updating pending techniques" 
rudder server reload-techniques