Bug #19044
closed
Upgrade script fails in system technique update (empty commit)
Description
WORAROUND
This error seems to only impact debian-based system.
If you are impacted by that error, you can correct by yourself without waiting for the next patch release being done to address it with the command:
sed -i -e 's/rsync --delete/rsync --delete --ignore-times/g' /opt/rudder/bin/rudder-upgrade
Then, execute again the post-upgrade script:
/opt/rudder/bin/rudder-upgrade
It should terminate without error
ORIGINAL REPORT
On the hub server, I'm getting an error when upgrading, though it seems to end correctly.
See below :
apt-get install rudder-server-root
Lecture des listes de paquets... Fait
Construction de l'arbre des dépendances
Lecture des informations d'état... Fait
Les paquets supplémentaires suivants seront installés :
rudder-agent rudder-reports rudder-server-relay rudder-webapp
Les paquets suivants seront mis à jour :
rudder-agent rudder-reports rudder-server-relay rudder-server-root rudder-webapp
5 mis à jour, 0 nouvellement installés, 0 à enlever et 110 non mis à jour.
Il est nécessaire de prendre 141 Mo dans les archives.
Après cette opération, 1 717 ko d'espace disque supplémentaires seront utilisés.
Souhaitez-vous continuer ? [O/n]
Réception de :1 http://repository.rudder.io/apt/latest buster/main amd64 rudder-server-root all 6.2.3-debian10 [3 324 B]
Réception de :2 http://repository.rudder.io/apt/latest buster/main amd64 rudder-reports all 6.2.3-debian10 [8 552 B]
Réception de :3 http://repository.rudder.io/apt/latest buster/main amd64 rudder-agent amd64 6.2.3-debian10 [3 206 kB]
Réception de :4 http://repository.rudder.io/apt/latest buster/main amd64 rudder-server-relay amd64 6.2.3-debian10 [2 210 kB]
Réception de :5 http://repository.rudder.io/apt/latest buster/main amd64 rudder-webapp amd64 6.2.3-debian10 [135 MB]
141 Mo réceptionnés en 14s (9 881 ko/s)
Lecture des fichiers de modifications (« changelog »)... Terminé
Préconfiguration des paquets...
(Lecture de la base de données... 51368 fichiers et répertoires déjà installés.)
Préparation du dépaquetage de .../rudder-server-root_6.2.3-debian10_all.deb ...
Dépaquetage de rudder-server-root (6.2.3-debian10) sur (6.2.2-debian10) ...
Préparation du dépaquetage de .../rudder-reports_6.2.3-debian10_all.deb ...
Dépaquetage de rudder-reports (6.2.3-debian10) sur (6.2.2-debian10) ...
Préparation du dépaquetage de .../rudder-agent_6.2.3-debian10_amd64.deb ...
INFO: A back up copy of the /opt/rudder/etc/uuid.hive has been created in /var/backups/rudder
INFO: A back up copy of the /var/rudder/cfengine-community/policy_server.dat has been created in /var/backups/rudder
INFO: A back up copy of the /var/rudder/cfengine-community/ppkeys has been created in /var/backups/rudder
INFO: A back up copy of the /opt/rudder/etc/ssl/agent.cert has been created in /var/backups/rudder
Dépaquetage de rudder-agent (6.2.3-debian10) sur (6.2.2-debian10) ...
Préparation du dépaquetage de .../rudder-server-relay_6.2.3-debian10_amd64.deb ...
Dépaquetage de rudder-server-relay (6.2.3-debian10) sur (6.2.2-debian10) ...
Paramétrage de rudder-agent (6.2.3-debian10) ...
INFO: Scheduling an inventory during next run...
INFO: No disable file detected and no agent executor process either. Restarting agent service... Done
FINISH: Rudder agent check ran properly, please look at messages above to see if there has been any error.
Paramétrage de rudder-server-relay (6.2.3-debian10) ...
INFO: Creating users ... Done
INFO: Configuring systemd services... Done
INFO: Starting services... Done
INFO: rudder-server-relay setup complete.
(Lecture de la base de données... 51368 fichiers et répertoires déjà installés.)
Préparation du dépaquetage de .../rudder-webapp_6.2.3-debian10_amd64.deb ...
************************************************************************
INFO: dpkg may ask you if you want to replace rudder-web.properties or
INFO: slapd.conf with the maintainer version.
INFO:
INFO: If you accept, please keep in mind that you will lose any manual
INFO: modifications in this file, like batchs configuration, or LDAP
INFO: authentication. It will also reset passwords to default values and break
INFO: upgrade scripts.
INFO:
INFO: We strongly advise you to respond NO to this question
INFO: ('keep your currently-installed version'). Rudder's upgrade script will
INFO: take care of upgrading these file automatically.
INFO:
************************************************************************
Dépaquetage de rudder-webapp (6.2.3-debian10) sur (6.2.2-debian10) ...
Paramétrage de rudder-reports (6.2.3-debian10) ...
INFO: Setting PostgreSQL as a boot service...Synchronizing state of postgresql.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable postgresql
Done
INFO: Waiting for PostgreSQL to be up... Done
Paramétrage de rudder-webapp (6.2.3-debian10) ...
Installation de la nouvelle version du fichier de configuration /opt/rudder/etc/rudder-jetty.conf ...
INFO: Creating groups ... Done
INFO: Creating users ... Done
INFO: Removing ncf-api-venv user and group ... Done
INFO: Setting up systemd ...INFO: Restarting rudder-slapd... Done
INFO: Launching script to check if a migration is needed ...
******************************************************************************
ERROR: rudder-webapp postinstall script failed !
Trying to recover the problem, you should check that your instance is properly working
You may need to execute: apt-get install -f
You should also try to manually execute: /opt/rudder/bin/rudder-upgrade
******************************************************************************
Paramétrage de rudder-server-root (6.2.3-debian10) ...
Traitement des actions différées (« triggers ») pour rsyslog (8.1901.0-1) ...
Traitement des actions différées (« triggers ») pour libc-bin (2.28-10) ...
Such errors should not happen, please open an issue for this problem on
https://issues.rudder.io/projects/rudder/issues/newFiles
Updated by Vincent MEMBRÉ over 3 years ago
Hello Nicolas,
Thanks for reporting !
Can you look into /var/log/rudder/install for the log file about rudder-webapp update (it should be named rudder-webapp-<timestamp> ) if you got any error ? or even paste the whole content here!
Updated by Nicolas Ecarnot over 3 years ago
Hi Vincent,
here it is :
# for f in `ls /tmp/rudder-install-logs/`; do echo '-----------'$f'---------'; cat $f;done -----------rudder-agent-20210318133217.log--------- jeudi 18 mars 2021, 13:32:17 (UTC+0100) - Starting rudder-agent post installation script rudder-cf-serverd: enabled rudder-cf-execd: enabled ok: stop service rudder-agent succeeded INFO: No disable file detected and no agent executor process either. Restarting agent service... Done FINISH: Rudder agent check ran properly, please look at messages above to see if there has been any error. jeudi 18 mars 2021, 13:32:19 (UTC+0100) - Ending rudder-agent post installation script -----------rudder-reports-20210318133235.log--------- jeudi 18 mars 2021, 13:32:35 (UTC+0100) - Starting rudder-reports post installation script jeudi 18 mars 2021, 13:32:36 (UTC+0100) - Ending rudder-reports post installation script -----------rudder-server-relay-20210318133219.log--------- jeudi 18 mars 2021, 13:32:19 (UTC+0100) - Starting rudder-server-relay post installation script Adding password for user rudder Adding password for user rudder jeudi 18 mars 2021, 13:32:30 (UTC+0100) - Ending rudder-server-relay post installation script -----------rudder-server-root-20210318133238.log--------- jeudi 18 mars 2021, 13:32:38 (UTC+0100) - Starting rudder-server-root post installation script jeudi 18 mars 2021, 13:32:38 (UTC+0100) - Ending rudder-server-root post installation script -----------rudder-webapp-20210318133236.log--------- jeudi 18 mars 2021, 13:32:36 (UTC+0100) - Starting rudder-webapp post installation script INFO: Checking if rudder-web.properties database access credentials are all right... LDAP OK, SQL OK ERROR: An error happened during the step: Upgrade system Techniques (l 191) ERROR: The migration will now stop ERROR: Please try to run /opt/rudder/bin/rudder-upgrade again -----------rudder-webapp-fail-20210318133237.log--------- INFO: Correcting permissions on /var/rudder/configuration-repository... Done
Updated by Alexis Mousset over 3 years ago
What does git status give in /var/rudder/configuration-repository?
Updated by Nicolas Ecarnot over 3 years ago
/var/rudder/configuration-repository# git status
Sur la branche master
Modifications qui ne seront pas validées :
(utilisez "git add <fichier>..." pour mettre à jour ce qui sera validé)
(utilisez "git checkout -- <fichier>..." pour annuler les modifications dans la copie de travail)
modifié : .gitignore
Fichiers non suivis:
(utilisez "git add <fichier>..." pour inclure dans ce qui sera validé)
shared-files/
aucune modification n'a été ajoutée à la validation (utilisez "git add" ou "git commit -a")
You may remember that this shared-files directory is itself a custom git-pulled directory containing files that I need to deploy via Rudder.
This repo in repo setup was taken into account in a previous ticket one year ago.
Updated by Nicolas Ecarnot over 3 years ago
Updated by François ARMAND over 3 years ago
So, we successfully reproduced that, and it's a bit puzzling:
There's a commit that fails for distributePolicy because it is empty. But it shouldn't:
## we check that there is something to do - there is !
diff -Naur /opt/rudder/share/techniques/system/distributePolicy/1.0/propagatePromises.cf /var/rudder/configuration-repository/techniques/system/distributePolicy/1.0/propagatePromises.cf
--- /opt/rudder/share/techniques/system/distributePolicy/1.0/propagatePromises.cf 2017-11-22 12:47:27.000000000 +0100
+++ /var/rudder/configuration-repository/techniques/system/distributePolicy/1.0/propagatePromises.cf 2017-11-22 12:47:27.000000000 +0100
@@ -185,7 +185,7 @@
"/usr/bin/rsync --archive --checksum --compress --sparse --delete ${server_info.policy_server}:${server_data}/ ${client_data}"
classes => if_else("promises_propagated", "could_not_propagate_promises");
policy_server.!root_server.rsync_sharedfiles::
- "/usr/bin/rsync --archive --checksum --compress --sparse --delete ${server_info.policy_server}:${system_common.sharedfiles_folder}/ ${system_common.sharedfiles_folder}/"
+ "/usr/bin/rsync --archive --checksum --compress --sparse --delete ${server_info.policy_server}:/${system_common.sharedfiles_folder} ${system_common.sharedfiles_folder}/"
classes => if_else("files_propagated", "could_not_propagate_files");
}
## we synchronize the change !
root@server:/var/rudder/configuration-repository# rsync --delete -rptgoq /opt/rudder/share/techniques/system/distributePolicy/ /var/rudder/configuration-repository/techniques/system/distributePolicy/
## it worked!
root@server:/var/rudder/configuration-repository# echo $?
0
## but no, actually
root@server:/var/rudder/configuration-repository# git status
On branch master
nothing to commit, working tree clean
Updated by François ARMAND over 3 years ago
- Category set to System integration
- Target version set to 6.2.4
To be more precise, the error in is in /opt/rudder/bin/rudder-upgrade (around line #288):
update_rudder_repository_from_system_directory /opt/rudder/share/techniques/system/distributePolicy/ techniques/system/distributePolicy/
Updated by François ARMAND over 3 years ago
- File clipboard-202103181830-x0av3.png clipboard-202103181830-x0av3.png added
- Target version changed from 6.2.4 to 6.1.11
So, the problem is that our rsync compared only newer file, and for some obscure packaging reason, that file is seen as coming from 2017 - but it's the correct file. So the destination is more recent, and rsync doesn't check: we need to add option: --ignore-times:
The bug can happen in previous version of rudder upgrade script too.
Updated by François ARMAND over 3 years ago
- Subject changed from Upgrade from 6.2.2 to 6.2.3 : ERROR: rudder-webapp postinstall script failed ! to Upgrade script fails in system technique update (empty commit)
Updated by François ARMAND over 3 years ago
- Status changed from New to In progress
- Assignee set to François ARMAND
Updated by François ARMAND over 3 years ago
- Status changed from In progress to Pending technical review
- Assignee changed from François ARMAND to Alexis Mousset
- Pull Request set to https://github.com/Normation/rudder-packages/pull/2461
Updated by François ARMAND over 3 years ago
- Status changed from Pending technical review to Pending release
Applied in changeset rudder-packages|f351c31a05a17e6c8bcaf36167049bf69d6feb66.
Updated by François ARMAND over 3 years ago
- Severity set to Minor - inconvenience | misleading | easy workaround
- User visibility set to Operational - other Techniques | Rudder settings | Plugins
- Priority changed from 0 to 32
Now that we have a documented workaround and that we understand that it only prevent the synchronisation of the most recent version of system technique, we can set it's criticity to minor.
Updated by Vincent MEMBRÉ over 3 years ago
- Fix check changed from To do to Checked
Updated by Vincent MEMBRÉ over 3 years ago
- Status changed from Pending release to Released
This bug has been fixed in Rudder 6.1.11 and 6.2.4 which were released today.