Project

General

Profile

Actions

Bug #16797

closed

After upgrading from 5.0.16 to 6.0.3 on centos7 with plugins, jetty is stopped

Added by Nicolas CHARLES about 4 years ago. Updated about 4 years ago.

Status:
Released
Priority:
N/A
Category:
Packaging
Target version:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
Reviewed
Fix check:
To do
Regression:

Description

I have a Rudder 5.0.16 that I upgraded to 6.0.3 on centos7
after the upgrade, Rudder is not working, because rudder-jetty is stopped
As it is an upgrade, policies are not working, so nothing can restore the state of the service

I had to manually restart rudder-jetty

Install logs are

Total                                                                                                                                                                        1.0 MB/s | 132 MB  00:02:10     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : audit-libs-2.8.5-4.el7.x86_64                                                                                                                                                            1/32 
  Updating   : 1398866025:rudder-agent-6.0.4.rc1.git202002250103-1.EL.7.x86_64                                                                                                                          2/32 
INFO: Scheduling an inventory during next run...
INFO: Agent certificate is missing, creating it... Done
INFO: No disable file detected and no agent executor process either. Restarting agent service...rudder-cf-serverd: enabled
rudder-cf-execd: enabled
ok: stop service rudder-agent succeeded
rudder-cf-serverd: enabled
rudder-cf-execd: enabled
ok: start service rudder-agent succeeded
 Done
FINISH: Rudder agent check ran properly, please look at messages above to see if there has been any error.
  Installing : audit-libs-python-2.8.5-4.el7.x86_64                                                                                                                                                     3/32 
  Updating   : policycoreutils-2.5-33.el7.x86_64                                                                                                                                                        4/32 
  Installing : libcgroup-0.41-21.el7.x86_64                                                                                                                                                             5/32 
  Installing : python-chardet-2.2.1-3.el7.noarch                                                                                                                                                        6/32 
  Installing : checkpolicy-2.5-8.el7.x86_64                                                                                                                                                             7/32 
  Installing : libsemanage-python-2.5-14.el7.x86_64                                                                                                                                                     8/32 
  Installing : setools-libs-3.3.8-4.el7.x86_64                                                                                                                                                          9/32 
  Installing : python-IPy-0.75-6.el7.noarch                                                                                                                                                            10/32 
  Installing : policycoreutils-python-2.5-33.el7.x86_64                                                                                                                                                11/32 
  Installing : python-urllib3-1.10.2-7.el7.noarch                                                                                                                                                      12/32 
  Installing : python-requests-2.6.0-8.el7_7.noarch                                                                                                                                                    13/32 
  Updating   : 1398866025:rudder-server-relay-6.0.4.rc1.git202002250103-1.EL.7.x86_64                                                                                                                  14/32 
INFO: Setting Apache httpd as a boot service... Done
INFO: Creating users ... Done
INFO: Configuring systemd services... Done
INFO: Starting services... Done
INFO: rudder-server-relay setup complete.
  Installing : 1398866025:rudder-webapp-6.0.4.rc1.git202002250103-1.EL.7.x86_64                                                                                                                        15/32 
warning: /opt/rudder/etc/openldap/slapd.conf created as /opt/rudder/etc/openldap/slapd.conf.rpmnew
warning: /opt/rudder/etc/rudder-web.properties created as /opt/rudder/etc/rudder-web.properties.rpmnew
warning: user rudder-slapd does not exist - using root
warning: group rudder-slapd does not exist - using root
INFO: Creating groups ...INFO: Creating users ... Done
INFO: Setting up systemd ...Created symlink from /etc/systemd/system/multi-user.target.wants/rudder-jetty.service to /usr/lib/systemd/system/rudder-jetty.service.
Created symlink from /etc/systemd/system/rudder-server.service.requires/rudder-jetty.service to /usr/lib/systemd/system/rudder-jetty.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/rudder-slapd.service to /usr/lib/systemd/system/rudder-slapd.service.
Created symlink from /etc/systemd/system/rudder-server.service.requires/rudder-slapd.service to /usr/lib/systemd/system/rudder-slapd.service.
INFO: Restarting rudder-slapd... Done
INFO: Launching script to check if a migration is needed ...
Done
********************************************************************************
Due to incompatibility reasons for this specific version all plugins have been
disabled. Make sure to update your repo index and install the latest version of
your plugins before re-enabling them.

You can use the following commands to upgrade your plugins:
rudder package update
rudder package upgrade-all
********************************************************************************
Successfully triggered a new policy generation
INFO: Make sure Rudder webapp is started... Done
INFO: Make sure Apache HTTPd is started... Done
INFO: Applying selinux policy... Done
  Updating   : 1398866025:rudder-reports-6.0.4.rc1.git202002250103-1.EL.7.noarch                                                                                                                       16/32 
INFO: Setting PostgreSQL as a boot service... Done
INFO: Waiting for PostgreSQL to be up... Done
  Updating   : 1398866025:rudder-server-root-6.0.4.rc1.git202002250103-1.EL.7.noarch                                                                                                                   17/32 
Created symlink from /etc/systemd/system/multi-user.target.wants/rudder-server.service to /usr/lib/systemd/system/rudder-server.service.
  Updating   : audit-2.8.5-4.el7.x86_64                                                                                                                                                                18/32 
  Cleanup    : 1398866025:rudder-server-root-5.0.16.release-1.EL.7.noarch                                                                                                                              19/32 
  Cleanup    : 1398866025:rudder-webapp-5.0.16.release-1.EL.7.noarch                                                                                                                                   20/32 
warning: file /var/rudder/configuration-repository/ncf/ncf-hooks.d/post.write_technique.50_rudderify.sh: remove failed: No such file or directory
warning: file /var/rudder/configuration-repository/ncf/ncf-hooks.d/post.write_technique.10_commit.sh: remove failed: No such file or directory
warning: file /var/rudder/configuration-repository/ncf/ncf-hooks.d: remove failed: No such file or directory
  Erasing    : 1398866025:rudder-inventory-endpoint-5.0.16.release-1.EL.7.noarch                                                                                                                       21/32 
  Erasing    : 1398866025:ncf-api-virtualenv-5.0.16.release-1.EL.7.noarch                                                                                                                              22/32 
INFO: Removing the ncf-api-venv user... Done
INFO: Removing ncf-api-virtualenv selinux policy...libsemanage.semanage_direct_remove_key: Removing last ncf-api-virtualenv module (no other ncf-api-virtualenv module exists at another priority).
 Done
  Cleanup    : 1398866025:rudder-server-relay-5.0.16.release-1.EL.7.x86_64                                                                                                                             23/32 
  Erasing    : 1398866025:rudder-techniques-5.0.16.release-1.EL.7.noarch                                                                                                                               24/32 
  Cleanup    : policycoreutils-2.5-29.el7_6.1.x86_64                                                                                                                                                   25/32 
  Cleanup    : audit-2.8.4-4.el7.x86_64                                                                                                                                                                26/32 
  Erasing    : 1398866025:ncf-5.0.16.release-1.EL.7.noarch                                                                                                                                             27/32 
warning: file /usr/share/ncf/tools/ncf_constraints.pyc: remove failed: No such file or directory
INFO: A back up copy of the /opt/rudder/etc/uuid.hive has been created in /var/backups/rudder
INFO: A back up copy of the /var/rudder/cfengine-community/policy_server.dat has been created in /var/backups/rudder
INFO: A back up copy of the /var/rudder/cfengine-community/ppkeys has been created in /var/backups/rudder
  Cleanup    : 1398866025:rudder-agent-5.0.16.release-1.EL.7.x86_64                                                                                                                                    28/32 
  Erasing    : 1398866025:rudder-inventory-ldap-5.0.16.release-1.EL.7.x86_64                                                                                                                           29/32 
Stopping Jetty: ERROR: no pid found at /var/rudder/run/rudder-jetty.pid
  Erasing    : 1398866025:rudder-jetty-5.0.16.release-1.EL.7.noarch                                                                                                                                    30/32 
  Cleanup    : 1398866025:rudder-reports-5.0.16.release-1.EL.7.noarch                                                                                                                                  31/32 
  Cleanup    : audit-libs-2.8.4-4.el7.x86_64                                                                                                                                                           32/32 
  Verifying  : 1398866025:rudder-reports-6.0.4.rc1.git202002250103-1.EL.7.noarch                                                                                                                        1/32 
  Verifying  : audit-libs-python-2.8.5-4.el7.x86_64                                                                                                                                                     2/32 
  Verifying  : python-urllib3-1.10.2-7.el7.noarch                                                                                                                                                       3/32 
  Verifying  : audit-2.8.5-4.el7.x86_64                                                                                                                                                                 4/32 
  Verifying  : 1398866025:rudder-agent-6.0.4.rc1.git202002250103-1.EL.7.x86_64                                                                                                                          5/32 
  Verifying  : python-IPy-0.75-6.el7.noarch                                                                                                                                                             6/32 
  Verifying  : setools-libs-3.3.8-4.el7.x86_64                                                                                                                                                          7/32 
  Verifying  : libsemanage-python-2.5-14.el7.x86_64                                                                                                                                                     8/32 
  Verifying  : policycoreutils-python-2.5-33.el7.x86_64                                                                                                                                                 9/32 
  Verifying  : 1398866025:rudder-webapp-6.0.4.rc1.git202002250103-1.EL.7.x86_64                                                                                                                        10/32 
  Verifying  : 1398866025:rudder-server-root-6.0.4.rc1.git202002250103-1.EL.7.noarch                                                                                                                   11/32 
  Verifying  : audit-libs-2.8.5-4.el7.x86_64                                                                                                                                                           12/32 
  Verifying  : checkpolicy-2.5-8.el7.x86_64                                                                                                                                                            13/32 
  Verifying  : 1398866025:rudder-server-relay-6.0.4.rc1.git202002250103-1.EL.7.x86_64                                                                                                                  14/32 
  Verifying  : policycoreutils-2.5-33.el7.x86_64                                                                                                                                                       15/32 
  Verifying  : python-chardet-2.2.1-3.el7.noarch                                                                                                                                                       16/32 
  Verifying  : python-requests-2.6.0-8.el7_7.noarch                                                                                                                                                    17/32 
  Verifying  : libcgroup-0.41-21.el7.x86_64                                                                                                                                                            18/32 
  Verifying  : 1398866025:rudder-inventory-endpoint-5.0.16.release-1.EL.7.noarch                                                                                                                       19/32 
  Verifying  : policycoreutils-2.5-29.el7_6.1.x86_64                                                                                                                                                   20/32 
  Verifying  : 1398866025:ncf-5.0.16.release-1.EL.7.noarch                                                                                                                                             21/32 
  Verifying  : 1398866025:rudder-server-relay-5.0.16.release-1.EL.7.x86_64                                                                                                                             22/32 
  Verifying  : 1398866025:rudder-agent-5.0.16.release-1.EL.7.x86_64                                                                                                                                    23/32 
  Verifying  : audit-libs-2.8.4-4.el7.x86_64                                                                                                                                                           24/32 
  Verifying  : audit-2.8.4-4.el7.x86_64                                                                                                                                                                25/32 
  Verifying  : 1398866025:rudder-webapp-5.0.16.release-1.EL.7.noarch                                                                                                                                   26/32 
  Verifying  : 1398866025:rudder-inventory-ldap-5.0.16.release-1.EL.7.x86_64                                                                                                                           27/32 
  Verifying  : 1398866025:ncf-api-virtualenv-5.0.16.release-1.EL.7.noarch                                                                                                                              28/32 
  Verifying  : 1398866025:rudder-server-root-5.0.16.release-1.EL.7.noarch                                                                                                                              29/32 
  Verifying  : 1398866025:rudder-jetty-5.0.16.release-1.EL.7.noarch                                                                                                                                    30/32 
  Verifying  : 1398866025:rudder-reports-5.0.16.release-1.EL.7.noarch                                                                                                                                  31/32 
  Verifying  : 1398866025:rudder-techniques-5.0.16.release-1.EL.7.noarch                                                                                                                               32/32 

Installed:
  rudder-webapp.x86_64 1398866025:6.0.4.rc1.git202002250103-1.EL.7                                                                                                                                           

Dependency Installed:
  audit-libs-python.x86_64 0:2.8.5-4.el7  checkpolicy.x86_64 0:2.5-8.el7       libcgroup.x86_64 0:0.41-21.el7          libsemanage-python.x86_64 0:2.5-14.el7  policycoreutils-python.x86_64 0:2.5-33.el7 
  python-IPy.noarch 0:0.75-6.el7          python-chardet.noarch 0:2.2.1-3.el7  python-requests.noarch 0:2.6.0-8.el7_7  python-urllib3.noarch 0:1.10.2-7.el7    setools-libs.x86_64 0:3.3.8-4.el7          

Updated:
  rudder-agent.x86_64 1398866025:6.0.4.rc1.git202002250103-1.EL.7                                       rudder-reports.noarch 1398866025:6.0.4.rc1.git202002250103-1.EL.7                                   
  rudder-server-relay.x86_64 1398866025:6.0.4.rc1.git202002250103-1.EL.7                                rudder-server-root.noarch 1398866025:6.0.4.rc1.git202002250103-1.EL.7                               

Dependency Updated:
  audit.x86_64 0:2.8.5-4.el7                                     audit-libs.x86_64 0:2.8.5-4.el7                                     policycoreutils.x86_64 0:2.5-33.el7                                    

Replaced:
  ncf.noarch 1398866025:5.0.16.release-1.EL.7                        ncf-api-virtualenv.noarch 1398866025:5.0.16.release-1.EL.7      rudder-inventory-endpoint.noarch 1398866025:5.0.16.release-1.EL.7     
  rudder-inventory-ldap.x86_64 1398866025:5.0.16.release-1.EL.7      rudder-jetty.noarch 1398866025:5.0.16.release-1.EL.7            rudder-techniques.noarch 1398866025:5.0.16.release-1.EL.7             

Complete!

journald tells me

Feb 25 10:02:48 server systemd[1]: Started Rudder OpenLDAP Server Daemon.
Feb 25 10:03:16 server systemd[1]: Starting Jetty Web Application Server...
Feb 25 10:03:17 server rudder-jetty.sh[31432]: Setting umask to 0007
Feb 25 10:03:17 server rudder-jetty.sh[31432]: Starting Jetty: already running.
Feb 25 10:03:18 server rudder-jetty.sh[31504]: Stopping Jetty: OK
Feb 25 10:03:18 server systemd[1]: Started Jetty Web Application Server.
Feb 25 10:03:18 server systemd[1]: Stopping The Apache HTTP Server...
Feb 25 10:03:19 server systemd[1]: Stopped The Apache HTTP Server.
Feb 25 10:03:19 server systemd[1]: Starting The Apache HTTP Server...
Feb 25 10:03:20 server systemd[1]: Started The Apache HTTP Server.

install og file says

Tue Feb 25 10:02:48 UTC 2020 - Starting rudder-webapp post installation script
INFO: Checking if rudder-web.properties database access credentials are all right... LDAP OK,  SQL OK
INFO: A Technique library reload is needed and has been scheduled.
INFO: A Technique library reload is needed and has been scheduled.
INFO: A Technique library reload is needed and has been scheduled.
INFO: A Technique library reload is needed and has been scheduled.
Overriding existing techniques
INFO: Correcting permissions on /var/rudder/configuration-repository... Done
Techniques have been updated, and update branch set to current state of the Techniques
rm 'ncf/ncf.conf'
[master de3d436] Remove ncf.conf from configuration repository
 1 file changed, 24 deletions(-)
 delete mode 100644 ncf/ncf.conf
rm 'ncf/ncf-hooks.d/post.write_technique.10_commit.sh'
rm 'ncf/ncf-hooks.d/post.write_technique.50_rudderify.sh'
[master 5298788] Remove ncf-hooks.d from configuration repository
 2 files changed, 125 deletions(-)
 delete mode 100755 ncf/ncf-hooks.d/post.write_technique.10_commit.sh
 delete mode 100755 ncf/ncf-hooks.d/post.write_technique.50_rudderify.sh
INFO: New configuration property inventories.watcher.enable added to /opt/rudder/etc/rudder-web.properties
INFO: Checking PostgreSQL service status... OK
INFO: Checking LDAP service status... OK
adding new entry "propertyName=rudder_report_protocol_default,ou=Application Properties,cn=rudder-configuration" 

Disabling rudder-plugin-scale-out-relay
[]
Restarting jetty
 $ service rudder-jetty restart
Stopping Jetty: ERROR: no pid found at /var/rudder/run/rudder-jetty.pid
Setting umask to 0007
Starting Jetty: . . . . OK Tue Feb 25 10:03:10 UTC 2020
done
Successfully triggered a new policy generation

INFO: The migration has completed successfully.
INFO: Correcting permissions on /var/rudder/configuration-repository... Done
Tue Feb 25 10:03:20 UTC 2020 - Ending rudder-webapp post installation script

Actions #1

Updated by Nicolas CHARLES about 4 years ago

  • Status changed from New to In progress
  • Assignee set to Nicolas CHARLES
Actions #2

Updated by Nicolas CHARLES about 4 years ago

  • Assignee changed from Nicolas CHARLES to Benoît PECCATTE
  • Pull Request set to https://github.com/Normation/rudder-packages/pull/2211
Actions #3

Updated by Nicolas CHARLES about 4 years ago

  • Status changed from In progress to Pending release
Actions #5

Updated by Vincent MEMBRÉ about 4 years ago

  • Target version changed from 6.0.4 to 6.0.3
Actions #6

Updated by Alexis Mousset about 4 years ago

  • Name check changed from To do to Reviewed
Actions #7

Updated by Vincent MEMBRÉ about 4 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 6.0.3 which was released today.

Actions

Also available in: Atom PDF