Rudder

I had a 8.0.4 with Rudder 8.0.4, system-updates & cve plugins
I tried upgraded Rudder to only find it not starting
Install procedure mostly worked

t# apt install rudder-server
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  rudder-agent rudder-api-client rudder-relay
The following packages will be upgraded:
  rudder-agent rudder-api-client rudder-relay rudder-server
4 upgraded, 0 newly installed, 0 to remove and 67 not upgraded.
Need to get 209 MB of archives.
After this operation, 22.3 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://download.rudder.io/rtf/apt/8.1-nightly jammy/main amd64 rudder-server amd64 8.1.0~beta2~git202403210219-ubuntu22.04 [200 MB]
Get:2 https://download.rudder.io/rtf/apt/8.1-nightly jammy/main amd64 rudder-api-client amd64 8.1.0~beta2~git202403210219-ubuntu22.04 [9728 B]
Get:3 https://download.rudder.io/rtf/apt/8.1-nightly jammy/main amd64 rudder-relay amd64 8.1.0~beta2~git202403210219-ubuntu22.04 [5890 kB]
Get:4 https://download.rudder.io/rtf/apt/8.1-nightly jammy/main amd64 rudder-agent amd64 8.1.0~beta2~git202403210219-ubuntu22.04 [3216 kB]
Fetched 209 MB in 3s (64.2 MB/s)   
Preconfiguring packages ...
(Reading database ... 104946 files and directories currently installed.)
Preparing to unpack .../rudder-server_8.1.0~beta2~git202403210219-ubuntu22.04_amd64.deb ...
********************************************************************************
INFO: dpkg may ask you if you want to replace rudder-web.properties or          
INFO: slapd.conf with the maintainer version.                                   
INFO:                                                                           
INFO: If you accept, please keep in mind that you will lose any manual          
INFO: modifications in this file, like batchs configuration, or LDAP            
INFO: authentication. It will also reset passwords to default values and break  
INFO: upgrade scripts.                                                          
INFO:                                                                           
INFO: We strongly advise you to respond NO to this question                     
INFO: ('keep your currently-installed version'). Rudder's upgrade script will   
INFO: take care of upgrading these file automatically.                          
INFO:                                                                           
********************************************************************************
Unpacking rudder-server (8.1.0~beta2~git202403210219-ubuntu22.04) over (8.0.4-ubuntu22.04) ...
Preparing to unpack .../rudder-api-client_8.1.0~beta2~git202403210219-ubuntu22.04_amd64.deb ...
Unpacking rudder-api-client (8.1.0~beta2~git202403210219-ubuntu22.04) over (8.0.4-ubuntu22.04) ...
Preparing to unpack .../rudder-relay_8.1.0~beta2~git202403210219-ubuntu22.04_amd64.deb ...
Unpacking rudder-relay (8.1.0~beta2~git202403210219-ubuntu22.04) over (8.0.4-ubuntu22.04) ...
Preparing to unpack .../rudder-agent_8.1.0~beta2~git202403210219-ubuntu22.04_amd64.deb ...
Unpacking rudder-agent (8.1.0~beta2~git202403210219-ubuntu22.04) over (8.0.4-ubuntu22.04) ...
Setting up rudder-agent (8.1.0~beta2~git202403210219-ubuntu22.04) ...
Setting up rudder-api-client (8.1.0~beta2~git202403210219-ubuntu22.04) ...
Setting up rudder-relay (8.1.0~beta2~git202403210219-ubuntu22.04) ...
Job for apache2.service canceled.
Setting up rudder-server (8.1.0~beta2~git202403210219-ubuntu22.04) ...

Configuration file '/opt/rudder/etc/rudder-web.properties'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** rudder-web.properties (Y/I/N/O/D/Z) [default=N] ? 

Configuration file '/opt/rudder/etc/rudder-users.xml'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** rudder-users.xml (Y/I/N/O/D/Z) [default=N] ? 
Already on 'master'
 INFO rudder_package: Index and licenses successfully updated
 INFO rudder_package::database: Installing rudder-plugin-cve
 INFO rudder_package::database: Plugin cve already installed, upgrading
 INFO rudder_package::archive: Installing rpkg '/var/rudder/tmp/plugins/rudder-plugin-cve-8.1.0~beta2-2.10-nightly.rpkg'...
 INFO rudder_package::archive: Plugin cve was successfully installed
 INFO rudder_package::database: Installing rudder-plugin-system-updates
 INFO rudder_package::database: Plugin system-updates already installed, upgrading
 INFO rudder_package::archive: Installing rpkg '/var/rudder/tmp/plugins/rudder-plugin-system-updates-8.1.0~beta2-1.15-nightly.rpkg'...
 INFO rudder_package::archive: Plugin system-updates was successfully installed
 INFO rudder_package: All plugins were upgraded successfully
🗸 Restarting the Web application to apply changes
ERROR rudder_package: No such file or directory (os error 2)
 INFO rudder_package: Plugins successfully disabled
Job for rudder-jetty.service failed because the control process exited with error code.
See "systemctl status rudder-jetty.service" and "journalctl -xeu rudder-jetty.service" for details.
 INFO rudder_package: All postinstall scripts ran successfully
Processing triggers for libc-bin (2.35-0ubuntu3.6) ...
Scanning processes...                                                                                                                      
Scanning linux images...                                                                                                                   

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

 INFO rudder_package: All plugins were upgraded successfully
🗸 Restarting the Web application to apply changes
ERROR rudder_package: No such file or directory (os error 2)
 INFO rudder_package: Plugins successfully disabled
Job for rudder-jetty.service failed because the control process exited with error code.
See "systemctl status rudder-jetty.service" and "journalctl -xeu rudder-jetty.service" for details.

no logs mention the No such file or directory

journalctl nor install log don't mention any of that, but rudder-jetty doesn't start
The logs of webapp are empty (0 bytes), and it fails quite quickly

Mar 21 11:45:08 server rudder-jetty.sh[15881]: Stopping Jetty: OK
Mar 21 11:45:12 server systemd[1]: rudder-jetty.service: Main process exited, code=exited, status=129/n/a
Mar 21 11:45:12 server systemd[1]: rudder-jetty.service: Failed with result 'exit-code'.

plugins are upgraded

# rudder package list
+----------------+--------------------------+------------+--------------+------------------+-----------------------------------------------------+
| Name           | Installed                | Latest     | Web plugin   | License          | Description                                         |
+----------------+--------------------------+------------+--------------+------------------+-----------------------------------------------------+
| cve            | 8.1.0~beta2-2.10-nightly | up-to-date | yes: enabled | until 2024/04/20 | Manage known vulnerabilities in system components   |
| system-updates | 8.1.0~beta2-1.15-nightly | up-to-date | yes: enabled | until 2024/04/20 | Available upgrades management and upgrade campaigns |
+----------------+--------------------------+------------+--------------+------------------+-----------------------------------------------------+

disabling them don't fix the issue

Finally, I got it to work when running

/opt/rudder/bin/rudder-jetty-pre-check.sh
/opt/rudder/bin/rudder-jetty.sh start