Bug #27367
open
Log the list of restarted services
Description
Confirmation of a customer-reported issue (customer ticket, reproduced)
On Rudder 8.3.3, system-updates 8.3.3-1.17-nightly (or not nightly), Debian 12,
It seems that system updates do not restart services when it should.
Given an update campaign configured with : “Reboot or restart services only when needed”
Campaign stderr shows :
Use of uninitialized value $ucode_vars{"CURRENT"} in concatenation (.) or string at /usr/sbin/needrestart line 951.
Use of uninitialized value $ucode_vars{"AVAIL"} in concatenation (.) or string at /usr/sbin/needrestart line 952.
Campaign stdout ends with :
The hook directory /var/rudder/system-update/hooks.d/pre-reboot does not exist, skipping NEEDRESTART-VER: 3.6 NEEDRESTART-KCUR: 6.12.34+rpt-rpi-2712 NEEDRESTART-KSTA: 0 NEEDRESTART-UCSTA: 1 NEEDRESTART-UCCUR: NEEDRESTART-UCEXP: NEEDRESTART-SVC: wsdd.service The hook directory /var/rudder/system-update/hooks.d/post-upgrade does not exist, skipping
Identifying that a service (wsdd.service) needs to be restarted.
But the following day, running manually on the node :
❯ needrestart -rl Scanning processes... Scanning candidates... Scanning processor microcode... Scanning linux images... Failed to retrieve available kernel versions. The processor microcode seems to be up-to-date. Services to be restarted: systemctl restart wsdd.service No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host.
Shows that the same service still needs to be restarted, which probably means that it hasn't been when it should have.
This is not ARM or RaspberryPi specific, as the customer reported this on x86_64 hardware.
Updated by Alexis Mousset about 1 month ago
- Status changed from New to In progress
- Assignee set to Alexis Mousset
Updated by Alexis Mousset about 1 month ago
- Project changed from 92 to Rudder
- Category set to Module - system-updates
- Status changed from In progress to New
- Target version deleted (
8.3)
Updated by Michel BOUISSOU about 1 month ago
I actually have a doubt.
I have performed one-shot full system upgrades with “Reboot or restart services only when needed” on 3 debian servers.
- On 2 servers the campaign stdout NEEDRESTART output shows openmediavault-engined.service needs to be restarted
- On the same 2 servers, the campaign stdout NEEDRESTART output shows wsdd.service
- On one server, the campaign stdout NEEDRESTART output shows apache2.service needs to be restarted
After the campaing has ended, checking on all 3 servers with “needrestart rl” shows that : openmediavault-engined.service and apache2.service do not appear anymore, so they seem to have been properly restarted by the campaign.
- wsdd.service still shows as needing a restart on 2 servers.
But on these servers, “needrestart -rl” always shows that wsdd.service needs to be restarted, event after a manual "systemctl restart wsdd.service”
So it seems that the problem resides either with this specific service or the way needrestart evaluates it, and not with Rudder.
From these tests it would appear that Rudder actually properly restarts services that need to be on Debian 12, and the one that appears to stil be needing a restart has an issue which is not Rudder related.
❯ needrestart -rl Scanning processes... Scanning candidates... Scanning processor microcode... Scanning linux images... Failed to retrieve available kernel versions. The processor microcode seems to be up-to-date. Services to be restarted: systemctl restart wsdd.service No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. root in 🌐 naspi in ~ as 🧙 ❯ systemctl restart wsdd.service root in 🌐 naspi in ~ as 🧙 ❯ needrestart -rl Scanning processes... Scanning candidates... Scanning processor microcode... Scanning linux images... Failed to retrieve available kernel versions. The processor microcode seems to be up-to-date. Services to be restarted: systemctl restart wsdd.service No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. root in 🌐 naspi in ~ as 🧙 ❯ systemctl stop wsdd.service root in 🌐 naspi in ~ as 🧙 ❯ needrestart -rl Scanning processes... Scanning processor microcode... Scanning linux images... Failed to retrieve available kernel versions. The processor microcode seems to be up-to-date. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. root in 🌐 naspi in ~ as 🧙 ❯ systemctl start wsdd.service root in 🌐 naspi in ~ as 🧙 ❯ needrestart -rl Scanning processes... Scanning candidates... Scanning processor microcode... Scanning linux images... Failed to retrieve available kernel versions. The processor microcode seems to be up-to-date. Services to be restarted: systemctl restart wsdd.service No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host.
Updated by Alexis Mousset about 1 month ago
- Status changed from New to Pending technical review
- Assignee changed from Alexis Mousset to Elaad FURREEDAN
- Pull Request set to https://github.com/Normation/rudder/pull/6544
Updated by Alexis Mousset about 1 month ago
- Subject changed from Services are not restarted when they should be to Log the list of restarted services
Ok, adding a log to list the restarted services, this will help track a similar problem if it occurs again.
Updated by Alexis Mousset about 1 month ago
- Status changed from Pending technical review to Pending release
Applied in changeset rudder|91d855aaaf2a6b2120f147afed037bd312a1d1a6.
Updated by