Project

General

Profile

Actions

Bug #27367

open

Log the list of restarted services

Added by Michel BOUISSOU 12 days ago. Updated 2 days ago.

Status:
Pending release
Priority:
2
Category:
Module - system-updates
Target version:
Severity:
Major - prevents use of part of Rudder | no simple workaround
UX impact:
User visibility:
Effort required:
Priority:
0
Name check:
To do
Fix check:
To do
Regression:
No

Description

Confirmation of a customer-reported issue (customer ticket, reproduced)

On Rudder 8.3.3, system-updates 8.3.3-1.17-nightly (or not nightly), Debian 12,

It seems that system updates do not restart services when it should.

Given an update campaign configured with : “Reboot or restart services only when needed”

Campaign stderr shows :

Use of uninitialized value $ucode_vars{"CURRENT"} in concatenation (.) or string at /usr/sbin/needrestart line 951.
Use of uninitialized value $ucode_vars{"AVAIL"} in concatenation (.) or string at /usr/sbin/needrestart line 952.

Campaign stdout ends with :

The hook directory /var/rudder/system-update/hooks.d/pre-reboot does not exist, skipping
NEEDRESTART-VER: 3.6
NEEDRESTART-KCUR: 6.12.34+rpt-rpi-2712
NEEDRESTART-KSTA: 0
NEEDRESTART-UCSTA: 1
NEEDRESTART-UCCUR: 
NEEDRESTART-UCEXP: 
NEEDRESTART-SVC: wsdd.service

The hook directory /var/rudder/system-update/hooks.d/post-upgrade does not exist, skipping

Identifying that a service (wsdd.service) needs to be restarted.

But the following day, running manually on the node :

❯ needrestart -rl
Scanning processes...                                                                                                                                                                               
Scanning candidates...                                                                                                                                                                              
Scanning processor microcode...                                                                                                                                                                     
Scanning linux images...                                                                                                                                                                            

Failed to retrieve available kernel versions.

The processor microcode seems to be up-to-date.

Services to be restarted:
 systemctl restart wsdd.service

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

Shows that the same service still needs to be restarted, which probably means that it hasn't been when it should have.

This is not ARM or RaspberryPi specific, as the customer reported this on x86_64 hardware.

Actions #3

Updated by Alexis Mousset 12 days ago

  • Status changed from New to In progress
  • Assignee set to Alexis Mousset
Actions #4

Updated by Alexis Mousset 11 days ago

  • Project changed from 92 to Rudder
  • Category set to Module - system-updates
  • Status changed from In progress to New
  • Target version deleted (8.3)
Actions #5

Updated by Alexis Mousset 11 days ago

  • Target version set to 8.3.4
Actions #6

Updated by Michel BOUISSOU 11 days ago

I actually have a doubt.

I have performed one-shot full system upgrades with “Reboot or restart services only when needed” on 3 debian servers.

- On 2 servers the campaign stdout NEEDRESTART output shows openmediavault-engined.service needs to be restarted
- On the same 2 servers, the campaign stdout NEEDRESTART output shows wsdd.service
- On one server, the campaign stdout NEEDRESTART output shows apache2.service needs to be restarted

After the campaing has ended, checking on all 3 servers with “needrestart rl” shows that :
openmediavault-engined.service and apache2.service do not appear anymore, so they seem to have been properly restarted by the campaign.
- wsdd.service still shows as needing a restart on 2 servers.

But on these servers, “needrestart -rl” always shows that wsdd.service needs to be restarted, event after a manual "systemctl restart wsdd.service”

So it seems that the problem resides either with this specific service or the way needrestart evaluates it, and not with Rudder.

From these tests it would appear that Rudder actually properly restarts services that need to be on Debian 12, and the one that appears to stil be needing a restart has an issue which is not Rudder related.

❯ needrestart -rl
Scanning processes...                                                                                                                                                                                                                       
Scanning candidates...                                                                                                                                                                                                                      
Scanning processor microcode...                                                                                                                                                                                                             
Scanning linux images...                                                                                                                                                                                                                    

Failed to retrieve available kernel versions.

The processor microcode seems to be up-to-date.

Services to be restarted:
 systemctl restart wsdd.service

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

root in 🌐 naspi in ~ as 🧙 
❯ systemctl restart wsdd.service

root in 🌐 naspi in ~ as 🧙 
❯ needrestart -rl
Scanning processes...                                                                                                                                                                                                                       
Scanning candidates...                                                                                                                                                                                                                      
Scanning processor microcode...                                                                                                                                                                                                             
Scanning linux images...                                                                                                                                                                                                                    

Failed to retrieve available kernel versions.

The processor microcode seems to be up-to-date.

Services to be restarted:
 systemctl restart wsdd.service

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

root in 🌐 naspi in ~ as 🧙 
❯ systemctl stop wsdd.service

root in 🌐 naspi in ~ as 🧙 
❯ needrestart -rl
Scanning processes...                                                                                                                                                                                                                       
Scanning processor microcode...                                                                                                                                                                                                             
Scanning linux images...                                                                                                                                                                                                                    

Failed to retrieve available kernel versions.

The processor microcode seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

root in 🌐 naspi in ~ as 🧙 
❯ systemctl start wsdd.service

root in 🌐 naspi in ~ as 🧙 
❯ needrestart -rl
Scanning processes...                                                                                                                                                                                                                       
Scanning candidates...                                                                                                                                                                                                                      
Scanning processor microcode...                                                                                                                                                                                                             
Scanning linux images...                                                                                                                                                                                                                    

Failed to retrieve available kernel versions.

The processor microcode seems to be up-to-date.

Services to be restarted:
 systemctl restart wsdd.service

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
Actions #7

Updated by Alexis Mousset 11 days ago

  • Status changed from New to Pending technical review
  • Assignee changed from Alexis Mousset to Elaad FURREEDAN
  • Pull Request set to https://github.com/Normation/rudder/pull/6544
Actions #8

Updated by Alexis Mousset 11 days ago

  • Subject changed from Services are not restarted when they should be to Log the list of restarted services

Ok, adding a log to list the restarted services, this will help track a similar problem if it occurs again.

Actions #9

Updated by Alexis Mousset 11 days ago

  • Status changed from Pending technical review to Pending release
Actions #10

Updated by François ARMAND 2 days ago

  • Priority changed from To review to 2
Actions

Also available in: Atom PDF