Rudder

Server and node Rudder 8.3.2, Debian 12 x86_64 node.

I noticed that the “system updates campaigns“ were not running on a node, showing that apt was locked by another process.

Checking PS I foudn out that an “apt-get update” started by

python3 /var/rudder/cfengine-community/modules/packages/apt-get list-updates

had been running from ~60 hours and was obviously stuck without ever being terminated.

This doesn't seem to come from the system updates campaigns but rather from a “package” method started by the Rudder agent.

Agent execution output in /var/rudder/cfengine-community/outputs, corresponding to the agent run that started the stuck process, is truncated and ends in :

2025-06-16T18:05:13+00:00 R: @@upgrade_rudder_agent_on_debian_family@@log_info@@4a0d9768-070d-4e5c-8153-d119da3706e8@@8a0e373b-d8dc-41f2-a91a-34e52df832c7@@5de65f37-e442-4201-bc7d-567125891616@@10-050 - Rudder repository set to download.rudder.io for debian-family OS with Rudder 8.3@@/etc/apt/sources.list.d/rudder.list@@2025-06-16 18:05:08+00:00##872d703d-2fd2-4641-b3f8-0f98d0f7226b@#Build file /etc/apt/sources.list.d/rudder.list from mustache type template /var/rudder/cfengine-community/inputs/upgrade_rudder_agent_on_debian_family/1.0/resources/rudder.list.tpl was correct
2025-06-16T18:05:13+00:00 R: @@upgrade_rudder_agent_on_debian_family@@result_success@@4a0d9768-070d-4e5c-8153-d119da3706e8@@8a0e373b-d8dc-41f2-a91a-34e52df832c7@@5de65f37-e442-4201-bc7d-567125891616@@10-050 - Rudder repository set to download.rudder.io for debian-family OS with Rudder 8.3@@/etc/apt/sources.list.d/rudder.list@@2025-06-16 18:05:08+00:00##872d703d-2fd2-4641-b3f8-0f98d0f7226b@#Build file /etc/apt/sources.list.d/rudder.list from mustache template /var/rudder/cfengine-community/inputs/upgrade_rudder_agent_on_debian_family/1.0/resources/rudder.list.tpl was correct
2025-06-16T18:05:13+00:00 R: @@upgrade_rudder_agent_on_debian_family@@log_info@@4a0d9768-070d-4e5c-8153-d119da3706e8@@8a0e373b-d8dc-41f2-a91a-34e52df832c7@@10f780e3-d11d-41a4-bb69-912d3d707935@@10-060 - Rudder repository file permissions@@/etc/apt/sources.list.d/rudder.list@@2025-06-16 18:05:08+00:00##872d703d-2fd2-4641-b3f8-0f98d0f7226b@#Ensure permissions mode 644, owner root and group root on /etc/apt/sources.list.d/rudder.list on type all with 0 recursion level was correct
2025-06-16T18:05:13+00:00 R: @@upgrade_rudder_agent_on_debian_family@@result_success@@4a0d9768-070d-4e5c-8153-d119da3706e8@@8a0e373b-d8dc-41f2-a91a-34e52df832c7@@10f780e3-d11d-41a4-bb69-912d3d707935@@10-060 - Rudder repository file permissions@@/etc/apt/sources.list.d/rudder.list@@2025-06-16 18:05:08+00:00##872d703d-2fd2-4641-b3f8-0f98d0f7226b@#Ensure permissions mode 644, owner root and group root on /etc/apt/sources.list.d/rudder.list was correct
2025-06-16T18:05:13+00:00 R: @@upgrade_rudder_agent_on_debian_family@@result_na@@4a0d9768-070d-4e5c-8153-d119da3706e8@@8a0e373b-d8dc-41f2-a91a-34e52df832c7@@f1673845-2d12-4974-ab45-98dd3b6f0a2d@@10-070-10 - Rudder repository credentials file absent for public repository@@/etc/apt/auth.conf.d/rudder.conf@@2025-06-16 18:05:08+00:00##872d703d-2fd2-4641-b3f8-0f98d0f7226b@#Skipping method 'File absent' with key parameter '/etc/apt/auth.conf.d/rudder.conf' since condition 'debian.(!debian_rudder_password_provided_true)' is not reached was not applicable
2025-06-16T18:05:13+00:00 R: @@upgrade_rudder_agent_on_debian_family@@result_success@@4a0d9768-070d-4e5c-8153-d119da3706e8@@8a0e373b-d8dc-41f2-a91a-34e52df832c7@@ea37b528-0a4e-4269-ade3-1ab78cd66cb3@@10-070-20-10 - Rudder repository credentials file present@@/etc/apt/auth.conf.d/rudder.conf@@2025-06-16 18:05:08+00:00##872d703d-2fd2-4641-b3f8-0f98d0f7226b@#Presence of file /etc/apt/auth.conf.d/rudder.conf was correct
2025-06-16T18:05:13+00:00 R: @@upgrade_rudder_agent_on_debian_family@@result_success@@4a0d9768-070d-4e5c-8153-d119da3706e8@@8a0e373b-d8dc-41f2-a91a-34e52df832c7@@cee42f3b-7e17-4478-9540-07d5bd8c5d36@@10-070-20-20 - Rudder repository credentials file content@@/etc/apt/auth.conf.d/rudder.conf@@2025-06-16 18:05:08+00:00##872d703d-2fd2-4641-b3f8-0f98d0f7226b@#Insert content into /etc/apt/auth.conf.d/rudder.conf was correct
2025-06-16T18:05:13+00:00 R: @@upgrade_rudder_agent_on_debian_family@@log_info@@4a0d9768-070d-4e5c-8153-d119da3706e8@@8a0e373b-d8dc-41f2-a91a-34e52df832c7@@2f1ce6fc-6eb0-437a-aaca-2eabe1c8ae27@@10-070-20-30 - Rudder repository credentials file permissions@@/etc/apt/auth.conf.d/rudder.conf@@2025-06-16 18:05:08+00:00##872d703d-2fd2-4641-b3f8-0f98d0f7226b@#Ensure permissions mode 640, owner root and group root on /etc/apt/auth.conf.d/rudder.conf on type all with 0 recursion level was correct
2025-06-16T18:05:13+00:00 R: @@upgrade_rudder_agent_on_debian_family@@result_success@@4a0d9768-070d-4e5c-8153-d119da3706e8@@8a0e373b-d8dc-41f2-a91a-34e52df832c7@@2f1ce6fc-6eb0-437a-aaca-2eabe1c8ae27@@10-070-20-30 - Rudder repository credentials file permissions@@/etc/apt/auth.conf.d/rudder.conf@@2025-06-16 18:05:08+00:00##872d703d-2fd2-4641-b3f8-0f98d0f7226b@#Ensure permissions mode 640, owner root and group root on /etc/apt/auth.conf.d/rudder.conf was correct
2025-06-16T18:05:13+00:00 R: @@upgrade_rudder_agent_on_debian_family@@result_na@@4a0d9768-070d-4e5c-8153-d119da3706e8@@8a0e373b-d8dc-41f2-a91a-34e52df832c7@@6738eedc-9fea-4a3f-9f9f-4e3e856d3e55@@10-080 - Update packages list@@apt update@@2025-06-16 18:05:08+00:00##872d703d-2fd2-4641-b3f8-0f98d0f7226b@#Skipping method 'Command execution' with key parameter 'apt update' since condition 'debian.(file_from_template__etc_apt_sources_list_d_rudder_list_repaired|file_lines_present__etc_apt_auth_conf_d_rudder_conf_repaired)' is not reached was not applicable
2025-06-16T18:05:13+00:00 R: @@upgrade_rudder_agent_on_debian_family@@result_na@@4a0d9768-070d-4e5c-8153-d119da3706e8@@8a0e373b-d8dc-41f2-a91a-34e52df832c7@@4060dd81-5fb7-4c3e-870b-891c1a9f4b2a@@10-090 - Purge available packages cache@@rm -f /var/rudder/cfengine-community/state/packages_updates*@@2025-06-16 18:05:08+00:00##872d703d-2fd2-4641-b3f8-0f98d0f7226b@#Skipping method 'Command execution' with key parameter 'rm -f /var/rudder/cfengine-community/state/packages_updates*' since condition 'debian.(command_execution_apt_update_kept|command_execution_apt_update_repaired)' is not reached was not applicable
cf-execd: timeout waiting for output from agent (agent_expireafter=120) - terminating it