Project

General

Profile

Bug #17917

On debian, package with new dependencies is not upgraded to latest available version

Added by Nicolas Ecarnot about 1 month ago. Updated 13 days ago.

Status:
Released
Priority:
N/A
Category:
System techniques
Target version:
Severity:
Major - prevents use of part of Rudder | no simple workaround
User visibility:
Getting started - demo | first install | Technique editor and level 1 Techniques
Effort required:
Priority:
70

Description

Hello,

One of our directives is ensuring the availability of the 'rudder-agent' package in the latest version on all our nodes.
When our root server was in version 6.0.6 and before, the upgrades went fine on our nodes (a mix of 6.0.4, 6.0.6 and 6.0.7 nodes, in Ubuntu, Debian, Centos, OL7...).

On 2020-07-06 morning, I upgraded the server into 6.1.1. On 3 test nodes, I changed the /etc/apt/sources.list.d/rudder.list accordingly to the doc.
For instance, on a ubuntu bionic, this is :
deb http://repository.rudder.io/apt/6.1/ bionic main

One night later, I see that the the rudder agent hasn't been upgraded :
  1. rudder agent version
    Rudder agent 6.0.6-ubuntu18.04

FYI, when manually upgrading the agent, it gets upgraded correctly.

UPDATE: as found in comments, the problem arise only with apt for packages with new dependencies.

WORKAROUND

- on server, edit file: /usr/share/ncf/tree/10_ncf_internals/modules/packages/apt_get
- add --with-new-pkgs on call to apt-get in method def list_updates(online) so that the line looks like:

process = subprocess_Popen([apt_get_cmd] + apt_get_options + ["--simulate", "--ignore-hold", "--with-new-pkgs", "upgrade"], stdout=subprocess.PIPE)

- save
- execute rudder agent run -u on the server
- on node, remove cache: rm -f /var/rudder/cfengine-community/state/packages_updates_*
- run rudder agent run -u

#2

Updated by François ARMAND about 1 month ago

Hello,

When you say "manually upgrading", it's with direct call to apt?

It may be due to cache in the package list (see our "rudder by example" on that use case: https://docs.rudder.io/rudder-by-example/current/system/update-rudder-agent-package.html#_delete_rpm_list_cache_when_repository_change) but after a night, it should be OK.

On an node where the update is not done yet, can you give us output of rudder agent run -i ? The same, after rm -f /var/rudder/cfengine-community/state/packages_updates_* ?

Alexis, do you know how we could debug that more?

#3

Updated by François ARMAND about 1 month ago

Ah thanks for the output (our messages were written at the same time). Can you try the rm/agent run too?

#4

Updated by François ARMAND about 1 month ago

In output, line 21509:

rudder  verbose: Package 'rudder-agent' is already in the latest version. Skipping installation.

So rudder believes there's no update available. So perhaps the cache problem.

#5

Updated by Félix DALLIDET about 1 month ago

This has some similarities with another bug reported some days ago see https://issues.rudder.io/issues/17893

#6

Updated by Nicolas Ecarnot about 1 month ago

After removing the packages_updates_*, and a rudder agent run, I still don't see any attempt to upgrade the package.

Felix, in my case, I see zero error.

#7

Updated by Nicolas CHARLES about 1 month ago

Logs states
rudder verbose: Ignoring failed to parse integer '$(ncf_def.package_module_query_installed_ifelapsed)' because of possibly unexpanded variable

this value is defined by

${node.properties[rudder][packages][installed_cache_expire]}

which is a global parameter
so something has probably failed during upgrade

#9

Updated by François ARMAND about 1 month ago

So, the problem is that rudder agent get new dependencies between ruder 6.0 and 6.1.

The command line used in /var/rudder/ncf/common/10_ncf_internals/modules/packages/apt_get doesn't accept that. If you translate list_update call, you get:

/usr/bin/apt-get -o 'Dpkg::Options::=--force-confold' -o 'Dpkg::Options::=--force-confdef' -y --allow-downgrades --allow-remove-essential --allow-change-held-packages --simulate --ignore-hold upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  rudder-agent
The following packages will be upgraded:
  ca-certificates

So this line need to have also --with-new-pkgs:

/usr/bin/apt-get -o 'Dpkg::Options::=--force-confold' -o 'Dpkg::Options::=--force-confdef' -y --allow-downgrades --allow-remove-essential --allow-change-held-packages --simulate --ignore-hold --with-new-pkgs upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  jq libjq1 libonig5
The following packages will be upgraded:
  ca-certificates rudder-agent

Unfortunately, if you try to add that parameter with generic methods `package state with option`, it doesn't get there.

#10

Updated by François ARMAND about 1 month ago

And so, the problem is that the apt package module ignore all option for list_updates methods (while they are parsed for yum module):

apt:

def list_updates(online):
    # Ignore everything.
    sys.stdin.readlines()

yum:

def list_updates(online):
    global yum_options
    for line in sys.stdin:
        line = line.strip()
        if line.startswith("options="):
            option = line[len("options="):]
            if option.startswith("-"):
                yum_options.append(option)
            elif option.startswith("enablerepo=") or option.startswith("disablerepo="):
                yum_options.append("--" + option)

Note: adding the option directly in apt module:

    process = subprocess_Popen([apt_get_cmd] + apt_get_options + ["--simulate", "--ignore-hold", "--with-new-pkgs", "upgrade"], stdout=subprocess.PIPE)

Leads to error:

E: Command line option --with-new-pkgs is not understood in combination with the other options

Even if the exact same command line in the console works as in the previous comment.

#11

Updated by François ARMAND about 1 month ago

  • Subject changed from Package is not upgraded to latest available version to On debian, package with new dependencies is not upgraded to latest available version
  • Description updated (diff)
#12

Updated by Nicolas Ecarnot about 1 month ago

François ARMAND wrote in #note-10:

And so, the problem is that the apt package module ignore all option for list_updates methods (while they are parsed for yum module):

apt:
[...]

yum:
[...]

Note: adding the option directly in apt module:
[...]

Leads to error:
[...]

Even if the exact same command line in the console works as in the previous comment.

Here [1], it's noticed that the error msg of apt is wrong, so that could be something to check.
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816437

#13

Updated by François ARMAND about 1 month ago

Unfortunatly it happens wherever I put that option in the line, so I doesn't seems to be that problem. Too bad, I hoped for a second.

#14

Updated by François ARMAND about 1 month ago

Update: we were looking for a workaround, even if it meant editing by hand /var/rudder/ncf/common/10_ncf_internals/modules/packages/apt_get on the server until we release a new version that handle option and we hit the error:

E: Command line option --with-new-pkgs is not understood in combination with the other options

So after several iterations, we ended with replacing the direct call to apt-get by a script (/tmp/apt-get) to be able to log. The script is:

#!/bin/bash

# log everything about about the command call / env
/usr/bin/apt-get --version >> /tmp/logs
echo "$@" >> /tmp/logs
env >> /tmp/logs

# a test to check that there's no error with a predefined output
#echo "Inst php5-cli [5.3.10-1ubuntu3.17] (5.3.10-1ubuntu3.18 Ubuntu:12.04/precise-updates [amd64]) []" 

# direct call to command line to avoid a problem in arg passing between python and command
#/usr/bin/apt-get -o 'Dpkg::Options::=--force-confold' -o 'Dpkg::Options::=--force-confdef' -y --allow-downgrades --allow-remove-essential --allow-change-held-packages --simulate --ignore-hold --with-new-pkgs -- upgrade

# simplified line
/usr/bin/apt-get --simulate --with-new-pkgs -- upgrade

And still, if we call the script in cli, it works as expected, seeing rudder-agent as upgradable. If we call it through rudder agent method, we got the error about unrecognized option. Everything in env/etc is the same.

So no idea right now what the problem is. And without that workaround working, we won't be able to pass the option by the generic method.

#15

Updated by Nicolas Ecarnot about 1 month ago

UPDATE: as found in comments, the problem arise only with apt for packages with new dependencies.

And to be precise, the issue is also hitting yum.

#16

Updated by François ARMAND about 1 month ago

  • Description updated (diff)
#17

Updated by François ARMAND about 1 month ago

So, actually modifying the python module to add --with-new-pkgs do work. The problem was that I used generic method "package with options" and added --with-new-pkgs as an option. That was leading to an other apt command failing with the message "option not recognized".

That means that the with option need a much better documentation to explain what methods are expected to get the option, or we need to find a way to filter out unrecognized option for commands (at least for apt).
It also means that perhaps the only patch possible for that case is to just add the proposed workaround in module.

#18

Updated by Nicolas Ecarnot 29 days ago

François ARMAND wrote in #note-17:

So, actually modifying the python module to add --with-new-pkgs do work. The problem was that I used generic method "package with options" and added --with-new-pkgs as an option. That was leading to an other apt command failing with the message "option not recognized".

That means that the with option need a much better documentation to explain what methods are expected to get the option, or we need to find a way to filter out unrecognized option for commands (at least for apt).
It also means that perhaps the only patch possible for that case is to just add the proposed workaround in module.

On the server, the file that needs to be changed is :
/usr/share/ncf/tree/10_ncf_internals/modules/packages/apt_get
according to @amousset.
Indeed, other files changes are overwritten.

When changing /usr/share/ncf/tree/10_ncf_internals/modules/packages/apt_get, I see the changes are forwarded towards the nodes.

Yet, no updates happens.

#19

Updated by François ARMAND 29 days ago

  • Description updated (diff)

Update workaround description

#20

Updated by François ARMAND 29 days ago

  • Description updated (diff)
#21

Updated by François ARMAND 23 days ago

  • Status changed from New to In progress
  • Assignee changed from Nicolas CHARLES to François ARMAND
#22

Updated by François ARMAND 23 days ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from François ARMAND to Alexis MOUSSET
  • Pull Request set to https://github.com/Normation/ncf/pull/1231
#23

Updated by François ARMAND 22 days ago

  • Assignee changed from Alexis MOUSSET to Nicolas CHARLES
#24

Updated by Vincent MEMBRÉ 21 days ago

  • Target version changed from 6.1.2 to 6.1.3
#25

Updated by François ARMAND 17 days ago

  • Status changed from Pending technical review to Pending release
#26

Updated by François ARMAND 14 days ago

  • Target version changed from 6.1.3 to 6.1.2
#28

Updated by François ARMAND 13 days ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 6.1.2 which was released today.

Also available in: Atom PDF