Rudder

Here is the typescript of a rudder agent run -v :
https://wetransfer.com/downloads/e721ea1ab05c722c6aaa590e9d1d758920200707085419/605c9d255344d536875a439318fa304420200707085642/aeb62a

Hello,

When you say "manually upgrading", it's with direct call to apt?

It may be due to cache in the package list (see our "rudder by example" on that use case: https://docs.rudder.io/rudder-by-example/current/system/update-rudder-agent-package.html#_delete_rpm_list_cache_when_repository_change) but after a night, it should be OK.

On an node where the update is not done yet, can you give us output of rudder agent run -i ? The same, after rm -f /var/rudder/cfengine-community/state/packages_updates_* ?

Alexis, do you know how we could debug that more?

Ah thanks for the output (our messages were written at the same time). Can you try the rm/agent run too?

In output, line 21509:

rudder  verbose: Package 'rudder-agent' is already in the latest version. Skipping installation.

So rudder believes there's no update available. So perhaps the cache problem.

This has some similarities with another bug reported some days ago see https://issues.rudder.io/issues/17893

After removing the packages_updates_*, and a rudder agent run, I still don't see any attempt to upgrade the package.

Felix, in my case, I see zero error.

Logs states
rudder verbose: Ignoring failed to parse integer '$(ncf_def.package_module_query_installed_ifelapsed)' because of possibly unexpanded variable

this value is defined by

${node.properties[rudder][packages][installed_cache_expire]}

So, the problem is that rudder agent get new dependencies between ruder 6.0 and 6.1.

The command line used in /var/rudder/ncf/common/10_ncf_internals/modules/packages/apt_get doesn't accept that. If you translate list_update call, you get:

/usr/bin/apt-get -o 'Dpkg::Options::=--force-confold' -o 'Dpkg::Options::=--force-confdef' -y --allow-downgrades --allow-remove-essential --allow-change-held-packages --simulate --ignore-hold upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  rudder-agent
The following packages will be upgraded:
  ca-certificates

So this line need to have also --with-new-pkgs:

/usr/bin/apt-get -o 'Dpkg::Options::=--force-confold' -o 'Dpkg::Options::=--force-confdef' -y --allow-downgrades --allow-remove-essential --allow-change-held-packages --simulate --ignore-hold --with-new-pkgs upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  jq libjq1 libonig5
The following packages will be upgraded:
  ca-certificates rudder-agent

Unfortunately, if you try to add that parameter with generic methods `package state with option`, it doesn't get there.

And so, the problem is that the apt package module ignore all option for list_updates methods (while they are parsed for yum module):

apt:

def list_updates(online):
    # Ignore everything.
    sys.stdin.readlines()

yum:

def list_updates(online):
    global yum_options
    for line in sys.stdin:
        line = line.strip()
        if line.startswith("options="):
            option = line[len("options="):]
            if option.startswith("-"):
                yum_options.append(option)
            elif option.startswith("enablerepo=") or option.startswith("disablerepo="):
                yum_options.append("--" + option)

Note: adding the option directly in apt module:

    process = subprocess_Popen([apt_get_cmd] + apt_get_options + ["--simulate", "--ignore-hold", "--with-new-pkgs", "upgrade"], stdout=subprocess.PIPE)

Leads to error:

E: Command line option --with-new-pkgs is not understood in combination with the other options

Even if the exact same command line in the console works as in the previous comment.

François ARMAND wrote in #note-10:

And so, the problem is that the apt package module ignore all option for list_updates methods (while they are parsed for yum module):

apt:
[...]

yum:
[...]

Note: adding the option directly in apt module:
[...]

Leads to error:
[...]

Even if the exact same command line in the console works as in the previous comment.

Here [1], it's noticed that the error msg of apt is wrong, so that could be something to check.
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816437

Unfortunatly it happens wherever I put that option in the line, so I doesn't seems to be that problem. Too bad, I hoped for a second.

Update: we were looking for a workaround, even if it meant editing by hand /var/rudder/ncf/common/10_ncf_internals/modules/packages/apt_get on the server until we release a new version that handle option and we hit the error:

E: Command line option --with-new-pkgs is not understood in combination with the other options

So after several iterations, we ended with replacing the direct call to apt-get by a script (/tmp/apt-get) to be able to log. The script is:

#!/bin/bash

# log everything about about the command call / env
/usr/bin/apt-get --version >> /tmp/logs
echo "$@" >> /tmp/logs
env >> /tmp/logs

# a test to check that there's no error with a predefined output
#echo "Inst php5-cli [5.3.10-1ubuntu3.17] (5.3.10-1ubuntu3.18 Ubuntu:12.04/precise-updates [amd64]) []" 

# direct call to command line to avoid a problem in arg passing between python and command
#/usr/bin/apt-get -o 'Dpkg::Options::=--force-confold' -o 'Dpkg::Options::=--force-confdef' -y --allow-downgrades --allow-remove-essential --allow-change-held-packages --simulate --ignore-hold --with-new-pkgs -- upgrade

# simplified line
/usr/bin/apt-get --simulate --with-new-pkgs -- upgrade

And still, if we call the script in cli, it works as expected, seeing rudder-agent as upgradable. If we call it through rudder agent method, we got the error about unrecognized option. Everything in env/etc is the same.

So no idea right now what the problem is. And without that workaround working, we won't be able to pass the option by the generic method.

UPDATE: as found in comments, the problem arise only with apt for packages with new dependencies.

And to be precise, the issue is also hitting yum.

So, actually modifying the python module to add --with-new-pkgs do work. The problem was that I used generic method "package with options" and added --with-new-pkgs as an option. That was leading to an other apt command failing with the message "option not recognized".

That means that the with option need a much better documentation to explain what methods are expected to get the option, or we need to find a way to filter out unrecognized option for commands (at least for apt).
It also means that perhaps the only patch possible for that case is to just add the proposed workaround in module.

François ARMAND wrote in #note-17:

So, actually modifying the python module to add --with-new-pkgs do work. The problem was that I used generic method "package with options" and added --with-new-pkgs as an option. That was leading to an other apt command failing with the message "option not recognized".

That means that the with option need a much better documentation to explain what methods are expected to get the option, or we need to find a way to filter out unrecognized option for commands (at least for apt).
It also means that perhaps the only patch possible for that case is to just add the proposed workaround in module.

On the server, the file that needs to be changed is :
/usr/share/ncf/tree/10_ncf_internals/modules/packages/apt_get
according to @Alexis Mousset.
Indeed, other files changes are overwritten.

When changing /usr/share/ncf/tree/10_ncf_internals/modules/packages/apt_get, I see the changes are forwarded towards the nodes.

Yet, no updates happens.