Project

General

Profile

Actions

Bug #3830

closed

Technique "Package management for APT systems": packages with suffix :amd64 are not correctly detected

Added by Fabrice FLORE-THÉBAULT almost 11 years ago. Updated over 9 years ago.

Status:
Released
Priority:
N/A
Category:
Techniques
Target version:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
Name check:
Fix check:
Regression:

Description

Let's install openjdk-6-jre on debian squeeze.

If we create a directive to install openjdk-6-jre package, then the package gets reinstalled at each run of the agent.

    Installing openjdk-6-jre...
    Q:env DEBIAN_FRONTEND= ...:Reading package lists...
    Q:env DEBIAN_FRONTEND= ...:Building dependency tree...
    Q:env DEBIAN_FRONTEND= ...:Reading state information...
    Q:env DEBIAN_FRONTEND= ...:Reading extended state information...
    Q:env DEBIAN_FRONTEND= ...:Initializing package states...
    Q:env DEBIAN_FRONTEND= ...:Reading task descriptions...
    Q:env DEBIAN_FRONTEND= ...:No packages will be installed, upgraded, or removed.
    Q:env DEBIAN_FRONTEND= ...:0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Q:env DEBIAN_FRONTEND= ...:Need to get 0 B of archives. After unpacking 0 B will be used.
    Q:env DEBIAN_FRONTEND= ...:Writing extended state information...
    Q:env DEBIAN_FRONTEND= ...:Reading package lists...
    Q:env DEBIAN_FRONTEND= ...:Building dependency tree...
    Q:env DEBIAN_FRONTEND= ...:Reading state information...
    Q:env DEBIAN_FRONTEND= ...:Reading extended state information...
    Q:env DEBIAN_FRONTEND= ...:Initializing package states...
    Q:env DEBIAN_FRONTEND= ...:Reading task descriptions...
    Q:env DEBIAN_FRONTEND= ...:
    -> Command related to promiser "openjdk-6-jre" returned code defined as promise kept (0)

The reason seems to be that aptitude and dpkg give different results:

aptitude search openjdk-6-jre
i   openjdk-6-jre                                      - environnement d'exécution Java OpenJDK qui utilise Hotspot J
i   openjdk-6-jre-headless                             - environnement d'exécution Java OpenJDK utilisant Hotspot JIT
i   openjdk-6-jre-lib                                  - environnement d'exécution Java OpenJDK - bibliothèques indép
p   openjdk-6-jre-zero                                 - Alternative JVM for OpenJDK, using Zero/Shark  
dpkg -l openjdk-6-jre*
Souhait=inconnU/Installé/suppRimé/Purgé/H=à garder
| État=Non/Installé/fichier-Config/dépaqUeté/échec-conFig/H=semi-installé/W=attend-traitement-déclenchements
|/ Err?=(aucune)/besoin Réinstallation (État,Err: majuscule=mauvais)
||/ Nom                     Version          Architecture     Description
+++-=======================-================-================-===================================================
ii  openjdk-6-jre:amd64     6b27-1.12.6-1~de amd64            OpenJDK Java runtime, using Hotspot JIT
ii  openjdk-6-jre-headless: 6b27-1.12.6-1~de amd64            OpenJDK Java runtime, using Hotspot JIT (headless)
ii  openjdk-6-jre-lib       6b27-1.12.6-1~de all              OpenJDK Java runtime (architecture independent libr
un  openjdk-6-jre-zero      <aucun>                           (aucune description n'est disponible)

If we create a directive to install openjdk-6-jre:amd64 package, then the package installed only once.

Actions #1

Updated by Nicolas PERRON almost 11 years ago

  • Status changed from New to Discussion
  • Assignee set to Fabrice FLORE-THÉBAULT

Thank you for your report. Could you add the output of the command ?:

grep 'openjdk' /var/rudder/cfengine-community/state/software_packages.csv

It seems to me that this is the problem.

Actions #2

Updated by Fabrice FLORE-THÉBAULT almost 11 years ago

openjdk-6-jre-lib,6b27-1.12.6-1~deb7u1,default,dpkg
openjdk-6-jre-headless:amd64,6b27-1.12.6-1~deb7u1,default,dpkg
openjdk-6-jre:amd64,6b27-1.12.6-1~deb7u1,default,dpkg
Actions #3

Updated by Nicolas PERRON almost 11 years ago

  • Assignee changed from Fabrice FLORE-THÉBAULT to Matthieu CERDA

I was working on this bug and...

First, I was able to reproduce it:

  • On a Squeeze, the installation of the openjdk-6-jre will only be considered as a success:
    root@debian-6-64:~# /var/rudder/cfengine-community/bin/cf-agent -KIb check_apt_package_installation
     >> Using command line specified bundlesequence
    R: @@packageInstallation@@result_success@@49173e84-0818-4082-84e1-a00d8b6f2fbe@@78922455-116b-41ad-ae11-237abb4b4608@@2@@Debian/Ubuntu packages@@openjdk-6-jre@@2013-08-13 15:41:44+02:00##b9a71482-5030-4699-984d-b03d28bbbf36@#No action required for deb package openjdk-6-jre with policy: add
    
  • on a Wheezy, with multi arch activated (dpkg --add-architecture i386), the package installed will need the use of aptitude each time:
    root@debian-7-64:~# /var/rudder/cfengine-community/bin/cf-agent -KIb check_apt_package_installation
    [...]
    Installing openjdk-6-jre...
    Q:env DEBIAN_FRONTEND= ...:Reading package lists...
    Q:env DEBIAN_FRONTEND= ...:Building dependency tree...
    Q:env DEBIAN_FRONTEND= ...:Reading state information...
    Q:env DEBIAN_FRONTEND= ...:Reading extended state information...
    Q:env DEBIAN_FRONTEND= ...:Initializing package states...
    Q:env DEBIAN_FRONTEND= ...:Reading task descriptions...
    Q:env DEBIAN_FRONTEND= ...:No packages will be installed, upgraded, or removed.
    Q:env DEBIAN_FRONTEND= ...:0 packages upgraded, 0 newly installed, 0 to remove and 35 not upgraded.
    Q:env DEBIAN_FRONTEND= ...:Need to get 0 B of archives. After unpacking 0 B will be used.
    Q:env DEBIAN_FRONTEND= ...:Writing extended state information...
    Q:env DEBIAN_FRONTEND= ...:Reading package lists...
    Q:env DEBIAN_FRONTEND= ...:Building dependency tree...
    Q:env DEBIAN_FRONTEND= ...:Reading state information...
    Q:env DEBIAN_FRONTEND= ...:Reading extended state information...
    Q:env DEBIAN_FRONTEND= ...:Initializing package states...
    Q:env DEBIAN_FRONTEND= ...:Reading task descriptions...
    Q:env DEBIAN_FRONTEND= ...:
    -> Command related to promiser "openjdk-6-jre" returned code defined as promise kept (0)
    R: @@packageInstallation@@result_success@@b3d33690-89f6-46ab-982b-8365bfd6cfd7@@bacb72ee-6a0a-4228-b32d-040d172251de@@1@@Debian/Ubuntu packages@@openjdk-6-jre@@2013-08-13 15:57:12+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#No action required for deb package openjdk-6-jre with policy: add
    
And the content in the software_packages.csv file is different on the first and the second case:
  • On Squeeze:
    root@debian-6-64:~# grep openjdk-6 /var/rudder/cfengine-community/state/software_packages.csv 
    openjdk-6-jre-lib,6b27-1.12.6-1~deb6u1,default,dpkg
    openjdk-6-jre-headless,6b27-1.12.6-1~deb6u1,default,dpkg
    openjdk-6-jre,6b27-1.12.6-1~deb6u1,default,dpkg
    
  • On Wheezy multi arch:
    root@debian-7-64:~# grep openjdk-6 /var/rudder/cfengine-community/state/software_packages.csv 
    openjdk-6-jre-lib,6b27-1.12.6-1~deb7u1,default,dpkg
    openjdk-6-jre-headless:amd64,6b27-1.12.6-1~deb7u1,default,dpkg
    openjdk-6-jre:amd64,6b27-1.12.6-1~deb7u1,default,dpkg
    

Secondly, i've found that an option was missing into the package_method body used into the promise: package_default_arch_command (cf https://cfengine.com/archive/manuals/cf3-Reference#package_005fmethod-in-packages)

This command allows CFEngine to detect default architecture of packages managed by package manager. As an example, multiarch-enabled dpkg only lists architectures explicitly for multiarch-enabled packages.

In case this command is not provided, CFEngine treats all packages without explicit architecture set as belonging to implicit “default” architecture. 

Then I've added it to the generated promise but the problem was still there. With the verbose mode I was able to know that the problem is really from the file software_packages.csv (which contains the packages installed on the machine...this is a cache file):

  • On Squeeze:
    [...]
    rudder>    =========================================================
    rudder>    packages in bundle check_apt_package_installation (1)
    rudder>    =========================================================
    rudder>
    rudder>
    rudder>     .........................................................
    rudder>     Promise's handle:
    rudder>     Promise made by: "openjdk-6-jre" 
    rudder>
    rudder>     Comment:  Handling openjdk-6-jre using apt_nobulk, policy : add
    rudder>     .........................................................
    rudder>
    rudder> Obtaining default architecture for package manager: /usr/bin/dpkg --print-architecture
    rudder> Default architecture for package manager is 'amd64'
    rudder>  -> Cache file exists and is sufficiently fresh according to (package_list_update_ifelapsed)
    rudder>  -> Package (zlib1g-dev,1:1.2.3.4.dfsg-3,amd64) found
    [...]
    rudder>  -> Package (openjdk-6-jre-lib,6b27-1.12.6-1~deb6u1,amd64) found
    rudder>  -> Package (openjdk-6-jre-headless,6b27-1.12.6-1~deb6u1,amd64) found
    rudder>  -> Package (openjdk-6-jre,6b27-1.12.6-1~deb6u1,amd64) found
    [...]
    rudder>  -> Package version was not specified
    rudder>  -> Looking for (openjdk-6-jre,*,*)
    rudder>  -> Matched name openjdk-6-jre
    rudder>  -> Matched version *
    rudder>  -> Looking for (openjdk-6-jre,*,*)
    rudder>  -> Matched name openjdk-6-jre
    rudder>  -> Matched version *
    rudder> Checking if package (openjdk-6-jre,*,*) is at the desired state (installed=1,matched=1)
    rudder>  -> Package promises to refer to itself as "openjdk-6-jre" to the manager
    rudder>  -> Package version seems to match criteria
    rudder>  -> Package "openjdk-6-jre" already installed, so we never add it again
    rudder>  ?> defining promise result class debian_install_kept_openjdk_6_jre
    [...]
    
  • On Wheezy multiarch:
    [...]
    rudder>    =========================================================
    rudder>    packages in bundle check_apt_package_installation (1)
    rudder>    =========================================================
    rudder>
    rudder>
    rudder>     .........................................................
    rudder>     Promise's handle:
    rudder>     Promise made by: "openjdk-6-jre" 
    rudder>
    rudder>     Comment:  Handling openjdk-6-jre using apt_nobulk, policy : add
    rudder>     .........................................................
    rudder>
    rudder> Obtaining default architecture for package manager: /usr/bin/dpkg --print-architecture
    rudder> Default architecture for package manager is 'amd64'
    rudder>  -> Cache file exists and is sufficiently fresh according to (package_list_update_ifelapsed)
    rudder>  -> Package (zlib1g:amd64,1:1.2.7.dfsg-13,amd64) found
    [...]
    rudder>  -> Package (openjdk-6-jre-lib,6b27-1.12.6-1~deb7u1,amd64) found
    rudder>  -> Package (openjdk-6-jre-headless:amd64,6b27-1.12.6-1~deb7u1,amd64) found
    rudder>  -> Package (openjdk-6-jre:amd64,6b27-1.12.6-1~deb7u1,amd64) found
    [...]
    rudder>  -> Package version was not specified
    rudder>  -> Looking for (openjdk-6-jre,*,*)
    rudder> No installed packages matched (openjdk-6-jre,*,*)
    rudder>  -> Looking for (openjdk-6-jre,*,*)
    rudder> No installed packages matched (openjdk-6-jre,*,*)
    rudder> Checking if package (openjdk-6-jre,*,*) is at the desired state (installed=0,matched=0)
    rudder>  -> Package promises to refer to itself as "openjdk-6-jre" to the manager
    rudder>  -> Package version seems to match criteria
    rudder>  -> Schedule package for addition
    rudder>  -> Package (openjdk-6-jre,any,any) found
    [...]
    

The most important part is that to know if the package is installed, the regex used is wrong:

rudder>  -> Looking for (openjdk-6-jre,*,*)
rudder> No installed packages matched (openjdk-6-jre,*,*)

Because since the multiarch is activated the installed package is described as:

Package (openjdk-6-jre:amd64,6b27-1.12.6-1~deb7u1,amd64) found

Finally, I suppose this is a CFEngine bug. Do you agree with me Matthieu ?

Actions #4

Updated by Nicolas PERRON almost 11 years ago

  • Status changed from Discussion to In progress
  • Assignee changed from Matthieu CERDA to Nicolas PERRON

This is really a CFEngine bug, and this is not specific to the activation of multiarch

Actions #5

Updated by Nicolas PERRON almost 11 years ago

  • Status changed from In progress to 8

I've opened a bug into CFEngine bug tracker here: https://cfengine.com/dev/issues/3280

I hope they will work on it since it concerns CFEngine 3.5.1-3...

Actions #6

Updated by Nicolas PERRON almost 11 years ago

  • Status changed from 8 to In progress

I've found the problem. It was in the definition of the package_method. It should just not match the ':' which correspond to the architecture. I will make a Pull Request ASAP :)

Actions #7

Updated by Nicolas PERRON almost 11 years ago

  • Target version changed from 2.6.4 to 2.4.8

This issue concerns all versions of Rudder, of course

Actions #8

Updated by Nicolas PERRON almost 11 years ago

  • Project changed from Rudder to 24
  • Category deleted (Techniques)
Actions #9

Updated by Nicolas PERRON almost 11 years ago

  • Status changed from In progress to Pending technical review
  • Assignee changed from Nicolas PERRON to Matthieu CERDA
  • % Done changed from 0 to 100

Pull Request URL added: https://github.com/Normation/rudder-techniques/pull/165

Matthieu, since Jonathan and Nicolas are absent, could you review it please ?

Actions #10

Updated by Matthieu CERDA almost 11 years ago

  • Assignee changed from Matthieu CERDA to Nicolas PERRON

I approve this PR. Having multiarch support is good !

Actions #11

Updated by Nicolas PERRON almost 11 years ago

  • Assignee changed from Nicolas PERRON to Matthieu CERDA

Matthieu, I have updated the PR. Could you review it please ?

Actions #12

Updated by Jonathan CLARKE almost 11 years ago

  • Status changed from Pending technical review to Discussion
  • Assignee changed from Matthieu CERDA to Nicolas PERRON

I'm sorry but I'm not keen on this approach. If there are bugs upstream (ie, in CFEngine, or it's stdlib as in this case), we must strive to fix them upstream. Otherwise, this is equivalent to forking their code, with all the inconvenvience that brings: we have to maintain it, we won't benefit from other bug fixes, etc...

So I think a better way to go about this would be to send a PR to CFEngine to change the buggy package_method on their stdlib, and patch our copy of cfengine_stdlib.cf until the PR is merged upstream.

Nicolas, can you work on this please?

Actions #13

Updated by Nicolas PERRON almost 11 years ago

  • Assignee changed from Nicolas PERRON to Jonathan CLARKE

Jonathan CLARKE wrote:

I'm sorry but I'm not keen on this approach. If there are bugs upstream (ie, in CFEngine, or it's stdlib as in this case), we must strive to fix them upstream. Otherwise, this is equivalent to forking their code, with all the inconvenvience that brings: we have to maintain it, we won't benefit from other bug fixes, etc...

So I think a better way to go about this would be to send a PR to CFEngine to change the buggy package_method on their stdlib, and patch our copy of cfengine_stdlib.cf until the PR is merged upstream.

Nicolas, can you work on this please?

Ok, now i've made a Pull Request what should we do for our Techniques ? Should we patch our CFEngine standard Library and continue to use the body from it ?

Actions #14

Updated by Jonathan CLARKE almost 11 years ago

Nicolas PERRON wrote:

Jonathan CLARKE wrote:

I'm sorry but I'm not keen on this approach. If there are bugs upstream (ie, in CFEngine, or it's stdlib as in this case), we must strive to fix them upstream. Otherwise, this is equivalent to forking their code, with all the inconvenvience that brings: we have to maintain it, we won't benefit from other bug fixes, etc...

So I think a better way to go about this would be to send a PR to CFEngine to change the buggy package_method on their stdlib, and patch our copy of cfengine_stdlib.cf until the PR is merged upstream.

Ok, now i've made a Pull Request what should we do for our Techniques ? Should we patch our CFEngine standard Library and continue to use the body from it ?

We should patch all package_methods in our promises that need it. That includes, but is not limited to, the cfengine_stdlib.cf. That way, we avoid modifying our Techniques, and we ensure that things will keep running smoothly when the stdlib is update, and we continue to benefit from other improvements in the stdlib.

Actions #15

Updated by Jonathan CLARKE almost 11 years ago

  • Assignee changed from Jonathan CLARKE to Nicolas PERRON
Actions #16

Updated by Nicolas PERRON almost 11 years ago

  • Target version changed from 2.4.8 to 2.4.9
Actions #17

Updated by Nicolas PERRON over 10 years ago

  • Assignee changed from Nicolas PERRON to Jonathan CLARKE

Jon, patch has been accepted and merged but for CFEngine 3.5.*. What should we do now ? merging the PR here ? wait for a Rudder release including CFEngine 3.5.* ?

Actions #18

Updated by Jonathan CLARKE over 10 years ago

  • Assignee changed from Jonathan CLARKE to Nicolas PERRON
  • Pull Request set to https://github.com/Normation/rudder-techniques/pull/165

Nicolas PERRON wrote:

Jon, patch has been accepted and merged but for CFEngine 3.5.*. What should we do now ? merging the PR here ? wait for a Rudder release including CFEngine 3.5.* ?

Nicolas, I already explained the approach to take in comment #14 above.

The current Pull Request attached to this issue will not be merged as is, as it constitues a fork of one of CFEngine's most complex bodies. We have provided upstream with a patch, we must apply this to our copy too and move on.

Actions #19

Updated by Nicolas PERRON over 10 years ago

  • Target version changed from 2.4.9 to 2.4.10
Actions #20

Updated by Nicolas PERRON over 10 years ago

  • Target version changed from 2.4.10 to 2.4.11
Actions #21

Updated by Nicolas PERRON over 10 years ago

  • Target version changed from 2.4.11 to 2.4.12
Actions #22

Updated by Nicolas PERRON over 10 years ago

  • Target version changed from 2.4.12 to 2.4.13
Actions #23

Updated by Nicolas CHARLES over 10 years ago

  • Category set to Techniques
  • Status changed from Discussion to 8
  • Assignee changed from Nicolas PERRON to Nicolas CHARLES
  • Target version changed from 2.4.13 to 2.6.11

Bug has been fixed upstream \o/
https://github.com/cfengine/core/pull/1294/files

Going to import the fix from CFEngine

Actions #24

Updated by Nicolas CHARLES over 10 years ago

ha, i completly missed that one, as it was qualified
yeah for fix from upstream !

Actions #25

Updated by Nicolas CHARLES over 10 years ago

  • Pull Request changed from https://github.com/Normation/rudder-techniques/pull/165 to https://github.com/Normation/rudder-techniques/pull/298
Actions #26

Updated by Nicolas CHARLES over 10 years ago

  • Status changed from 8 to Pending technical review
  • Assignee changed from Nicolas CHARLES to Jonathan CLARKE
Actions #27

Updated by Nicolas CHARLES over 10 years ago

  • Status changed from Pending technical review to Pending release

Applied in changeset commit:168e2fe0bc1b63fc651d8d347b68a8d46282ee06.

Actions #28

Updated by Jonathan CLARKE over 10 years ago

Applied in changeset commit:a538bd3518afad3e150f9b183c2bfc205f031576.

Actions #29

Updated by Vincent MEMBRÉ over 10 years ago

  • Subject changed from some inconsistencies in detection of packages installed on debian when package has the :amd64 suffix to Technique "Package management for APT systems": packages with suffix :amd64 are not correctly detected
Actions #30

Updated by Vincent MEMBRÉ over 10 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 2.6.11, which was released today.
Check out:

Actions #31

Updated by Benoît PECCATTE over 9 years ago

  • Project changed from 24 to Rudder
  • Category changed from Techniques to Techniques
Actions

Also available in: Atom PDF