Project

General

Profile

Bug #3855

Rudder causes RHEL nodes to be blacklisted in RHN due to intensive yum usage

Added by Dennis Cabooter over 6 years ago. Updated almost 5 years ago.

Status:
Released
Priority:
1
Category:
Techniques
Target version:
Severity:
User visibility:
Effort required:
Priority:

Description

Twelve of our RHEL nodes are managed by Rudder. Since Rudder executes Yum many times a day, RHN blacklists that twelve nodes every week. We already tried to set "Section: Check interval" > "Check package statuses every (minutes): " to 60 minutes. However, this only slows down the problem. To "solve" this problem we have to unregister and reregister all twelve machines every week, which is quitte annoying.

- What exactly happens is described here: https://bugzilla.redhat.com/show_bug.cgi?id=824663.
- I can't find anything about this problem related to Cfengine. However, the one node not managed by Rudder doesn't have the problem at all.
- I found something about this problem related to Puppet: http://code.ohloh.net/file?fid=kzak1FGMdZUOw6az9Ka_k1whGic&cid=e5mmST0PD9o&s=&fp=274571&mpundefined=&projSelected=true#L0. However, it doesn't seem to be an official Puppet solution.
- We use a local RHEL yum mirror, so we can upgrade without using RHN. I'm not sure if disabling the RHN plugin is a solution for nodes registered to RHN without a local yum mirror.
- It would be nice to be able to have the RHN plugin enabled on a node while it's managed by Rudder. However, I haven't found a solution other then disabling it.


Subtasks

Bug #4898: The rudder_yum package method fails due to a wrong attribute presenceReleased2014-05-27Jonathan CLARKEActions

Related issues

Related to Rudder - User story #3490: Adapt the generic package_method from the cfengine_stdlib with individual package_changesRejectedActions
Related to Rudder - Bug #4892: Some Techniques are using insidiously (or not) package manager without package_method 'rudder_yum' on RHEL leading to a blacklist from RHNRejectedActions
Has duplicate Rudder - Bug #4891: Techniques Inventory, System and NTP use type promises 'packages' and could blacklist from RHNRejected2014-05-23Jonathan CLARKEActions
#1

Updated by Dennis Cabooter over 6 years ago

Our /etc/yum/pluginconf.d/rhnplugin.conf looks like this for now:

[main]
enabled = 0
gpgcheck = 1

[rhel-x86_64-server-5]
enabled = 0

# You can specify options per channel, e.g.:
#
#[rhel-i386-server-5]
#enabled = 1
#
#[some-unsigned-custom-channel]
#gpgcheck = 0
#2

Updated by Matthieu CERDA over 6 years ago

  • Status changed from New to Discussion
  • Assignee set to Nicolas PERRON
  • Priority changed from N/A to 1

This is a serious issue, setting priority to 1.

Nicolas, I do not have a clear idea as of now about how to solve this, do you ?

#3

Updated by Nicolas PERRON over 6 years ago

  • Assignee changed from Nicolas PERRON to Dennis Cabooter

Matthieu CERDA wrote:

This is a serious issue, setting priority to 1.

Nicolas, I do not have a clear idea as of now about how to solve this, do you ?

The problem seems to be the several use of yum. In my opinion, the package manager does not need to be used more than one time a day.

I see that you've modified Check interval to 60 mins. Why don't you try to set it to 360 ?

#4

Updated by Olivier Mauras over 6 years ago

So if one update his/her internal repos several times a day, new packages wouldn't be deployed straight ahead...
One would have to remove manually the /var/rudder/cfengine-community/state/software_packages.csv which defeats automation.

#5

Updated by Vincent MEMBRÉ over 6 years ago

Maybe the interval could be defined for each package, overriding the default interval for that package ?

#6

Updated by Nicolas PERRON over 6 years ago

Olivier Mauras wrote:

So if one update his/her internal repos several times a day, new packages wouldn't be deployed straight ahead...

It seems to me that the problem is the same manually. If soeone update the repository too many times, the blacklist will apply too. How should it work normally ?

#7

Updated by Dennis Cabooter over 6 years ago

Olivier Mauras wrote:

So if one update his/her internal repos several times a day, new packages wouldn't be deployed straight ahead...
One would have to remove manually the /var/rudder/cfengine-community/state/software_packages.csv which defeats automation.

That's the problem indeed.

#8

Updated by Olivier Mauras over 6 years ago

Nicolas PERRON wrote:

Olivier Mauras wrote:

So if one update his/her internal repos several times a day, new packages wouldn't be deployed straight ahead...

It seems to me that the problem is the same manually. If soeone update the repository too many times, the blacklist will apply too. How should it work normally ?

Well when not using RHEL you won't ever be blacklisted and you may still need to make a lot of changes to your internal repositories in a day - At least i do :)

#9

Updated by Dennis Cabooter over 6 years ago

I am using a local mirror. However, disabling rhn causes rhn to list my nodes as out of date. There might be a solution in this topic: https://groups.google.com/forum/#!msg/help-cfengine/FaCPLJjj0mA/3N-vbSivkdgJ.

#10

Updated by Dennis Cabooter over 6 years ago

Is Rudder using this to list packages?

body package_method yum_rpm

# Contributed by Trond Hasle Amundsen

# More efficient package method for RedHat - uses rpm to list instead of yum
#11

Updated by Nicolas PERRON over 6 years ago

Dennis Cabooter wrote:

Is Rudder using this to list packages?

[...]

The package_list used by RHEL/CentOS is:

[...]
redhat::
    package_list_command => "/usr/bin/yum list installed";
[...]

We were inspired by the body yum:

[...]
body package_method yum
{
        package_changes => "bulk";
        package_list_command => "/usr/bin/yum list installed";
[...]

If I understand well, the fix could be to use the body yum_rpm for listing packages ?

body package_method yum_rpm

# Contributed by Trond Hasle Amundsen

# More efficient package method for RedHat - uses rpm to list instead of yum
# Notes:
# - using $(name).$(arch) instead of $(name) for package_name_convention
#   causes uninstallation to fail.
# - using allmatches to remove for all architectures
#

{
        package_changes => "bulk";
        package_list_command => "/bin/rpm -qa --qf '%{name} %{version}-%{release} %{arch}\n'";
[...]
#12

Updated by Dennis Cabooter over 6 years ago

As far as I know a node be blacklisted in rhn classic if yum connects too many times. So if rudder just needs to check for a package if it's installed, it would be better to use rpm. The author of yum_rpm also says it's faster. :)

#13

Updated by Dennis Cabooter over 6 years ago

For now I'm testing with a fork of the rpmPackageInstallation technique:

diff -Nru 5.0/rpmPackageInstallation.st 5.1/rpmPackageInstallation.st
--- 5.0/rpmPackageInstallation.st    2013-09-04 08:51:47.000000000 +0200
+++ 5.1/rpmPackageInstallation.st    2013-09-12 10:46:35.000000000 +0200
@@ -124,16 +124,27 @@
     package_changes => "individual";

 redhat::
-    package_list_command => "/usr/bin/yum list installed";
-    package_list_name_regex    => "([^.]+).*";
-    package_list_version_regex => "[^\s]\s+([^\s]+).*";
-    package_list_arch_regex    => "[^.]+\.([^\s]+).*";
-    package_installed_regex => ".*installed.*";
-    package_name_convention => "${name}";
+#    package_list_command => "/usr/bin/yum list installed";
+#    package_list_name_regex    => "([^.]+).*";
+#    package_list_version_regex => "[^\s]\s+([^\s]+).*";
+#    package_list_arch_regex    => "[^.]+\.([^\s]+).*";
+#    package_installed_regex => ".*installed.*";
+#    package_name_convention => "${name}";
+#    package_list_update_ifelapsed => "$(rpm_pkg_timeout)";
+#    package_add_command => "/usr/bin/yum -y install";
+#    package_delete_command => "/bin/rpm -e";
+#    package_verify_command => "/bin/rpm -V";
+
+        package_list_command => "/bin/rpm -qa --qf '%{name} %{version}-%{release} %{arch}\n'";
+        package_list_name_regex    => "^(\S+?)\s\S+?\s\S+$";
+        package_list_version_regex => "^\S+?\s(\S+?)\s\S+$";
+        package_list_arch_regex    => "^\S+?\s\S+?\s(\S+)$";
+        package_installed_regex => ".*";
+        package_name_convention => "$(name)";
     package_list_update_ifelapsed => "$(rpm_pkg_timeout)";
-    package_add_command => "/usr/bin/yum -y install";
-    package_delete_command => "/bin/rpm -e";
-    package_verify_command => "/bin/rpm -V";
+        package_add_command    => "/usr/bin/yum -y install";
+        package_delete_command => "/bin/rpm -e --nodeps --allmatches";
+        package_verify_command => "/bin/rpm -V";

 SuSE_11::
     package_list_command => "/bin/rpm -qa --queryformat \"i | repos | %{name} | %{version}-%{release} | %{arch}\n\"";
#14

Updated by Dennis Cabooter over 6 years ago

Somehow, even when all my packages are installed, so rudder only needs to check once in a while, yum still gets executed every run. The only one rhel machine which is not (yet) in rudder is the only one never gets blacklisted. I find myself deleting and re-adding nodes at leat twice a week.

Proposal: It would be nice if all techniques only use yum to install/remove packages

#15

Updated by Jonathan CLARKE over 6 years ago

This is a pretty serious problem. I can imagine that this comes from running the package_list_command and package_patch_list_command regularly. It would make sense to use some variation of "rpm -qal" instead.

A related issue to this is #3490 where we are reworking all Techniques to only use one central package_method. Once that's done, we'll be able to clean it up and make sure that all Techniques get fixed by this.

#16

Updated by Dennis Cabooter over 6 years ago

Any update on this?

#17

Updated by Matthieu CERDA almost 6 years ago

  • Category set to Techniques
  • Status changed from Discussion to 8
  • Assignee changed from Dennis Cabooter to Matthieu CERDA
  • Target version set to 2.10.0

Taking back #3490, which is expected to fix this.

#18

Updated by Vincent MEMBRÉ almost 6 years ago

  • Target version changed from 2.10.0 to 2.11.0~beta1
#19

Updated by Nicolas PERRON almost 6 years ago

It is possible to prevent yum to be used at each agent execution. We have to use specific package_method like:

########################################################################
# Install a package using yum but with a check from rpm                #
########################################################################
body package_method rudder_yum
{
 package_changes => "bulk";
 package_list_command => "/bin/rpm -qa --qf '%{name} %{version}-%{release} %{arch}\n'";
 package_list_name_regex    => "^(\S+?)\s\S+?\s\S+$";
 package_list_version_regex => "^\S+?\s(\S+?)\s\S+$";
 package_list_arch_regex    => "^\S+?\s\S+?\s(\S+)$";
 package_installed_regex => ".*";
 package_name_convention => "$(name)";
 package_list_update_command => "/usr/bin/yum --quiet check-update";
 package_list_update_ifelapsed => "240";
 package_patch_installed_regex => "^\s.*";
 package_patch_name_regex    => "([^.]+).*";
 package_patch_version_regex => "[^\s]\s+([^\s]+).*";
 package_patch_arch_regex    => "[^.]+\.([^\s]+).*";
 package_add_command    => "/usr/bin/yum -y install";
 package_update_command => "/usr/bin/yum -y update";
 package_patch_command => "/usr/bin/yum -y update";
 package_delete_command => "/bin/rpm -e --nodeps --allmatches";
 package_verify_command => "/bin/rpm -V";

}

With this package_method, we have to specify at each use of type promise 'package' its use on RHEL...

Or another solution could be to impletement possibility to deactive RHN with the edition of the file /etc/yum/pluginconf.d/rhnplugin.conf:

[main]
enabled = 0

#20

Updated by Nicolas PERRON almost 6 years ago

Nicolas PERRON wrote:

It is possible to prevent yum to be used at each agent execution. We have to use specific package_method like:
[...]

With this package_method, we have to specify at each use of type promise 'package' its use on RHEL...

Or another solution could be to impletement possibility to deactive RHN with the edition of the file /etc/yum/pluginconf.d/rhnplugin.conf:
[...]

A part of the fix is #4891

#21

Updated by Nicolas PERRON almost 6 years ago

  • Status changed from 8 to Pending technical review
  • Assignee changed from Matthieu CERDA to Nicolas PERRON

Nicolas PERRON wrote:

Nicolas PERRON wrote:

It is possible to prevent yum to be used at each agent execution. We have to use specific package_method like:
[...]

With this package_method, we have to specify at each use of type promise 'package' its use on RHEL...

Or another solution could be to impletement possibility to deactive RHN with the edition of the file /etc/yum/pluginconf.d/rhnplugin.conf:
[...]

A part of the fix is #4891

Rollback...

Hereafter is the explanation of the solution I found.
Actually, the type pormises packages on RHEL are using yum, yum_rpm or generic and these package_method are :
- using yum for listing installed packages instead of rpm:

body package_method yum
{
[...]
  package_list_command => "/usr/bin/yum --quiet list installed";
[...]

- using a command to know if a list of patch are available:
body package_method yum_rpm
[...]
  package_patch_list_command => "/usr/bin/yum --quiet check-update";
[...]

- using yum for checking the updates of packages at each agent execution:
body package_method generic
{
[...]
 package_list_update_command => "/usr/bin/yum --quiet check-update";
 package_list_update_ifelapsed => "0";     # sometimes, caching is pretty disturbing
[...]

So, what seems logic was to add another package_method combining all of them with:
- The use of rpm to list the installed packages:

[...]
 package_list_command => "/bin/rpm -qa --queryformat \"i | repos | %{name} | %{version}-%{release} | %{arch}\n\"";
[...]

- removing the search of patch files with yum
- using a lock to prevent CFEngine to call yum at each execution:
      package_list_update_command => "/usr/bin/yum --quiet check-update";
      package_list_update_ifelapsed => "240";

I made a new package_method with these attributes:

########################################################################
# Install a package using yum but with a check from rpm                #
########################################################################
body package_method rudder_yum
{
 package_changes => "bulk";
 package_list_command => "/bin/rpm -qa --qf '%{name} %{version}-%{release} %{arch}\n'";
 package_list_name_regex    => "^(\S+?)\s\S+?\s\S+$";
 package_list_version_regex => "^\S+?\s(\S+?)\s\S+$";
 package_list_arch_regex    => "^\S+?\s\S+?\s(\S+)$";
 package_installed_regex => ".*";
 package_name_convention => "$(name)";
 package_list_update_command => "/usr/bin/yum --quiet check-update";
 package_list_update_ifelapsed => "240";
 package_patch_installed_regex => "^\s.*";
 package_patch_name_regex    => "([^.]+).*";
 package_patch_version_regex => "[^\s]\s+([^\s]+).*";
 package_patch_arch_regex    => "[^.]+\.([^\s]+).*";
 package_add_command    => "/usr/bin/yum -y install";
 package_update_command => "/usr/bin/yum -y update";
 package_patch_command => "/usr/bin/yum -y update";
 package_delete_command => "/bin/rpm -e --nodeps --allmatches";
 package_verify_command => "/bin/rpm -V";

}

This new package_method needs to be used on all Techniques that could call promises packages on RHEL. As first approach, I made the modifications on those Techniques:
- System
- Inventory
- NTP

Here is the PR URL: https://github.com/Normation/rudder-techniques/pull/380

#22

Updated by Nicolas PERRON almost 6 years ago

  • Assignee changed from Nicolas PERRON to Jonathan CLARKE
  • % Done changed from 0 to 100

To ensure that the whole problem is fixed we have to search for each call of package_methods on all Techniques and check that on RHEL, the yum,*yum_rpm* or generic package_method are replaced by rudder_yum.
For this, a new ticket will be created

#23

Updated by Nicolas PERRON almost 6 years ago

  • Target version changed from 2.11.0~beta1 to 2.6.13
  • Pull Request set to https://github.com/Normation/rudder-techniques/pull/380

Jon, could you review the PR (https://github.com/Normation/rudder-techniques/pull/380) please ?

#24

Updated by Nicolas PERRON almost 6 years ago

Nicolas PERRON wrote:

To ensure that the whole problem is fixed we have to search for each call of package_methods on all Techniques and check that on RHEL, the yum,*yum_rpm* or generic package_method are replaced by rudder_yum.
For this, a new ticket will be created

The new ticket to fix other Techniques is: #4892

#25

Updated by Nicolas PERRON almost 6 years ago

  • Status changed from Pending technical review to Pending release

Applied in changeset commit:0f902da0a6a5f074cf4afe1761fe3d4d67da4fe0.

#26

Updated by Jonathan CLARKE almost 6 years ago

Applied in changeset commit:34e25b0e19b0197323864be5c52fcf4a6da4cead.

#27

Updated by Vincent MEMBRÉ over 5 years ago

  • Subject changed from Rudder causes RHEL nodes to be blacklisted in RHN to Rudder causes RHEL nodes to be blacklisted in RHN due to intensive yum usage
#28

Updated by Vincent MEMBRÉ over 5 years ago

  • Status changed from Pending release to Released

This bug has been fixed in Rudder 2.6.13 (announcement , changelog), 2.9.5 (announcement , changelog) and 2.10.1 (announcement , changelog), which were released today.

#29

Updated by Benoît PECCATTE almost 5 years ago

  • Project changed from Techniques to Rudder
  • Category changed from Techniques to Techniques

Also available in: Atom PDF