Bug #3855
closed
Rudder causes RHEL nodes to be blacklisted in RHN due to intensive yum usage
Added by Dennis Cabooter over 11 years ago. Updated over 9 years ago.
Description
Twelve of our RHEL nodes are managed by Rudder. Since Rudder executes Yum many times a day, RHN blacklists that twelve nodes every week. We already tried to set "Section: Check interval" > "Check package statuses every (minutes): " to 60 minutes. However, this only slows down the problem. To "solve" this problem we have to unregister and reregister all twelve machines every week, which is quitte annoying.
- What exactly happens is described here: https://bugzilla.redhat.com/show_bug.cgi?id=824663.
- I can't find anything about this problem related to Cfengine. However, the one node not managed by Rudder doesn't have the problem at all.
- I found something about this problem related to Puppet: http://code.ohloh.net/file?fid=kzak1FGMdZUOw6az9Ka_k1whGic&cid=e5mmST0PD9o&s=&fp=274571&mpundefined=&projSelected=true#L0. However, it doesn't seem to be an official Puppet solution.
- We use a local RHEL yum mirror, so we can upgrade without using RHN. I'm not sure if disabling the RHN plugin is a solution for nodes registered to RHN without a local yum mirror.
- It would be nice to be able to have the RHN plugin enabled on a node while it's managed by Rudder. However, I haven't found a solution other then disabling it.
Updated by Dennis Cabooter over 11 years ago
Our /etc/yum/pluginconf.d/rhnplugin.conf looks like this for now:
[main] enabled = 0 gpgcheck = 1 [rhel-x86_64-server-5] enabled = 0 # You can specify options per channel, e.g.: # #[rhel-i386-server-5] #enabled = 1 # #[some-unsigned-custom-channel] #gpgcheck = 0
Updated by Matthieu CERDA over 11 years ago
- Status changed from New to Discussion
- Assignee set to Nicolas PERRON
- Priority changed from N/A to 1 (highest)
This is a serious issue, setting priority to 1.
Nicolas, I do not have a clear idea as of now about how to solve this, do you ?
Updated by Nicolas PERRON over 11 years ago
- Assignee changed from Nicolas PERRON to Dennis Cabooter
Matthieu CERDA wrote:
This is a serious issue, setting priority to 1.
Nicolas, I do not have a clear idea as of now about how to solve this, do you ?
The problem seems to be the several use of yum. In my opinion, the package manager does not need to be used more than one time a day.
I see that you've modified Check interval to 60 mins. Why don't you try to set it to 360 ?
Updated by Olivier Mauras over 11 years ago
So if one update his/her internal repos several times a day, new packages wouldn't be deployed straight ahead...
One would have to remove manually the /var/rudder/cfengine-community/state/software_packages.csv which defeats automation.
Updated by Vincent MEMBRÉ over 11 years ago
Maybe the interval could be defined for each package, overriding the default interval for that package ?
Updated by Nicolas PERRON over 11 years ago
Olivier Mauras wrote:
So if one update his/her internal repos several times a day, new packages wouldn't be deployed straight ahead...
It seems to me that the problem is the same manually. If soeone update the repository too many times, the blacklist will apply too. How should it work normally ?
Updated by Dennis Cabooter over 11 years ago
Olivier Mauras wrote:
So if one update his/her internal repos several times a day, new packages wouldn't be deployed straight ahead...
One would have to remove manually the /var/rudder/cfengine-community/state/software_packages.csv which defeats automation.
That's the problem indeed.
Updated by Olivier Mauras over 11 years ago
Nicolas PERRON wrote:
Olivier Mauras wrote:
So if one update his/her internal repos several times a day, new packages wouldn't be deployed straight ahead...
It seems to me that the problem is the same manually. If soeone update the repository too many times, the blacklist will apply too. How should it work normally ?
Well when not using RHEL you won't ever be blacklisted and you may still need to make a lot of changes to your internal repositories in a day - At least i do :)
Updated by Dennis Cabooter over 11 years ago
I am using a local mirror. However, disabling rhn causes rhn to list my nodes as out of date. There might be a solution in this topic: https://groups.google.com/forum/#!msg/help-cfengine/FaCPLJjj0mA/3N-vbSivkdgJ.
Updated by Dennis Cabooter over 11 years ago
Is Rudder using this to list packages?
body package_method yum_rpm # Contributed by Trond Hasle Amundsen # More efficient package method for RedHat - uses rpm to list instead of yum
Updated by Nicolas PERRON about 11 years ago
Dennis Cabooter wrote:
Is Rudder using this to list packages?
[...]
The package_list used by RHEL/CentOS is:
[...]
redhat::
package_list_command => "/usr/bin/yum list installed";
[...]
We were inspired by the body yum:
[...]
body package_method yum
{
package_changes => "bulk";
package_list_command => "/usr/bin/yum list installed";
[...]
If I understand well, the fix could be to use the body yum_rpm for listing packages ?
body package_method yum_rpm
# Contributed by Trond Hasle Amundsen
# More efficient package method for RedHat - uses rpm to list instead of yum
# Notes:
# - using $(name).$(arch) instead of $(name) for package_name_convention
# causes uninstallation to fail.
# - using allmatches to remove for all architectures
#
{
package_changes => "bulk";
package_list_command => "/bin/rpm -qa --qf '%{name} %{version}-%{release} %{arch}\n'";
[...]
Updated by Dennis Cabooter about 11 years ago
As far as I know a node be blacklisted in rhn classic if yum connects too many times. So if rudder just needs to check for a package if it's installed, it would be better to use rpm. The author of yum_rpm also says it's faster. :)
Updated by Dennis Cabooter about 11 years ago
For now I'm testing with a fork of the rpmPackageInstallation technique:
diff -Nru 5.0/rpmPackageInstallation.st 5.1/rpmPackageInstallation.st
--- 5.0/rpmPackageInstallation.st 2013-09-04 08:51:47.000000000 +0200
+++ 5.1/rpmPackageInstallation.st 2013-09-12 10:46:35.000000000 +0200
@@ -124,16 +124,27 @@
package_changes => "individual";
redhat::
- package_list_command => "/usr/bin/yum list installed";
- package_list_name_regex => "([^.]+).*";
- package_list_version_regex => "[^\s]\s+([^\s]+).*";
- package_list_arch_regex => "[^.]+\.([^\s]+).*";
- package_installed_regex => ".*installed.*";
- package_name_convention => "${name}";
+# package_list_command => "/usr/bin/yum list installed";
+# package_list_name_regex => "([^.]+).*";
+# package_list_version_regex => "[^\s]\s+([^\s]+).*";
+# package_list_arch_regex => "[^.]+\.([^\s]+).*";
+# package_installed_regex => ".*installed.*";
+# package_name_convention => "${name}";
+# package_list_update_ifelapsed => "$(rpm_pkg_timeout)";
+# package_add_command => "/usr/bin/yum -y install";
+# package_delete_command => "/bin/rpm -e";
+# package_verify_command => "/bin/rpm -V";
+
+ package_list_command => "/bin/rpm -qa --qf '%{name} %{version}-%{release} %{arch}\n'";
+ package_list_name_regex => "^(\S+?)\s\S+?\s\S+$";
+ package_list_version_regex => "^\S+?\s(\S+?)\s\S+$";
+ package_list_arch_regex => "^\S+?\s\S+?\s(\S+)$";
+ package_installed_regex => ".*";
+ package_name_convention => "$(name)";
package_list_update_ifelapsed => "$(rpm_pkg_timeout)";
- package_add_command => "/usr/bin/yum -y install";
- package_delete_command => "/bin/rpm -e";
- package_verify_command => "/bin/rpm -V";
+ package_add_command => "/usr/bin/yum -y install";
+ package_delete_command => "/bin/rpm -e --nodeps --allmatches";
+ package_verify_command => "/bin/rpm -V";
SuSE_11::
package_list_command => "/bin/rpm -qa --queryformat \"i | repos | %{name} | %{version}-%{release} | %{arch}\n\"";
Updated by Dennis Cabooter about 11 years ago
Somehow, even when all my packages are installed, so rudder only needs to check once in a while, yum still gets executed every run. The only one rhel machine which is not (yet) in rudder is the only one never gets blacklisted. I find myself deleting and re-adding nodes at leat twice a week.
Proposal: It would be nice if all techniques only use yum to install/remove packages
Updated by Jonathan CLARKE about 11 years ago
This is a pretty serious problem. I can imagine that this comes from running the package_list_command and package_patch_list_command regularly. It would make sense to use some variation of "rpm -qal" instead.
A related issue to this is #3490 where we are reworking all Techniques to only use one central package_method. Once that's done, we'll be able to clean it up and make sure that all Techniques get fixed by this.
Updated by Matthieu CERDA over 10 years ago
- Category set to Techniques
- Status changed from Discussion to 8
- Assignee changed from Dennis Cabooter to Matthieu CERDA
- Target version set to 2.10.0
Taking back #3490, which is expected to fix this.
Updated by Vincent MEMBRÉ over 10 years ago
- Target version changed from 2.10.0 to 2.11.0~beta1
Updated by Nicolas PERRON over 10 years ago
It is possible to prevent yum to be used at each agent execution. We have to use specific package_method like:
########################################################################
# Install a package using yum but with a check from rpm #
########################################################################
body package_method rudder_yum
{
package_changes => "bulk";
package_list_command => "/bin/rpm -qa --qf '%{name} %{version}-%{release} %{arch}\n'";
package_list_name_regex => "^(\S+?)\s\S+?\s\S+$";
package_list_version_regex => "^\S+?\s(\S+?)\s\S+$";
package_list_arch_regex => "^\S+?\s\S+?\s(\S+)$";
package_installed_regex => ".*";
package_name_convention => "$(name)";
package_list_update_command => "/usr/bin/yum --quiet check-update";
package_list_update_ifelapsed => "240";
package_patch_installed_regex => "^\s.*";
package_patch_name_regex => "([^.]+).*";
package_patch_version_regex => "[^\s]\s+([^\s]+).*";
package_patch_arch_regex => "[^.]+\.([^\s]+).*";
package_add_command => "/usr/bin/yum -y install";
package_update_command => "/usr/bin/yum -y update";
package_patch_command => "/usr/bin/yum -y update";
package_delete_command => "/bin/rpm -e --nodeps --allmatches";
package_verify_command => "/bin/rpm -V";
}
With this package_method, we have to specify at each use of type promise 'package' its use on RHEL...
Or another solution could be to impletement possibility to deactive RHN with the edition of the file /etc/yum/pluginconf.d/rhnplugin.conf:
[main] enabled = 0
Updated by Nicolas PERRON over 10 years ago
Nicolas PERRON wrote:
It is possible to prevent yum to be used at each agent execution. We have to use specific package_method like:
[...]With this package_method, we have to specify at each use of type promise 'package' its use on RHEL...
Or another solution could be to impletement possibility to deactive RHN with the edition of the file /etc/yum/pluginconf.d/rhnplugin.conf:
[...]
A part of the fix is #4891
Updated by Nicolas PERRON over 10 years ago
- Status changed from 8 to Pending technical review
- Assignee changed from Matthieu CERDA to Nicolas PERRON
Nicolas PERRON wrote:
Nicolas PERRON wrote:
It is possible to prevent yum to be used at each agent execution. We have to use specific package_method like:
[...]With this package_method, we have to specify at each use of type promise 'package' its use on RHEL...
Or another solution could be to impletement possibility to deactive RHN with the edition of the file /etc/yum/pluginconf.d/rhnplugin.conf:
[...]A part of the fix is #4891
Rollback...
Hereafter is the explanation of the solution I found.
Actually, the type pormises packages on RHEL are using yum, yum_rpm or generic and these package_method are :
- using yum for listing installed packages instead of rpm:
body package_method yum
{
[...]
package_list_command => "/usr/bin/yum --quiet list installed";
[...]
- using a command to know if a list of patch are available:
body package_method yum_rpm [...] package_patch_list_command => "/usr/bin/yum --quiet check-update"; [...]
- using yum for checking the updates of packages at each agent execution:
body package_method generic
{
[...]
package_list_update_command => "/usr/bin/yum --quiet check-update";
package_list_update_ifelapsed => "0"; # sometimes, caching is pretty disturbing
[...]
So, what seems logic was to add another package_method combining all of them with:
- The use of rpm to list the installed packages:
[...]
package_list_command => "/bin/rpm -qa --queryformat \"i | repos | %{name} | %{version}-%{release} | %{arch}\n\"";
[...]
- removing the search of patch files with yum
- using a lock to prevent CFEngine to call yum at each execution:
package_list_update_command => "/usr/bin/yum --quiet check-update";
package_list_update_ifelapsed => "240";
I made a new package_method with these attributes:
########################################################################
# Install a package using yum but with a check from rpm #
########################################################################
body package_method rudder_yum
{
package_changes => "bulk";
package_list_command => "/bin/rpm -qa --qf '%{name} %{version}-%{release} %{arch}\n'";
package_list_name_regex => "^(\S+?)\s\S+?\s\S+$";
package_list_version_regex => "^\S+?\s(\S+?)\s\S+$";
package_list_arch_regex => "^\S+?\s\S+?\s(\S+)$";
package_installed_regex => ".*";
package_name_convention => "$(name)";
package_list_update_command => "/usr/bin/yum --quiet check-update";
package_list_update_ifelapsed => "240";
package_patch_installed_regex => "^\s.*";
package_patch_name_regex => "([^.]+).*";
package_patch_version_regex => "[^\s]\s+([^\s]+).*";
package_patch_arch_regex => "[^.]+\.([^\s]+).*";
package_add_command => "/usr/bin/yum -y install";
package_update_command => "/usr/bin/yum -y update";
package_patch_command => "/usr/bin/yum -y update";
package_delete_command => "/bin/rpm -e --nodeps --allmatches";
package_verify_command => "/bin/rpm -V";
}
This new package_method needs to be used on all Techniques that could call promises packages on RHEL. As first approach, I made the modifications on those Techniques:
- System
- Inventory
- NTP
Here is the PR URL: https://github.com/Normation/rudder-techniques/pull/380
Updated by Nicolas PERRON over 10 years ago
- Assignee changed from Nicolas PERRON to Jonathan CLARKE
- % Done changed from 0 to 100
To ensure that the whole problem is fixed we have to search for each call of package_methods on all Techniques and check that on RHEL, the yum,*yum_rpm* or generic package_method are replaced by rudder_yum.
For this, a new ticket will be created
Updated by Nicolas PERRON over 10 years ago
- Target version changed from 2.11.0~beta1 to 2.6.13
- Pull Request set to https://github.com/Normation/rudder-techniques/pull/380
Jon, could you review the PR (https://github.com/Normation/rudder-techniques/pull/380) please ?
Updated by Nicolas PERRON over 10 years ago
Nicolas PERRON wrote:
To ensure that the whole problem is fixed we have to search for each call of package_methods on all Techniques and check that on RHEL, the yum,*yum_rpm* or generic package_method are replaced by rudder_yum.
For this, a new ticket will be created
The new ticket to fix other Techniques is: #4892
Updated by Nicolas PERRON over 10 years ago
- Status changed from Pending technical review to Pending release
Applied in changeset commit:0f902da0a6a5f074cf4afe1761fe3d4d67da4fe0.
Updated by Jonathan CLARKE over 10 years ago
Applied in changeset commit:34e25b0e19b0197323864be5c52fcf4a6da4cead.
Updated by Vincent MEMBRÉ over 10 years ago
- Subject changed from Rudder causes RHEL nodes to be blacklisted in RHN to Rudder causes RHEL nodes to be blacklisted in RHN due to intensive yum usage
Updated by Vincent MEMBRÉ over 10 years ago
- Status changed from Pending release to Released
This bug has been fixed in Rudder 2.6.13 (announcement , changelog), 2.9.5 (announcement , changelog) and 2.10.1 (announcement , changelog), which were released today.
- Download information: https://www.rudder-project.org/site/get-rudder/downloads/
Updated by Benoît PECCATTE over 9 years ago
- Project changed from 24 to Rudder
- Category changed from Techniques to Techniques