User story #3944
closed
Prevent the /etc/cron.d/rudder-agent script from sending unsollicited e-mails
Description
Hello
We noticed that after rudder is installed inside a server this file is created:
cat /etc/cron.d/rudder-agent # Cron file for Rudder # # Will manually run cf-agent in case cf-execd is no longer running. cf-agent will fire up a new cf-execd. # # To temporarily avoid this behaviour, touch /opt/rudder/etc/disable-agent. # Don't forget to remove that file when you're done! 0,5,10,15,20,25,30,35,40,45,50,55 * * * * root if [ ! -e /opt/rudder/etc/disable-agent -a `ps -efww | grep -E "(cf-execd|cf-agent)" | grep -E "/var/rudder/cfengine-community/bin/(cf-execd|cf-agent)" | grep -v grep | wc -l` -eq 0 ]; then /var/rudder/cfengine-community/bin/cf-agent -f failsafe.cf && /var/rudder/cfengine-community/bin/cf-agent; fi
The problem is that the cron is sometime generating some output and a email is sent to administrator of that server. We would like to avoid this by appending : > /dev/null 2>&1 at the end of cron entry.
Unfortunately we didn't find a way to do it. We tried to do it by adding a technique to System settings-> System management->Cron daemon configuration -> and adding the appropriate content to the file , also by enforcing file content technique in Distributing files ->Enforce file content but each time the file was getting change each time the cron was running thus creating lots of files in /var/rudder/modified-files. Can you tell us which is the appropriate way to do this ?
The output of manually running the agent is similar to this
R: @@Common@@log_info@@hasPolicyServer-root@@common-root@@77@@common@@StartRun@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#Start execution R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@77@@Update@@None@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#Policy and dependencies already up to date. No action required. !! Method invoked repairs R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@77@@Security parameters@@None@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#The internal environment security is acceptable R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@77@@Red Button@@None@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#Red Button is not in effect, continuing as normal... -> Edited file /etc/cron.d/rudder-agent R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@77@@Process checking@@None@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#There is an acceptable number of CFEngine processes running on the machine R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@77@@CRON Daemon@@None@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#The CRON daemon is running R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@77@@Binaries update@@None@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#The CFengine binaries in /var/rudder/cfengine-community/bin are up to date R: @@Common@@log_info@@hasPolicyServer-root@@common-root@@77@@Log system for reports@@None@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#Detected running syslog as rsyslog R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@77@@Log system for reports@@None@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#Logging system for report centralization is already correctly configured -> Executing '/usr/bin/curl -s -f --proxy '' -o "/var/rudder/tmp/uuid.txt" http://1.2.3.4/uuid' ... (no timeout) -> Completed execution of /usr/bin/curl -s -f --proxy '' -o "/var/rudder/tmp/uuid.txt" http://1.2.3.4/uuid R: @@Inventory@@result_success@@inventory-all@@inventory-all@@72@@inventory@@None@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#Next inventory scheduled between 00:00 and 06:00 -> Edited file /etc/cron.d/rudder-agent Moved /etc/cron.d/rudder-agent_1379590921_Thu_Sep_19_12_42_02_2013.cf-before-edit to repository location /var/rudder/modified-files/_etc_cron_d_rudder_agent_1379590921_Thu_Sep_19_12_42_02_2013_cf_before_edit R: @@checkGenericFileContent@@result_repaired@@fa21718e-c620-4cd1-9937-820eb537cb2e@@e7425d44-39af-4ee2-ad1a-5530cec86098@@4@@File@@/etc/cron.d/rudder-agent@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#The file /etc/cron.d/rudder-agent was successfully updated R: @@checkGenericFileContent@@result_success@@fa21718e-c620-4cd1-9937-820eb537cb2e@@e7425d44-39af-4ee2-ad1a-5530cec86098@@4@@Line deletion regular expressions@@/etc/cron.d/rudder-agent@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#The file /etc/cron.d/rudder-agent was not set for any line deletion R: @@checkGenericFileContent@@result_success@@fa21718e-c620-4cd1-9937-820eb537cb2e@@e7425d44-39af-4ee2-ad1a-5530cec86098@@4@@Line replacement regular expressions@@/etc/cron.d/rudder-agent@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#The file /etc/cron.d/rudder-agent was not set for any line replacement R: @@checkGenericFileContent@@result_success@@fa21718e-c620-4cd1-9937-820eb537cb2e@@e7425d44-39af-4ee2-ad1a-5530cec86098@@4@@Permission adjustment@@/etc/cron.d/rudder-agent@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#The file /etc/cron.d/rudder-agent uses default permissions R: @@checkGenericFileContent@@result_success@@fa21718e-c620-4cd1-9937-820eb537cb2e@@e7425d44-39af-4ee2-ad1a-5530cec86098@@4@@Post-modification hook@@/etc/cron.d/rudder-agent@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#No command for /etc/cron.d/rudder-agent was to be executed R: @@Common@@log_info@@hasPolicyServer-root@@common-root@@77@@common@@EndRun@@2013-09-19 12:42:01+01:00##ee35116e-6a31-4416-9f68-d4c8a286b076@#End execution
Updated by Nicolas PERRON about 11 years ago
- Status changed from New to Discussion
- Target version set to 2.6.6
Hello Dan,
Thanks for your feedback !
The problem you're encounter is due to the fact that /etc/cron.d/rudder-agent is already managed by Rudder...but with what we call System Techniques.
In fact, you're trying to modify the cron entry /etc/cron.d/rudder-agent with the Technique "Cron management" which is a good try, but Rudder will try to recover the previous one. This is not convergent.
If you want to modify this cron, you'll need to modify System Techniques but for your need this is simple: We're using a template which is located on the server here: /var/rudder/configuration-repository/techniques/system/common/1.0/rudder_agent_community_cron.st.
Once this file will be modified, commited (/var/rudder/configuration-repository is a git repository) and reloaded in the WebUI (Administration>Policy Server>Reload Techniques), the next generation of promises will take your modification into account.
Are these informations what you asked for ?
Updated by Daniel Stan about 11 years ago
I have on extra question. If we change that file it is possible that it gets overwritten on a future rudder update?
Updated by Nicolas PERRON about 11 years ago
Daniel Stan wrote:
I have on extra question. If we change that file it is possible that it gets overwritten on a future rudder update?
You're absolutely right. System Techniques are "silently" upgraded unlike the other Techniques. If you update Rudder, your modification on it will be overwritten... but the solution could be to fix it upstream :)
What I understand of your problem is that a mail is sent by cron because of a relaunch of cf-agent but it could be a symptom of another problem. What do you know about these relaunch ? Are they regularly ?
Updated by Andrew Cranson about 11 years ago
What I understand of your problem is that a mail is sent by cron because of a relaunch of cf-agent but it could be a symptom of another problem. What do you know about these relaunch ? Are they regularly ?
Yes, this is correct. This could actually be for many reasons (these are Managed VPS's which we manage for our customers - they don't get root access) - for example, the VPS ran out of memory and the OOM killer decided to kill cfengine, or it was accidentally killed from host level, or was stopped by mistake, failed upgrade, not set to start at boot, etc. The problem is, when using Plesk Control Panel (which we provide to all customers) the root system mail is sent to the Plesk administrator (our customer, who has no root access) and they have idea what CFengine or Rudder are (and have no reason to know), can't fix it themselves and are scared by the email.
I'm sure you can imagine how scary the following subject looks to a non-technical user :)
Cron <root@hostname> if [ ! -e /opt/rudder/etc/disable-agent -a `ps -efww | grep -E \"(cf-execd|cf-agent)\" | grep -E \"/var/rudder/cfengine-community/bin/(cf-execd|cf-agent)\" | grep -v grep | wc -l` -eq 0 ]; then /var/rudder/cfengine-community/bin/c...
We simply want to append "> /dev/null 2>&1" to ensure these emails aren't sent - we can determine from rudder itself if an agent persistently isn't able to connect (though maybe some centralised list of these would be better than relying on the existing node list?).
What do you think?
Updated by Jonathan CLARKE about 11 years ago
- Assignee set to Matthieu CERDA
I think the best way to go for Rudder in the long run is to modify this cron script so that an email only gets sent if the cron failed to restart the daemons. So, if it did nothing, or successfully repaired the problem, it is silent. And if it does fail, we should of course replace this ugly message with a nice, user-friendly message (preferably configurable).
Andrew, as a quick workaround until we do that, you can edit /var/rudder/configuration-repository/techniques/system/common/1.0/rudder_agent_community_cron.st on your Rudder server, to add the > /dev/null, then cd to /var/rudder/configuration-repository/ and git commit that file. Then, go into the web interface and hit "Reload Techniques" in the Administration > Policy Server settings. All your nodes will update to the fixed cron file. BUT please be warned that on the next upgrade, your change will be overwritten.
Matthieu, can you fix this cron please?
Updated by Matthieu CERDA about 11 years ago
- Tracker changed from Question to Bug
- Subject changed from modifying /etc/cron.d/rudder-agent content to /etc/cron.d/rudder-agent sends unsollicited e-mails
- Status changed from Discussion to In progress
- Priority changed from N/A to 2
- Target version changed from 2.6.6 to 2.4.10
Aye aye. Grabbin' this.
Updated by Matthieu CERDA about 11 years ago
- Target version changed from 2.4.10 to 2.6.6
Oops, wrong version.
Updated by Matthieu CERDA about 11 years ago
- Status changed from In progress to Pending technical review
- Assignee changed from Matthieu CERDA to Jonathan CLARKE
- % Done changed from 0 to 100
- Pull Request set to https://github.com/Normation/rudder-techniques/pull/192
PR available.
Updated by Nicolas PERRON about 11 years ago
Matthieu CERDA wrote:
PR available.
It seems great to me !
Updated by Andrew Cranson about 11 years ago
Sorry, I wasn't 'watching' this.
I think the best way to go for Rudder in the long run is to modify this cron script so that an email only gets sent if the cron failed to restart the daemons. So, if it did nothing, or successfully repaired the problem, it is silent. And if it does fail, we should of course replace this ugly message with a nice, user-friendly message (preferably configurable).
Yes, this would be ok. However, it would be great if we could define the email address to send this to also (we should still prefer this message is sent to our monitoring emails rather than to our customers so we're sure we know about this failure).
I agree that it's better to notify only if this failed to restart the daemon.
Andrew, as a quick workaround until we do that, you can edit /var/rudder/configuration-repository/techniques/system/common/1.0/rudder_agent_community_cron.st on your Rudder server, to add the > /dev/null, then cd to /var/rudder/configuration-repository/ and git commit that file. Then, go into the web interface and hit "Reload Techniques" in the Administration > Policy Server settings. All your nodes will update to the fixed cron file. BUT please be warned that on the next upgrade, your change will be overwritten.
Thanks, we'll test with this today.
Updated by Jonathan CLARKE about 11 years ago
- Status changed from Pending technical review to Discussion
- Assignee changed from Jonathan CLARKE to Matthieu CERDA
Updated by Matthieu CERDA about 11 years ago
- Tracker changed from Bug to User story
- Subject changed from /etc/cron.d/rudder-agent sends unsollicited e-mails to Prevent the /etc/cron.d/rudder-agent script from sending unsollicited e-mails
- Status changed from Discussion to 14
- % Done changed from 100 to 50
An additional modification has to be done on the packaging. Moving this to a user story and creating sub-tickets
Updated by Matthieu CERDA about 11 years ago
- Status changed from 14 to 12
- Assignee changed from Matthieu CERDA to Jonathan CLARKE
Done, ready to review.
Updated by Jonathan CLARKE about 11 years ago
- Status changed from 12 to Pending release
Applied in changeset policy-templates:commit:4a6cd7d1637292e1759e929f34df1ae8699481b3.
Updated by Matthieu CERDA about 11 years ago
Applied in changeset packages:commit:e22019930ef045cabb3d550dbf112a31a1cb8797.
Updated by Jonathan CLARKE about 11 years ago
Applied in changeset packages:commit:acc03d98d5aa24479ff650cf35ffee0b42914cdf.
Updated by Nicolas PERRON about 11 years ago
- Status changed from Pending release to Released