Bug #3383
closed
Rudder status check fails if a proxy is defined in the environment
Added by Dennis Cabooter over 11 years ago.
Updated over 11 years ago.
Category:
System techniques
Description
I see the following error in /var/log/rudder/webapp/XXXX_XX_XX.stderrout.log and Jetty restarts all the time (so the web interface is unuseable for a little while):
[2013-04-02 16:01:24] ERROR non-compliant-reports - [2013-04-02 16:00:17+0200] N: root [rudder.example.com] S: [result_error] R: root-DP [distributePolicy] D: root-distributePolicy [Distribute Policy] T: distributePolicy/1.0 C: [Check rudder status] V: [None] The http://localhost:8080/rudder/api/status web application failed to respond for the second time. Restarting jetty NOW !
[2013-04-02 16:01:24] WARN non-compliant-reports - [2013-04-02 16:00:49+0200] N: root [rudder.example.com] S: [result_repaired] R: root-DP [distributePolicy] D: root-distributePolicy [Distribute Policy] T: distributePolicy/1.0 C: [Check jetty process] V: [None] The jetty process has been restarted
[2013-04-02 16:01:24] ERROR non-compliant-reports - [2013-04-02 16:00:49+0200] N: root [rudder.example.com] S: [result_error] R: root-DP [distributePolicy] D: root-distributePolicy [Distribute Policy] T: distributePolicy/1.0 C: [Check endpoint status] V: [None] The http://localhost:8080/endpoint/api/status web application failed to respond for the second time. Restarting jetty NOW !
When I invoke the command that has been used directly on the CLI, I get a strange answer:
root@winzts021:/tmp# curl -s http://localhost/rudder/api/status
<HTML>
<HEAD><TITLE>Redirection</TITLE></HEAD>
<BODY><H1>Redirect</H1></BODY>
</HTML>
Hi Dennis,
Did you change the context name of the application within Jetty ?
What is the output of the following command ? (note the :8080)
curl -s http://localhost:8080/endpoint/api/status
What do you mean with context name?
root@winzts021:/tmp# curl -s http://localhost:8080/endpoint/api/status
<HTML>
<HEAD><TITLE>Redirection</TITLE></HEAD>
<BODY><H1>Redirect</H1></BODY>
</HTML>
Also Apache has errors in /var/log/rudder/apache2/error.log:
[Tue Apr 02 15:01:27 2013] [error] [client 10.122.245.14] (20014)Internal error: proxy: error reading status line from remote server localhost:8080, referer: http://rudder.wintershall.nl/rudder/secure/nodeManager/searchNodes
[Tue Apr 02 15:01:27 2013] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed
[Tue Apr 02 15:01:27 2013] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Tue Apr 02 15:01:37 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:01:47 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:01:57 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:02:07 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:02:17 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:02:20 2013] [error] proxy: HTTP: disabled connection for (localhost)
Also from /var/log/rudder/apache2/error.log:
[Tue Apr 02 16:04:34 2013] [error] [client 10.122.245.14] client denied by server configuration: proxy:http://localhost:8080/rudder/api/status
When running cf-agent on the server I see this:
-> Transformer /var/rudder/inventories/accepted-nodes-updates/winzts022-2013-03-27-14-41-31.ocs => /var/rudder/tools/send-clean.sh http://localhost:8080/endpoint/upload/ /var/rudder/inventories/accepted-nodes-updates/winzts022-2013-03-27-14-41-31.ocs /var/rudder/inventories/received/ returned error
Transforming: /var/rudder/tools/send-clean.sh http://localhost:8080/endpoint/upload/ /var/rudder/inventories/accepted-nodes-updates/winzts021-2013-03-26-11-25-05.ocs /var/rudder/inventories/received/
% Total % Received 10ferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 196k 0 0 100 196k 0 608k --:--:-- --:--:-- --:--:-- 660k
curl: (22) The requested URL returned error: 404
Can you retry the exact same command, but with a:
unset http_proxy
unset https_proxy
unset ftp_proxy
You might have some proxy env variables interfering with cURL.
Now it works! The proxy was in the way:
root@winzts021:/tmp# unset http_proxy
root@winzts021:/tmp# unset https_proxy
root@winzts021:/tmp# unset ftp_proxy
root@winzts021:/tmp# echo $http_proxy
root@winzts021:/tmp# curl -s http://localhost:8080/endpoint/api/status
OK
I've implemented the following workaround for now:
root@winzts021:/tmp# . /etc/bash.bashrc
root@winzts021:/tmp# echo $http_proxy
http://proxy.example.com:8080
root@winzts021:/tmp# echo "noproxy = *" > ~/.curlrc
root@winzts021:/tmp# curl -s http://localhost:8080/endpoint/api/status
OKroot@winzts021:/tmp#
However, it would be better to add "--noproxy \*" to all curl commands. I think the best thing to do for now is adding the workaround in the documentation.
Thank you for your help! :)
I'm glad this is working for you. I think we should build this into the base Rudder Techniques, and use curl's "--no-proxy" option, for the hostname that we're downloading from.
I see no reason why the Rudder agent should contact it's root server via a HTTP proxy... This wouldn't work for syslog or CFEngine network flows, so let's consider that we don't need it for HTTP either. Unless I'm really mistaken here?
I think you can safely build this into the base Rudder Techniques. I see no reason for using a proxy to communicate with localhost. :)
- Subject changed from Rudder status check to Rudder status check fails if a proxy is defined in the environment
- Category set to System techniques
- Status changed from New to 8
- Assignee set to Matthieu CERDA
- Priority changed from N/A to 1 (highest)
- Target version set to 2.3.12
So, this is a confirmed bug, and the workaround has been tested successfully (thanks Dennis), I'm on it.
I will document a way of using proxies with Rudder nodes too in the FaQ.
- Status changed from 8 to Pending technical review
- Assignee changed from Matthieu CERDA to Jonathan CLARKE
- Pull Request set to https://github.com/Normation/rudder-techniques/pull/66
- % Done changed from 0 to 100
- Status changed from Pending technical review to Pending release
Applied in changeset policy-templates:commit:0b9f245e0d83b1c49802ea817af35562b71b1904.
Just commited this fix. It'll be available in tonight's nightly builds (tomorrow morning). All you need to use it is to upgrade the rudder-techniques package, and copy the Techniques from /opt/rudder/share/techniques to /var/rudder/configuration-repository/techniques as shown in the docs.
- Status changed from Pending release to Released
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by Matthieu CERDA 
Updated by Jonathan CLARKE 
Updated by Nicolas PERRON