Bug #3383
closed
Rudder status check fails if a proxy is defined in the environment
Description
I see the following error in /var/log/rudder/webapp/XXXX_XX_XX.stderrout.log and Jetty restarts all the time (so the web interface is unuseable for a little while):
[2013-04-02 16:01:24] ERROR non-compliant-reports - [2013-04-02 16:00:17+0200] N: root [rudder.example.com] S: [result_error] R: root-DP [distributePolicy] D: root-distributePolicy [Distribute Policy] T: distributePolicy/1.0 C: [Check rudder status] V: [None] The http://localhost:8080/rudder/api/status web application failed to respond for the second time. Restarting jetty NOW !
[2013-04-02 16:01:24] WARN non-compliant-reports - [2013-04-02 16:00:49+0200] N: root [rudder.example.com] S: [result_repaired] R: root-DP [distributePolicy] D: root-distributePolicy [Distribute Policy] T: distributePolicy/1.0 C: [Check jetty process] V: [None] The jetty process has been restarted
[2013-04-02 16:01:24] ERROR non-compliant-reports - [2013-04-02 16:00:49+0200] N: root [rudder.example.com] S: [result_error] R: root-DP [distributePolicy] D: root-distributePolicy [Distribute Policy] T: distributePolicy/1.0 C: [Check endpoint status] V: [None] The http://localhost:8080/endpoint/api/status web application failed to respond for the second time. Restarting jetty NOW !
When I invoke the command that has been used directly on the CLI, I get a strange answer:
root@winzts021:/tmp# curl -s http://localhost/rudder/api/status
<HTML>
<HEAD><TITLE>Redirection</TITLE></HEAD>
<BODY><H1>Redirect</H1></BODY>
</HTML>
Updated by Nicolas CHARLES over 11 years ago
Hi Dennis,
Did you change the context name of the application within Jetty ?
What is the output of the following command ? (note the :8080)
curl -s http://localhost:8080/endpoint/api/status
Updated by Dennis Cabooter over 11 years ago
What do you mean with context name?
root@winzts021:/tmp# curl -s http://localhost:8080/endpoint/api/status
<HTML>
<HEAD><TITLE>Redirection</TITLE></HEAD>
<BODY><H1>Redirect</H1></BODY>
</HTML>
Also Apache has errors in /var/log/rudder/apache2/error.log:
[Tue Apr 02 15:01:27 2013] [error] [client 10.122.245.14] (20014)Internal error: proxy: error reading status line from remote server localhost:8080, referer: http://rudder.wintershall.nl/rudder/secure/nodeManager/searchNodes
[Tue Apr 02 15:01:27 2013] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed
[Tue Apr 02 15:01:27 2013] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Tue Apr 02 15:01:37 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:01:47 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:01:57 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:02:07 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:02:17 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:02:20 2013] [error] proxy: HTTP: disabled connection for (localhost)
Updated by Dennis Cabooter over 11 years ago
Also from /var/log/rudder/apache2/error.log:
[Tue Apr 02 16:04:34 2013] [error] [client 10.122.245.14] client denied by server configuration: proxy:http://localhost:8080/rudder/api/status
Updated by Dennis Cabooter over 11 years ago
When running cf-agent on the server I see this:
-> Transformer /var/rudder/inventories/accepted-nodes-updates/winzts022-2013-03-27-14-41-31.ocs => /var/rudder/tools/send-clean.sh http://localhost:8080/endpoint/upload/ /var/rudder/inventories/accepted-nodes-updates/winzts022-2013-03-27-14-41-31.ocs /var/rudder/inventories/received/ returned error
Transforming: /var/rudder/tools/send-clean.sh http://localhost:8080/endpoint/upload/ /var/rudder/inventories/accepted-nodes-updates/winzts021-2013-03-26-11-25-05.ocs /var/rudder/inventories/received/
% Total % Received 10ferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 196k 0 0 100 196k 0 608k --:--:-- --:--:-- --:--:-- 660k
curl: (22) The requested URL returned error: 404
Updated by Matthieu CERDA over 11 years ago
Can you retry the exact same command, but with a:
unset http_proxy unset https_proxy unset ftp_proxy
You might have some proxy env variables interfering with cURL.
Updated by Dennis Cabooter over 11 years ago
Now it works! The proxy was in the way:
root@winzts021:/tmp# unset http_proxy
root@winzts021:/tmp# unset https_proxy
root@winzts021:/tmp# unset ftp_proxy
root@winzts021:/tmp# echo $http_proxy
root@winzts021:/tmp# curl -s http://localhost:8080/endpoint/api/status
OK
I've implemented the following workaround for now:
root@winzts021:/tmp# . /etc/bash.bashrc
root@winzts021:/tmp# echo $http_proxy
http://proxy.example.com:8080
root@winzts021:/tmp# echo "noproxy = *" > ~/.curlrc
root@winzts021:/tmp# curl -s http://localhost:8080/endpoint/api/status
OKroot@winzts021:/tmp#
However, it would be better to add "--noproxy \*" to all curl commands. I think the best thing to do for now is adding the workaround in the documentation.
Thank you for your help! :)
Updated by Jonathan CLARKE over 11 years ago
I'm glad this is working for you. I think we should build this into the base Rudder Techniques, and use curl's "--no-proxy" option, for the hostname that we're downloading from.
I see no reason why the Rudder agent should contact it's root server via a HTTP proxy... This wouldn't work for syslog or CFEngine network flows, so let's consider that we don't need it for HTTP either. Unless I'm really mistaken here?
Updated by Dennis Cabooter over 11 years ago
I think you can safely build this into the base Rudder Techniques. I see no reason for using a proxy to communicate with localhost. :)
Updated by Matthieu CERDA over 11 years ago
- Subject changed from Rudder status check to Rudder status check fails if a proxy is defined in the environment
- Category set to System techniques
- Status changed from New to 8
- Assignee set to Matthieu CERDA
- Priority changed from N/A to 1 (highest)
- Target version set to 2.3.12
So, this is a confirmed bug, and the workaround has been tested successfully (thanks Dennis), I'm on it.
I will document a way of using proxies with Rudder nodes too in the FaQ.
Updated by Matthieu CERDA over 11 years ago
- Status changed from 8 to Pending technical review
- Assignee changed from Matthieu CERDA to Jonathan CLARKE
- Pull Request set to https://github.com/Normation/rudder-techniques/pull/66
Technical review needed here !
Updated by Jonathan CLARKE over 11 years ago
- Status changed from Pending technical review to Pending release
Applied in changeset policy-templates:commit:0b9f245e0d83b1c49802ea817af35562b71b1904.
Updated by Jonathan CLARKE over 11 years ago
Just commited this fix. It'll be available in tonight's nightly builds (tomorrow morning). All you need to use it is to upgrade the rudder-techniques package, and copy the Techniques from /opt/rudder/share/techniques to /var/rudder/configuration-repository/techniques as shown in the docs.
Updated by Nicolas PERRON over 11 years ago
- Status changed from Pending release to Released