Project

General

Profile

Actions

Bug #3383

closed

Rudder status check fails if a proxy is defined in the environment

Added by Dennis Cabooter about 9 years ago. Updated about 9 years ago.

Status:
Released
Priority:
1
Category:
System techniques
Target version:
Severity:
UX impact:
User visibility:
Effort required:
Priority:

Description

I see the following error in /var/log/rudder/webapp/XXXX_XX_XX.stderrout.log and Jetty restarts all the time (so the web interface is unuseable for a little while):

[2013-04-02 16:01:24] ERROR non-compliant-reports - [2013-04-02 16:00:17+0200] N: root [rudder.example.com] S: [result_error] R: root-DP [distributePolicy] D: root-distributePolicy [Distribute Policy] T: distributePolicy/1.0 C: [Check rudder status] V: [None] The http://localhost:8080/rudder/api/status web application failed to respond for the second time. Restarting jetty NOW !
[2013-04-02 16:01:24] WARN non-compliant-reports - [2013-04-02 16:00:49+0200] N: root [rudder.example.com] S: [result_repaired] R: root-DP [distributePolicy] D: root-distributePolicy [Distribute Policy] T: distributePolicy/1.0 C: [Check jetty process] V: [None] The jetty process has been restarted
[2013-04-02 16:01:24] ERROR non-compliant-reports - [2013-04-02 16:00:49+0200] N: root [rudder.example.com] S: [result_error] R: root-DP [distributePolicy] D: root-distributePolicy [Distribute Policy] T: distributePolicy/1.0 C: [Check endpoint status] V: [None] The http://localhost:8080/endpoint/api/status web application failed to respond for the second time. Restarting jetty NOW !

When I invoke the command that has been used directly on the CLI, I get a strange answer:

root@winzts021:/tmp# curl -s http://localhost/rudder/api/status
<HTML>
<HEAD><TITLE>Redirection</TITLE></HEAD>
<BODY><H1>Redirect</H1></BODY>
</HTML>


Related issues 2 (0 open2 closed)

Related to Rudder - User story #3384: We should allow the user to specify a proxy to be used with the Rudder cURL callsRejectedFran├žois ARMANDActions
Related to Rudder - Bug #3401: The send-clean.sh script fails to send inventoriesReleasedMatthieu CERDA2013-04-09Actions
Actions #1

Updated by Nicolas CHARLES about 9 years ago

Hi Dennis,

Did you change the context name of the application within Jetty ?

What is the output of the following command ? (note the :8080)

curl -s http://localhost:8080/endpoint/api/status

Actions #2

Updated by Dennis Cabooter about 9 years ago

What do you mean with context name?

root@winzts021:/tmp# curl -s http://localhost:8080/endpoint/api/status
<HTML>
<HEAD><TITLE>Redirection</TITLE></HEAD>
<BODY><H1>Redirect</H1></BODY>
</HTML>

Also Apache has errors in /var/log/rudder/apache2/error.log:

[Tue Apr 02 15:01:27 2013] [error] [client 10.122.245.14] (20014)Internal error: proxy: error reading status line from remote server localhost:8080, referer: http://rudder.wintershall.nl/rudder/secure/nodeManager/searchNodes
[Tue Apr 02 15:01:27 2013] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed
[Tue Apr 02 15:01:27 2013] [error] ap_proxy_connect_backend disabling worker for (localhost)
[Tue Apr 02 15:01:37 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:01:47 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:01:57 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:02:07 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:02:17 2013] [error] proxy: HTTP: disabled connection for (localhost)
[Tue Apr 02 15:02:20 2013] [error] proxy: HTTP: disabled connection for (localhost)

Actions #3

Updated by Dennis Cabooter about 9 years ago

Also from /var/log/rudder/apache2/error.log:

[Tue Apr 02 16:04:34 2013] [error] [client 10.122.245.14] client denied by server configuration: proxy:http://localhost:8080/rudder/api/status

Actions #4

Updated by Dennis Cabooter about 9 years ago

When running cf-agent on the server I see this:

-> Transformer /var/rudder/inventories/accepted-nodes-updates/winzts022-2013-03-27-14-41-31.ocs => /var/rudder/tools/send-clean.sh http://localhost:8080/endpoint/upload/ /var/rudder/inventories/accepted-nodes-updates/winzts022-2013-03-27-14-41-31.ocs /var/rudder/inventories/received/ returned error
Transforming: /var/rudder/tools/send-clean.sh http://localhost:8080/endpoint/upload/ /var/rudder/inventories/accepted-nodes-updates/winzts021-2013-03-26-11-25-05.ocs /var/rudder/inventories/received/
  % Total    % Received 10ferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  196k    0     0  100  196k      0   608k --:--:-- --:--:-- --:--:--  660k
curl: (22) The requested URL returned error: 404
Actions #5

Updated by Matthieu CERDA about 9 years ago

Can you retry the exact same command, but with a:

unset http_proxy
unset https_proxy
unset ftp_proxy

You might have some proxy env variables interfering with cURL.

Actions #6

Updated by Dennis Cabooter about 9 years ago

Now it works! The proxy was in the way:

root@winzts021:/tmp# unset http_proxy
root@winzts021:/tmp# unset https_proxy
root@winzts021:/tmp# unset ftp_proxy
root@winzts021:/tmp# echo $http_proxy
root@winzts021:/tmp# curl -s http://localhost:8080/endpoint/api/status
OK

I've implemented the following workaround for now:

root@winzts021:/tmp# . /etc/bash.bashrc
root@winzts021:/tmp# echo $http_proxy
http://proxy.example.com:8080
root@winzts021:/tmp# echo "noproxy = *" > ~/.curlrc
root@winzts021:/tmp# curl -s http://localhost:8080/endpoint/api/status
OKroot@winzts021:/tmp#

However, it would be better to add "--noproxy \*" to all curl commands. I think the best thing to do for now is adding the workaround in the documentation.

Thank you for your help! :)

Actions #7

Updated by Jonathan CLARKE about 9 years ago

I'm glad this is working for you. I think we should build this into the base Rudder Techniques, and use curl's "--no-proxy" option, for the hostname that we're downloading from.

I see no reason why the Rudder agent should contact it's root server via a HTTP proxy... This wouldn't work for syslog or CFEngine network flows, so let's consider that we don't need it for HTTP either. Unless I'm really mistaken here?

Actions #8

Updated by Dennis Cabooter about 9 years ago

I think you can safely build this into the base Rudder Techniques. I see no reason for using a proxy to communicate with localhost. :)

Actions #9

Updated by Matthieu CERDA about 9 years ago

  • Subject changed from Rudder status check to Rudder status check fails if a proxy is defined in the environment
  • Category set to System techniques
  • Status changed from New to 8
  • Assignee set to Matthieu CERDA
  • Priority changed from N/A to 1
  • Target version set to 2.3.12

So, this is a confirmed bug, and the workaround has been tested successfully (thanks Dennis), I'm on it.

I will document a way of using proxies with Rudder nodes too in the FaQ.

Actions #10

Updated by Matthieu CERDA about 9 years ago

  • Status changed from 8 to Pending technical review
  • Assignee changed from Matthieu CERDA to Jonathan CLARKE
  • Pull Request set to https://github.com/Normation/rudder-techniques/pull/66
Actions #11

Updated by Matthieu CERDA about 9 years ago

  • % Done changed from 0 to 100
Actions #12

Updated by Jonathan CLARKE about 9 years ago

  • Status changed from Pending technical review to Pending release

Applied in changeset policy-templates:commit:0b9f245e0d83b1c49802ea817af35562b71b1904.

Actions #13

Updated by Jonathan CLARKE about 9 years ago

Just commited this fix. It'll be available in tonight's nightly builds (tomorrow morning). All you need to use it is to upgrade the rudder-techniques package, and copy the Techniques from /opt/rudder/share/techniques to /var/rudder/configuration-repository/techniques as shown in the docs.

Actions #14

Updated by Nicolas PERRON about 9 years ago

  • Status changed from Pending release to Released
Actions

Also available in: Atom PDF