Project

General

Profile

Actions
Copy link

Bug #3826

closed

curl command for download a file fails

Added by Fabrice FLORE-THÉBAULT over 10 years ago. Updated over 10 years ago.

Status:
Rejected
Priority:
2
Assignee:
Matthieu CERDA
Category:
Techniques
Target version:
2.4.9
Pull Request:
Severity:
UX impact:
User visibility:
Effort required:
Priority:
Name check:
Fix check:
Regression:

Description

The curl command actually generated to download the file fails.

Downloading the file with wget is correct.

  • rudder-agent-2.6.3.release-1.EL.5, Rudder v.2.6.3

Directive "Download a file"

  • URL to download file from: 
    http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar/from/http://cdn.mysql.com/
  • Absolute path to download file to on a node:
/root/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar

Execution of the directive

  • /opt/rudder/bin/cf-agent -KI | grep bundle.tar
 -> Executing '/usr/bin/curl -f --proxy '' -o "/root/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar" http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar/from/http://cdn.mysql.com/' ... (no timeout)
I: Last 3 quoted lines were generated by promiser "/usr/bin/curl -f --proxy '' -o "/root/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar" http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar/from/http://cdn.mysql.com/" 
 -> Completed execution of /usr/bin/curl -f --proxy '' -o "/root/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar" http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar/from/http://cdn.mysql.com/
R: @@downloadFile@@result_repaired@@d631b6be-d007-4db5-a04d-22527903bc92@@23a33c6c-afc4-4bd0-b37c-f97823c5986a@@2@@Download a file@@http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar/from/http://cdn.mysql.com/@@2013-08-08 14:25:02+02:00##59b4508b-95ae-4ced-9bb3-2ebab3f39362@#The file http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar/from/http://cdn.mysql.com/ has been successfully downloaded
  • ls -alh MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar
8,0K -rwxr----- 1 root root 200 aoû  8 14:25 MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar

the curl command doesn't work as expected

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   200  100   200    0     0    544      0 --:--:-- --:--:-- --:--:--     0

with wget it is working as expected

--2013-08-08 14:27:59--  http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar/from/http://cdn.mysql.com/
Résolution de dev.mysql.com... 137.254.60.11
Connexion vers dev.mysql.com|137.254.60.11|:80...connecté.
requête HTTP transmise, en attente de la réponse...302 Found
Emplacement: http://cdn.mysql.com/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar [suivant]
--2013-08-08 14:27:59--  http://cdn.mysql.com/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar
Résolution de cdn.mysql.com... 2.21.243.50, 23.62.99.17
Connexion vers cdn.mysql.com|2.21.243.50|:80...connecté.
requête HTTP transmise, en attente de la réponse...200 OK
Longueur: 311285760 (297M) [application/x-tar]
Saving to: `MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar.1'
Actions
Copy link
 #1

Updated by Vincent MEMBRÉ over 10 years ago

  • Assignee set to Matthieu CERDA
  • Priority changed from N/A to 2
  • Target version changed from 2.6.4 to 2.4.8

This is a quit annoiyign bug about that Technique.

It should happen in Rudder 2.4 too, and will be fixed in that version.

Mathieu or Nicolas can you look into this ?

Actions
Copy link
 #2

Updated by Nicolas PERRON over 10 years ago

  • Target version changed from 2.4.8 to 2.4.9
Actions
Copy link
 #3

Updated by Matthieu CERDA over 10 years ago

  • Status changed from New to Discussion
  • Assignee changed from Matthieu CERDA to Nicolas PERRON

I just tested with the updated URL:

/usr/bin/curl -f --proxy '' -o "/root/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar" http://cdn.mysql.com/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar

There is no problem, the file gets downloaded correctly.

I think that the MySQL site did funky things when dealing when some user agents like the "curl" one ...

Is there still a problem nowadays with that ?

NPE: Can you confirm that on your side it works too ?

Actions
Copy link
 #4

Updated by Nicolas PERRON over 10 years ago

  • Assignee changed from Nicolas PERRON to Matthieu CERDA

Matthieu CERDA wrote:

I just tested with the updated URL:
[...]

There is no problem, the file gets downloaded correctly.

I think that the MySQL site did funky things when dealing when some user agents like the "curl" one ...

Is there still a problem nowadays with that ?

NPE: Can you confirm that on your side it works too ?

The curl command works for me.... but the problem could be that it takes too many time. This is a file of 296M !

Actions
Copy link
 #5

Updated by Matthieu CERDA over 10 years ago

Le 20/09/2013 16:11, a écrit :

Issue #3826 has been updated by Nicolas PERRON.

  • Assignee changed from /Nicolas PERRON/ to /Matthieu CERDA/

NPE: Can you confirm that on your side it works too ?

The curl command works for me.... but the problem could be that it
takes too many time. This is a file of 296M !

It takes 18s on a machine in the Online datacenter, and approx. 5
minutes on a Free ADSL. Should not be a problem :)

Actions
Copy link
 #6

Updated by Matthieu CERDA over 10 years ago

  • Status changed from Discussion to In progress
  • Assignee changed from Matthieu CERDA to Nicolas PERRON

I'm testing this on an Orchestrator.

Actions
Copy link
 #7

Updated by Matthieu CERDA over 10 years ago

  • Status changed from In progress to Rejected
  • Assignee changed from Nicolas PERRON to Matthieu CERDA
  • % Done changed from 0 to 100

I confirm a Directive on Rudder 2.6 downlading "http://cdn.mysql.com/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar" works correctly:

R: @@Common@@log_info@@hasPolicyServer-root@@common-root@@5@@common@@StartRun@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#Start execution
R: @@Common@@log_repaired@@hasPolicyServer-root@@common-root@@5@@Update@@None@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#Node's policy (CFEngine promises) updated
R: @@Common@@result_repaired@@hasPolicyServer-root@@common-root@@5@@Update@@None@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#Policy or dependencies were updated or CFEngine service restarted
 !! Method invoked repairs
R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@5@@Security parameters@@None@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#The internal environment security is acceptable
R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@5@@Red Button@@None@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#Red Button is not in effect, continuing as normal...
R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@5@@Process checking@@None@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#There is an acceptable number of CFEngine processes running on the machine
R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@5@@CRON Daemon@@None@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#The CRON daemon is running
 -> Deleted file /var/rudder/cfengine-community/outputs/cf_debian_7_64_labo_normation_com__1377099387_Wed_Aug_21_17_36_27_2013_0x7f6177b6d700
R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@5@@Binaries update@@None@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#The CFengine binaries in /var/rudder/cfengine-community/bin are up to date
R: @@Common@@log_info@@hasPolicyServer-root@@common-root@@5@@Log system for reports@@None@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#Detected running syslog as rsyslog
R: @@Common@@result_success@@hasPolicyServer-root@@common-root@@5@@Log system for reports@@None@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#Logging system for report centralization is already correctly configured
 -> Executing '/usr/bin/curl -s -f --proxy '' -o "/var/rudder/tmp/uuid.txt" http://192.168.110.42/uuid' ... (no timeout)
 -> Completed execution of /usr/bin/curl -s -f --proxy '' -o "/var/rudder/tmp/uuid.txt" http://192.168.110.42/uuid
R: @@Inventory@@log_info@@inventory-all@@inventory-all@@2@@inventory@@None@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#An inventory was already sent less than 8 hours ago
R: @@Inventory@@result_success@@inventory-all@@inventory-all@@2@@inventory@@None@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#Next inventory scheduled between 00:00 and 06:00
 -> Executing '/usr/bin/curl -f --proxy '' -o "/tmp/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar" http://cdn.mysql.com/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar' ... (no timeout)
 32  296M   32 95.5M    0     0  1925k      0  0:02:36  0:00:49  0:01:47  989k
 : "...r/bin/curl -f -":   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Q: "...r/bin/curl -f -":                                  Dload  Upload   Total   Spent    Left  Speed
I: Last 3 quoted lines were generated by promiser "/usr/bin/curl -f --proxy '' -o "/tmp/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar" http://cdn.mysql.com/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar" 
 -> Completed execution of /usr/bin/curl -f --proxy '' -o "/tmp/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar" http://cdn.mysql.com/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar
R: @@downloadFile@@result_repaired@@e8ab81b9-0e8f-43e3-9ea6-5cb4f0b87713@@a9b0a947-13df-4188-90b3-70997c11b05b@@1@@Download a file@@http://cdn.mysql.com/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#The file http://cdn.mysql.com/Downloads/MySQL-5.6/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar has been successfully downloaded
R: @@MOTD@@result_success@@70ca49a2-7696-470e-b785-452230aabd78@@6a8f28c3-1bbb-4580-bc1f-e512dcb72ca1@@6@@MOTD Configuration@@None@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#The MOTD file was correct
 !! Method invoked repairs
 !! Method invoked repairs
 !! Method invoked repairs
R: @@Common@@log_info@@hasPolicyServer-root@@common-root@@5@@common@@EndRun@@2013-09-20 17:38:20+02:00##5ee53fbe-ff96-4606-a78b-5e408cbf77c9@#End execution

root@debian-7-64:/tmp# ls -hal MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar 
-rwxr--r-- 1 root root 297M sept. 20 17:41 MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar

root@debian-7-64:/tmp# tar tvf /tmp/MySQL-5.6.13-1.rhel5.x86_64.rpm-bundle.tar 
-rw-r--r-- bteam/staff 4560042 2013-07-12 14:04 MySQL-devel-5.6.13-1.rhel5.x86_64.rpm
-rw-r--r-- bteam/staff 5108409 2013-07-12 14:06 MySQL-shared-compat-5.6.13-1.rhel5.x86_64.rpm
-rw-r--r-- bteam/staff 113487964 2013-07-12 14:04 MySQL-embedded-5.6.13-1.rhel5.x86_64.rpm
-rw-r--r-- bteam/staff  89692969 2013-07-12 14:05 MySQL-server-5.6.13-1.rhel5.x86_64.rpm
-rw-r--r-- bteam/staff   2392023 2013-07-12 14:06 MySQL-shared-5.6.13-1.rhel5.x86_64.rpm
-rw-r--r-- bteam/staff  73082091 2013-07-12 14:06 MySQL-test-5.6.13-1.rhel5.x86_64.rpm
-rw-r--r-- bteam/staff  22953357 2013-07-12 14:04 MySQL-client-5.6.13-1.rhel5.x86_64.rpm

Rejecting this.

Actions
Copy link

Also available in: Atom PDF