more useful options | |
PerlMonks |
Infrequent LWP::UserAgent 500 connect: Invalid argumentby superfrink (Curate) |
on Dec 29, 2006 at 22:53 UTC ( [id://592305]=perlquestion: print w/replies, xml ) | Need Help?? |
superfrink has asked for the wisdom of the Perl Monks concerning the following question:
I have a script that gets a file using HTTP with BasicAuth. Sometimes the GET fails with a code 500 even though the apache server logs show a code 200. I thought it was an ISP proxy issue. I added code to re-try the GET request when is_success() is not true. After that I sometimes found that on a rare occasion even though LWP reported a code 200 on the download the actual contents of the file would be: My next step was to create a separate file on the web server that contained the MD5 of the file I was trying to download. I updated the script to GET both files, calculate the MD5 of the file I am interested in and compare them. If the checksum does not match then retry both GET requests. This worked in general. Sometimes the MD5 file contained the "500 ..." error message but that was okay because the MD5 checksum calculated from the data file would never match the error message in the MD5 file. Still there are some times that the downloaded file is written to the disk containing the "500 ..." message. I added use LWP::Debug qw(+); and started running the script and grepping the downloaded files between runs. After about 5 runs I got a case where the file contents contain an error. (Each run downloads 12 files + 12 MD5 files => 24 files). The output for a file that was downloaded without error looks like:
A download were an error occured looks like:
In the case of an error debugging printed: but is_success() returned true. The part of the code that does the downloading and checks for errors is: Does this code fail to correctly check for a failed request? I am certain that the check_file_md5() routine works (at least sometimes) because I have seen the "MD5 sum mismatch error" string output from time to time. Somehow there are times where the MD5 checksum matched and yet the "500 ..." string is written to the output file. OH! Why do I do this line twice? I wonder if the different calls to content() are making different GET requests to the server. I will remove the second call so the MD5 is calculated on the same data written to the output file and see if the problem goes away. By the way I am running the script with: Perl 5.8.8 LWP::UserAgent version 2.033 OpenBSD 4.0 I ran ktrace and see that the connect() system call really is failing but I am not sure why.
Update: I made two changes to the code. I replaced with I also removed the second call to ->content(). After several more runs of the script I have not yet noticed the error message in an output file.
Back to
Seekers of Perl Wisdom
|
|