Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??
Hi Neil,
I am having the very same problem. I was trying to run WWW::Mechanize through privoxy, which in turn was forwarding everything to Tor so I could run my scripts anonymously. I set everything up on my Linux machine and was able to confirm that it was working when I used Firefox and Privoxy to check my Tor status at...
status

When I then used my script, I kept getting error message...
500 Chunked must be last Transfer-Encoding 'identity'
Here's my program...
<tor_test.pl>
#!/usr/bin/perl -w use strict; use WWW::Mechanize; use HTTP::Cookies; # this script will test to see how WWW::Mechanize works with Tor sub main { my $cookie_jar = HTTP::Cookies->new( file => 'cookies.dat' +, autosave => 1, hide_cookie2 => 1 ); my $bot = WWW::Mechanize->new; $bot->max_redirect(100); $bot->cookie_jar($cookie_jar); $bot->add_header(Accept => 'text/xml,application/xml,application/x +html+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'); $bot->add_header('Accept-Language' => 'en-us,en;q=0.5'); $bot->add_header('Accept-Charset' => 'ISO-8859-1,utf-8;q=0.7,*;q=0 +.7'); $bot->add_header('Cache-Control' => 'max-age=0'); # port 8118 for privoxy $bot->proxy('http', 'http://127.0.0.1:8118'); $bot->agent('Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1. +8.0.3) Gecko/20060426 Firefox/1.5.0.3'); $bot->timeout(600); $bot->stack_depth("3"); my $url = 'http://serifos.eecs.harvard.edu/cgi-bin/ipaddr.pl?tor=1 +'; my $response = $bot->get($url); my $content = $bot->content; print ("$content"); print ("fin"); } &main;
</tor_test.pl>

The result was...
500 Chunked must be last Transfer-Encoding 'identity'
Initially, I thought that the problem was due to the fact that the timeout was not set long enough, but after setting the timeout to a range of values from small to very large, I still get the same problem. I also noticed that when I stepped through the code, the timeout did not seem to have any impact on how quickly the 500 was generated (instantaously). So, I edited the privoxy config file to increase logging.
<privoxy config>
debug 16
</privoxy config>

Now I restart privoxy and "tail -f /var/log/privoxy/logfile"
<privoxy logfile>
Aug 30 13:00:37 Privoxy(-1208476752) Request: serifos.eecs.harvard.edu +/cgi-bin/ipaddr.pl?tor=1 Aug 30 13:00:37 Privoxy(-1208476752) Writing: Aug 30 13:00:38 Privoxy( +-1208476752) Writing: GET /cgi-bin/ipaddr.pl?tor=1 HTTP/1.1 Cache-Control: max-age=0 Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9 +,text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Accept-Language: en-us,en;q=0.5 Host: serifos.eecs.harvard.edu User-Agent: Mozilla (X11; I; Linux 2.0.32 i586) Connection: close Aug 30 13:00:38 Privoxy(-1208476752) Writing: HTTP/1.1 200 OK Date: Wed, 30 Aug 2006 20:00:38 GMT Server: Apache/1.3.34 (Debian) Transfer-Encoding: identity Content-Type: text/html; charset=iso-8859-1 Connection: close Aug 30 13:00:38 Privoxy(-1208476752) Writing: <!doctype html public "- +//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html> <head><script>function PrivoxyWindowOpen(){return(null);}</script> <title>Tor Test Results</title> <meta name="Author" content="Geoffrey Goodell"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta http-equiv="Content-Style-Type" content="text/css"> <link rel="stylesheet" type="text/css" href="http://serifos.eecs.harva +rd.edu/style.css"> </head> <body> You seem to be using Tor! <p>You connected to this site from <b>140.247.62.119</b>, which is a v +alid <a href="http://tor.eff.org/">Tor</a> exit node named <b>serifos</b>. Congratulations!</p> </body>

</privoxy logfile>

So, you can see that I am getting a response from the remote server and everything is working!! But for some reason WWW::Mechanize doesn't like the response from Privoxy and issues the 500 error rather than accept the results. I grep'd the perl code and found several references...

find /usr/lib/perl5 -exec grep -H 'Transfer-Encoding' '{}' \;
...and this seems to be the line where it is choking...
/usr/lib/perl5/vendor_perl/5.8.5/Net/HTTP/Methods.pm: die "Chunked must be last Transfer-Encoding '$te'"

I haven't gotten any further on this problem, if someone else can suggest something, I'd be very appreciative!!

Edited by planetscape - linkified link and changed pre to code tags

( keep:0 edit:12 reap:0 )


In reply to Re^3: User agent through Privoxy? by Anonymous Monk
in thread User agent through Privoxy? by neilwatson

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others about the Monastery: (18)
    As of 2015-07-07 17:17 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









      Results (92 votes), past polls