http://www.perlmonks.org?node_id=479049

dpmott has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to use LWP::UserAgent to automate retrieval of a switch database on a 3Com 3300 switch.

I can view the switch database via the HTTP interface with IE. I can even telnet to the device on port 80 and provide the appropriate HTTP directives to retrieve the HTML representation of the database. The transfer is slow, but appears to be stable.

However, when I use LWP::UserAgent, I get the following error (via $response->status_line):
"500 Line too long (limit is 4096)"

A packet sniffer shows that I get the first 31 or so TCP packets, and then LWP appears to close the TCP connection. A similiar trace of the exchange with IE behind the wheel shows over 140 TCP packets to convey the entire page.

I've tried setting attributes on the $ua object via the max_size() and timeout() methods to no avail. I've attempted to use the ':content_cb' => sub{} key/value pair in the get() method to capture the data, but no data comes in before LWP errors out. I've attempted to use the mirror() method, but I get the same error and no data. I've also tried specifying 'keep_alive' => 1 in the LWP::UserAgent constructor, but I didn't see any change in behavior.

I'm at a loss. Could anyone explain to me why LWP would throw this error, and how I could coerce it to work for me?

Here is the basic framwork of code that I'm using:

#!perl use strict; use LWP::UserAgent; use constant URL => 'http://east/dev01/sd/action_sd.rscr'; my $ua = LWP::UserAgent->new(); $ua->credentials( 'east:80', 'device, '****', '****' ); my $response = $ua->get(URL); if ($response->is_success) { print $response->content; } else { die $response->status_line; }
Thanks,
-dpmott


Update: The problem appears to be multi-faceted.
First, the 3Com 3300 series switch is not sending back HTTP headers. This makes it an HTTP 0.9 or prior HTTP server, or worse a non-compliant HTTP server.
Second, the Net::HTTP module (from which LWP::Protocol::http inherits) has a couple of paragraphs for its read_response_headers() method which talk about how it handles non-compliant servers.
The LWP::Protocol::http class passes "'laxed'=>1" to that interface, but that is apparently not by itself enough to manage a response that is > 4k and lacking HTTP response headers. Thus, the solution of setting the MaxLineLength flag in the LWP::Protocol::http::EXTRA_SOCK_OPTS array is actually the correct and (as far as I can tell) only solution to this problem. That flag gets passed into the Net::HTTP constructor and serves to keep the module from dropping an HTTP response due to missing HTTP headers.


Replies are listed 'Best First'.
Re: Mysterious LWP status 500 Line too long
by puploki (Hermit) on Jul 28, 2005 at 17:44 UTC
    Sorry I can't help with your exact LWP problem, but is there a reason why you're querying the Web UI of the 3300?

    From experience SNMP is a much better way of obtaining information from networking equipment. Generally the mac address database/FDB is in the same place in the tree, however you could just look up the manufacturer's OID to find it and query that using Net::SNMP

    I know some networks groups get a bit twitchy (well, mine does at least!) about SNMP so maybe it's not a solution for you :(

      Someone else in the office was directed to extract the data, and was shown the HTTP interface. When that didn't prove to be straightforward, I was consulted as the resident perl guru to explain why LWP wasn't working as expected.

      Now that you've made the suggestion, though, it seems like the smarter way to go. Thanks for the suggestion :)

      -dpmott
Re: Mysterious LWP status 500 Line too long
by jhourcle (Prior) on Jul 28, 2005 at 19:48 UTC

    From what you've described, the web server you're connecting to doesn't speak HTTP/1.x. You'll need to find code that's meant to deal with HTTP/0.9.

    You might want to take a look at Web Client Programming with Perl. (circa 1997)

    (or try to find a copy of libwww-perl, from before version 1)

Re: Mysterious LWP status 500 Line too long
by ikegami (Patriarch) on Jul 28, 2005 at 17:42 UTC

    I'm not convinced making the limit bigger would help. Net::HTTP thinks a *header* line is longer than 4k. The switch shouldn't be returning a header line that big. Is it really doing that?

    I think the following hack would make the limit bigger:

    %OPTS = @LWP::Protocol::http::EXTRA_SOCK_OPTS; $OPTS{MaxLineLength} = ...; @LWP::Protocol::http::EXTRA_SOCK_OPTS = %OPTS;
      Hey, thanks, that works! At least, when I specify a limit of 64k.

      I took a closer look at the response from the switch. The beginning of it looks like this:

      <HTML><BODY BGCOLOR=#ffffff>
      Note that there are *no* <HEAD> tags in there. Would that be causing this problem?

      Thanks,
      -dpmott

      Update: There is no HTTP Header in the response, either. The response from the switch begins with the <HTML> tag

        The problem is with the header of the HTTP request. That's the body. An HTTP response looks like:

        HTTP/1.0 200 OK<CR><LF> Header: Value<CR><LF> Header: Value<CR><LF> Header: Value<CR><LF> <CR><LF> body

        The blank line seperates the header from the body.

        By the way, I think 0 means no limit.