Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Mimicing browser with LWP

by Anonymous Monk
on Jun 04, 2002 at 14:39 UTC ( #171488=perlquestion: print w/ replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

When submitting a particular form with netscape, I get a large number of results, but when submitting what I *think* is the same form with LWP, I get no results (the request succeeds, but the web server says "You must specify some information to search on."). The form is the "Search for Jobs" thing at the bottom of http://www.BrilliantPeople.com/jobsearch.asp I can with netscape select Information Technology, and leave the keyword field blank (I have javascript disabled, so it doesn't complain about this), and press the "Go ->" submit button. This works. But when I try to recreate the request with LWP:
sub searchreq($) { my $page = 0 + shift; return HTTP::Request->new(POST => "http://www.BrilliantPeople.com/jobsearch.asp", undef, # Search for Information Technologies stuff. "hdnOccupationSelectionCodes=".join(",",5118..5137) ."&txtKeywords=" #."&numberperpage=100". #."&currentpage=" . $page ); }
I get: "You must specify some information to search on." Any suggestions? The more I pound my head on this problem, the less like a BrilliantPerson I feel.

Comment on Mimicing browser with LWP
Download Code
Re: Mimicing browser with LWP
by Abigail-II (Bishop) on Jun 04, 2002 at 14:56 UTC
    If I leave the keyword fields blank, also with Netscape, I get the 'You must specify some information to search on' error. Did you try using LWP and filling out some keywords?

    Also, years ago, when I could still be bothered doing webbish stuff, I used a simple debugging proxy, just a handful of lines with with LWP and some networking code, that would dump all traffic to STDERR.

    Very useful if you want to know what exactly your browser or client is sending and receiving.

    Abigail

      When I ran it in netscape, I disabled javascript so that I could leave the keywords field blank.

      Anyway, adding keywords with LWP doesn't help any:

      When I use category Information Technology, and keyword programmer, netscape gives 75 matches, but with LWP it still says, "You must specify some information to search on."


      I guess your idea of writing a debugging proxy is probably what I'll need to figure this out. Do you have it? I could write my own, I guess, but it's nice to recycle :)

        I guess your idea of writing a debugging proxy is probably what I'll need to figure this out. Do you have it? I could write my own, I guess, but it's nice to recycle :)

        If I still have it somewhere (which I doubt), it's on a tape for which I currently don't have a drive to read.

        Abigail

        I prefer to use an ethernet sniffer for network protocol debugging. Look into tcpdump, which is free. There are also some very expensive but very good windows sniffers available. Iris is a particularly sexy one which comes to mind.
           MeowChow                                   
                       s aamecha.s a..a\u$&owag.print
Re: Mimicing browser with LWP
by Aristotle (Chancellor) on Jun 05, 2002 at 09:01 UTC
    You could make your life a bit simpler than constructing the post data yourself, using HTTP::Request::Common:
    return POST("http://www.BrilliantPeople.com/jobsearch.asp", [ undef, # Search for Information Technologies stuff. hdnOccupationSelectionCodes => join(",",5118..5137), txtKeywords => 'programmer', # numberperpage => 100, # currentpage => $page, ]);
    But that's just a sidenote. What I wanted to ask is did they change their site in the space of half a day? I can't find any form with a "hdnOccupationSelectionCodes" parameter there.

    Makeshifts last the longest.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://171488]
Approved by broquaint
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (12)
As of 2014-11-26 18:08 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (172 votes), past polls