Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

LWP script for Dice metro search form

by Plankton (Priest)
on Jan 08, 2004 at 20:30 UTC ( #319940=perlquestion: print w/ replies, xml ) Need Help??
Plankton has asked for the wisdom of the Perl Monks concerning the following question:

Friends,

I am trying to write a LWP script that I can use to download the search reults from Dice's Metor search page. I am not sure if my script has a problem or if Dice is just making it difficult (or impossible) for my script to work. Here's what I have so far.
#!/usr/bin/perl -w use strict; ... my $debug_level = 1; my $ACTIONURL="http://seeker.dice.com/jobsearch/servlet/JobSearch"; getopts( "U:d:h" ); if ( $opt_h ) { &usage(); exit 0; } if ( $opt_d ) { $debug_level = $opt_d; } my $browser; my $rc = do_LWPsetup( \$browser ); if ($rc==0) { die __FILE__ . "[" . __LINE__ . "] Failure: do_LWPStuff failed\n"; } do_getSearchResults ( $browser ); sub do_getSearchResults { my $browser = shift; my $resp = $browser->post("$ACTIONURL", [ 'op' => '1000', 'METRO_SEARCH' => '1', 'rel_code' => '1102', 'metro_id' => '1', 'secret' => '1', 'FREE_TEXT' => 'Perl', 'SKILLS' => "", 'RADIO' => 'AND', 'RADIO' => 'OR', 'RADIO' => 'BOOL', 'AREA_CODES' => '408', 'TAXTERM' => 'ALL', 'SORT_TYPE' => "", 'REQ_TRAVEL' => '1', 'TELECOMMUTE' => "", 'SORTSPEC' => '0', 'DAYSBACK' => '30', 'NUM_PER_PAGE' => '50', 'DETAILED_RESULTS' => '1', 'HOTRESULTS' => "" # 'Submit' => 'HOTRESULTS' ], 'Content_Type' => 'form-data' ); if (do_errorcheck($resp,"Search")) { return 0; } return 1; } ...
This is all I get ...
$ getdice.pl -d2 Begin HTML: End HTML

Can anyone give me a clue????
Thanks

Plankton: 1% Evil, 99% Hot Gas.

Comment on LWP script for Dice metro search form
Select or Download Code
Re: LWP script for Dice metro search form
by b10m (Vicar) on Jan 08, 2004 at 20:58 UTC
Re: LWP script for Dice metro search form
by Anonymous Monk on Jan 08, 2004 at 20:59 UTC

    http://seeker.dice.com/assets/docs/provider/terms_and_conditions.html

    CODE OF CONDUCT

    While using the Site or Site-related services, you agree not to do any of the following without our prior written authorization:
    ...
    • Use any search engine, software, tool, agent or other device or mechanism, including without limitation browsers, spiders, robots, avatars or intelligent agents (other than those made available by Dice on the Site or other generally available third party web browsers, e.g., Netscape Navigator or Microsoft Internet Explorer), to navigate or search the Site.
    ...
    • Use any robot, spider, site search/retrieval application, or other manual or automatic device or process to retrieve, index, "data mine," or in any way reproduce or circumvent the navigational structure or presentation of the Site or its contents.

    Anywho, the site sets a cookie. I don't know if the fields change but a couple were different from the fields you had listed. I'd recommend looking at Firebird with the Live HTTP Headers extension installed.

      Use any search engine, software, tool, agent or other device or mechanism, including without limitation browsers, spiders, robots, avatars or intelligent agents (other than those made available by Dice on the Site or other generally available third party web browsers, e.g., Netscape Navigator or Microsoft Internet Explorer), to navigate or search the Site.
      I'm no lawyer, but it seems from this statement as though you can only use a browser if it is 'generally available'. What is LWP but another browser? It is generally available -- anyone can download and use it if they like. Just because it doesn't have a GUI shouldn't change anything. Of course, I will never subject myself to the legal headache of trying to find out. Besides, I've never had great luck with dice anyway...

      bassplayer

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://319940]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (9)
As of 2014-11-23 14:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (132 votes), past polls