Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Script for a URL that constantly changes

by semrich (Initiate)
on Oct 19, 2011 at 21:52 UTC ( #932516=perlquestion: print w/ replies, xml ) Need Help??
semrich has asked for the wisdom of the Perl Monks concerning the following question:

Hey everybody, I'm new at Perl and I am trying to use this URL:

https://www.vwrsp.com/catalog/product/index.cgi?partnumber=$sku&sim_code=2.9&catalog_number=PAL1001&inE=1&highlight=PAL1001&reference_type=2

Where it says $sku, that will change constantly via a database, but I need a script that will change the "PAL1001" (catalog_number) each time $sku changes, so the $sku and catalog_number matches according to the site. Here is the script I used to insert the SKU I want from my database:

 my @sku = ('CODE TO THE DATABASE')

Hopefully this makes sense. Thanks in advance!

Comment on Script for a URL that constantly changes
Download Code
Re: Script for a URL that constantly changes
by anneli (Pilgrim) on Oct 19, 2011 at 23:03 UTC

      This is what I have so far:

      use strict; $|++; use Test::WWW::Mechanize; use WWW::Mechanize::Sleepy; use File::Basename; use WWW::Mechanize; use WWW::Mechanize::Image; use Storable; use HTTP::Cookies; use HTML::SimpleParse; my @sku = ('NUMBER'); for my $sku (@sku) { # sleep between 5 and 20 seconds between requests my $mech = WWW::Mechanize::Sleepy->new( sleep => '1..3' ); my $URL ="https://www.vwrsp.com/psearch/ControllerServlet.do?D=$sku&sp +age=header&CurSel=Ntt&Nty=1&Ntx=mode%2bmatchpartialmax&cntry=us&Ntk=A +ll&N=0&Ntt=$sku"; $mech->get( $URL ); $mech->success or die $mech->response->status_line +; $mech->success or die "post failed: "; my $url1= $mech->uri(); print "$url1\n"; my @links = $mech->find_all_links( url_regex => qr/\/catalog\/product/ +i); for my $links (@links) { $mech->get( $links->url() ); $mech->success or die $mech->response->status_line; $mech->success or die "post failed: "; my $pike = "|"; open (price_file, "FILE NAME") || die "can't open price.txt: $!\n"; my $some_html = $mech-> content(); my $p = new HTML::SimpleParse ($mech); print $p;

      What happens on the site is, you go to the URL listed above to get a list that matches the SKU fed to it, then from there each listing has it's own page where it holds all the information you see on the initial page (where the list is). When you go to that second page, in the URL there is a catalog number, that changes depending on which listing you clicked on (which is why I am not using that URL to parse), and I need to somehow capture that catalog number each time the SKU changes so I can get the correct information from the "second" page. I am not sure if that made any more sense. It is sort of complicated to explain.

Re: Script for a URL that constantly changes
by pvaldes (Chaplain) on Oct 19, 2011 at 23:57 UTC
    I'm trying to use this URL

    Define use...

    Where it says $sku, that will change constantly via a database

    Then you probably want to interact with the database in any way: use DBI

    my @sku = ('CODE TO THE DATABASE')

    Good try...

    my %code = 'CODE TO SOLVE YOUR PROBLEM HERE'; ;-)

    Aha, question updated with some real code, let's see...

    looks like a work for a hash... my @sku = ('NUMBER') is not what you want probably; first of all because @ mean a list of elements, not a single number, and also because a %hash is much better here. If you define @sku as an array you could have by mistake the same number for two products (and this is a BIG problem), but a hash will not permit you to do this. The number $sku is the key (unique) and the catalog ref is the value.

    so the first thing that you should try to fix is these two lines.

    my @sku = ('NUMBER'); for my $sku (@sku) {...

    Instead connect to the database and prepare something like: select field2 for mytable where field1 = ? and fill the gap with the correct number sku with execute($sku). Read the manual of DBI for the details

    finally when you have your hash you could write something like this

    https...partnumber=$sku...catalog_number=$hash{$sku}&inE=1&highlight...

      I have it connecting to the database like this:

      use DBI; use DBD::ODBC; my $conn_string = "driver={SQL Server};Server=SERVER NAME;Database=DB +NAME"; my $dbh = DBI->connect( "DBI:ODBC:$conn_string" ) or die $DBI::errstr; # get each SKU my ($sth, $stmt ); my $count=0; $stmt = qq { select sku from TABLE NAME }; $sth = $dbh->prepare ($stmt); $sth->execute ();
      I believe this is working, because I did a trial run of it. Do you have a suggestion as to where I can look up how to write a hash? Thank you for all your help!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://932516]
Approved by keszler
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (17)
As of 2014-12-18 21:01 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (64 votes), past polls