Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

Re: Best way to aggregate many silos of data into a search form

by moritz (Cardinal)
on Nov 30, 2011 at 20:19 UTC ( #940928=note: print w/ replies, xml ) Need Help??


in reply to Best way to aggregate many silos of data into a search form

The only way to gain simplicity is by glossing over lots of details. One way to do that is to request some HTML pages, and then scrape those for text, regardless of structure.

Of course that implies that you have a way to get all the (interesting) data out of each database in HTML form.


Comment on Re: Best way to aggregate many silos of data into a search form
Re^2: Best way to aggregate many silos of data into a search form
by MyMonkName (Acolyte) on Nov 30, 2011 at 23:12 UTC
    Screen scraping was my first approach, and frankly the one I am most comfortable with. But for the combination of net latency, the time it took for each of the databases to process the request, and the time for the script to execute, it was just too laggy. It would be fine for one or two sources I suppose, but we are talking about 6 or 7 databases, just for starters (!)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://940928]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (6)
As of 2014-12-20 14:23 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (96 votes), past polls