http://www.perlmonks.org?node_id=940928


in reply to Best way to aggregate many silos of data into a search form

The only way to gain simplicity is by glossing over lots of details. One way to do that is to request some HTML pages, and then scrape those for text, regardless of structure.

Of course that implies that you have a way to get all the (interesting) data out of each database in HTML form.

Replies are listed 'Best First'.
Re^2: Best way to aggregate many silos of data into a search form
by MyMonkName (Acolyte) on Nov 30, 2011 at 23:12 UTC
    Screen scraping was my first approach, and frankly the one I am most comfortable with. But for the combination of net latency, the time it took for each of the databases to process the request, and the time for the script to execute, it was just too laggy. It would be fine for one or two sources I suppose, but we are talking about 6 or 7 databases, just for starters (!)