http://www.perlmonks.org?node_id=940933


in reply to Best way to aggregate many silos of data into a search form

My first question is always the 'who' -- who are we building the application for? From what you said:

The users of these databases are used mainly by librarian types who sift and sort to get the information they want via the aforementioned doodads...

It appears that the consumers of the disparate databases are the 'librarian types.' They "sift and sort to get the information they want" -- so my guess is that these people are your target.

Since the data is already in databases my recommendation is that you look into data warehousing. I do not recommend that you build a single application to connect to multiple back-end databases. Instead, I would obtain the advice of a knowledgeable data warehouse person and pay that individual to draw up a plan to build a single warehouse from the existing databases. I would then look into using whatever reporting tools my staff was fluent in to extract the data in meaningful ways for my target audience.

If and when I find that the reporting utilities of the data warehouse application overlap with those of the former applications I would deprecate the application functionality and promote the warehouse application's use. If you find that a large portion of functionality can be supplanted by the warehouse then I would schedule that application for end of life.

What you might notice after going through a few cycles of this is that almost all of the hard analytical work will be done by the data warehouse and/or the tools associated with it. Then the front-end applications will mainly be used for dashboards/data collection points. At that point it would be feasible to consolidate one or more of them into a single front-end application. You could use any of the numerous web application toolkits available in Perl to build such a thing.

Celebrate Intellectual Diversity