Generic front end with back-end drivers?

[toma]

I am looking for any modules or advice about how to create a generic front end interface with a dynamic back end. I am thinking about something like DBI/DBD.

From the DBI pod:

The DBI "dispatches" the method calls to the appropriate driver for actual execution. The DBI is also responsible for the dynamic loading of drivers, error checking and handling, providing default implementations for methods, and many other non-database specific duties.

I am looking for a module or advice about the dynamic loading, error checking, or any other advice about creating a generic interface that has multiple back-end drivers.

My particular problem is that I am extracting similar data from about 20 different file formats. Some of them are platform-specific. A few of the parsers are quite heavy, and I would like to use dynamic driver loading. My existing module autodetects the file format and includes a parser for each format. Now I need to add some new formats and I would like to avoid loading all the parsers all the time.

The module gets used by different applications on different platforms. Some file formats are platform-specific. Some file formats only provide a subset of all the possible data, and I will support a subset of the whole API.

I could just write a DBD driver for the various file formats, but I am concerned about violating rules #1 and #2 of the DBD driver development guide.

[Fletch]

You might look at something like Module::Pluggable for providing a plugin architecture, but unless you're planning on providing an SQL interface to your files (perhaps using DBD::AnyData and AnyData) you probably don't have enough common cause to use DBI for just its driver loading.

[samtregar]

This should be pretty easy to do. Put each parser in a separate package with a common interface. Then in your front-end class, do:

  if ($type eq 'foo') {
     $parser = "Parser::Foo";
  } elsif ($type eq 'bar') {
     $parser = "Parser::Bar";
  } else {
     die "What the heck is $type?";
  }

  eval "use $parser" or die "Can't load $parser: $@";

Then you make calls to your parser via the package name:

  $parser->do_that_thing($with_this);

-sam

[pc88mxer]

This sounds like a classic many-to-many situation, and so it seems that it would be helpful for you to develop an intermediate representation for the data in your files. That is, instead of going from a file directly to DBD, write modules which convert or adapt the file data to a common intermediate form. Then write another module which will store the results via DBD. That would make it easier if later you want to add additional output formats, like HTML or XML or JSON, or if you just want to process it differently.

One idea for the intermediate representation is just an array of hashes. This assumes that your text files contain a list of "records" which each record having "fields". Just write one converter for each file format that normalizes the data to this canonical structure.

Also, are you thinking of treating each file as a DBD database that would be accessed through the DBI API? That's the only time you'd need to create a new DBD driver, and I'm not sure that's really what you want to do.