Re^2: MedlineParser: to parse and load MEDLINE into a RDBMS

[haricothoriz]

To address your concern about efficient data import: both the java and perl programs have options to generate flat-file representations of the tables for native table loaders.

Some other points about the BioText parsemedline.pl program:

• as with the java program, it is unnecessarily database-specific (although only in a very minor way compared with the java code);
• it was written without the strict or warnings pragmas, and as a result, there are actual (minor) bugs in it due to misspelling of the lengthy variable names;
• Medline keeps changing its DTDs, so to capture all the data in the 2005 release of Medline requires some tedious changes to the SQL table definitions and code.

Question: I know this is not necessarily a good idea for performance reasons, both in loading and querying, but ... is it possible to automatically translate DTD descriptions into SQL DDL and corresponding code to parse the XML and load the data? (Ignoring the complication of data types).

[graff]

I know this is not necessarily a good idea for performance reasons, both in loading and querying, but ... is it possible to automatically translate DTD descriptions into SQL DDL and corresponding code to parse the XML and load the data?

How do you know (or what makes you think) that parsing a DTD is "not ... a good idea for performance reasons ..."? I doubt that using this sort of facility would have any noticeable impact on run-time performance, and it could certainly be a major boost to programmer performance (and would a good way to reduce code that is too bulky and ad-hoc).

There appear to be at least a couple modules on CPAN for converting DTD's into perl-internal objects or data structures: XML::DTDParser, XML::Smart::DTD. (I haven't used either of them myself, but a brief look at the docs makes me think the first one might be more suitable; I expect there are others.)

As for converting a perl object or data structure into an SQL DDL (or going directly from DTD to DDL), I haven't searched CPAN for that (maybe you could try it), but it seems like it could be less of a cut-and-dried sort of task; there might be different ways of specifying a table, or designing a set of relational tables, based on a given DTD, depending on what the SQL users want to do with the data.

(The same could be said for deriving different perl data structures from a DTD, but since people have already posted solutions for this on the CPAN, it might be worth trying what they've come up with.)

[haricothoriz]

I know this is not necessarily a good idea for performance reasons, both in loading and querying, but ... is it possible to automatically translate DTD descriptions into SQL DDL and corresponding code to parse the XML and load the data?

How do you know (or what makes you think) that parsing a DTD is "not ... a good idea for performance reasons ..."? I doubt that using this sort of facility would have any noticeable impact on run-time performance, and it could certainly be a major boost to programmer performance (and would a good way to reduce code that is too bulky and ad-hoc).

You're right; I don't know for sure, and I agree about the savings in programmer time (frankly what I was interested in). The medline database has 15,000,000 big fat records, and I was guessing that a hand-teaked, denormalized table layout would be more optimal for querying and maybe data loading. But I don't know yet. Thanks for pointing out the CPAN modules. I was just wondering if I had overlooked a module that does exactly what I am interested in.