Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much

Resources for building database *server* interface

by etcshadow (Priest)
on Nov 12, 2004 at 17:48 UTC ( [id://407443] : perlquestion . print w/replies, xml ) Need Help??

etcshadow has asked for the wisdom of the Perl Monks concerning the following question:


I'm trying to find any resources (documents would be cool but modules would be great, too!) for building a database server interface in perl. Note: I am NOT talking about building a DBD. I mean: making a perl process that other processes could connect to and query by a standard database access method (quite possibly ODBC... but that's not specifically necessary). Again: this is the OPPOSITE of making a new DBD which would allow a perl process to connect as client to some database server. Anyway, just wanted to make that very clear, since it's bound to confuse folks.

So, the question is: has anyone here tried to do anything like this before? Or do you know of any work that's been done for this? Specifically, I'm looking for anything that might contribute at the layer of the connection between the database client and the perl process. I'm not looking for code that parses SQL or executes queries over data (obviously, there's already plenty of work like that around, anyway, such as DBD::AnyData, and that's not the aspect of this that I'm looking for help on). I have tried searching CPAN, but I can't find what I'm looking for, though maybe I am just being daft. If that's the case, then please correct me.

For context, what I am really trying to do is build a virtual database which fronts several actual datbases. Now, for my own code, I can obviously handle this with a phony sort of proxy DBD (actually, I don't even do that, I just built a wrapper around DBI, but that's not important). However, I'm looking into trying to hook up some third-party analysis tools, written in another language, running on another platform, etc. Now, it would be absolutely great if this third-part system could just connect to a virtual database and run queries against it, letting my code actually handle breaking up that query into the queries over the constituent data, execute them, and re-agregate them.

Thanks in advance!

------------ :Wq Not an editor command: Wq

Replies are listed 'Best First'.
Re: Resources for building database *server* interface
by iburrell (Chaplain) on Nov 12, 2004 at 18:07 UTC
    Check out DBD::Proxy and DBI::ProxyServer included with DBI. It is a pure Perl implementation of a proxy for DBI. I think it only works with a single database at a time. But the proxy can connect to different databases.

    If you want to have a single connection be multiplexed to multiple databases, then you are going to need something like DBD::AnyData. You will have to parse the SQL statement and produce a query plan. And then using schemas for the databases, figure out which data comes from where, and craft queries to retrieve it. And then put all the data together. DBD::AnyData can use DBI sources, but I don't know if it can use them for dynamic queries.

      Um... I think you misunderstand me. DBD's are used by perl code, to access external (or internal, or whatever) databases. I am trying to build the opposite. I am trying to build something that allows external database clients to connect to a perl-based data-store.

      It's true that DBI::ProxyServer is such a server (in a sense), but it only speaks one end of a database protocol that is spoken by DBD::Proxy at the other end. Hence, I still wouldn't be able to get a generic database client speaking (for example) ODBC to talk into a DBI::ProxyServer.

      ------------ :Wq Not an editor command: Wq
Re: Resources for building database *server* interface
by Anonymous Monk on Nov 12, 2004 at 20:42 UTC
    The problem you have to communicate to the monks is your misconception what ODBC is. Essentially you ask for a server for the ODBC network protocol. But there is no such thing. ODBC is like DBI. You need an ODBC-Driver (aka DBD) for your Database to connect to it with ODBC. Happily(?) every(?) Database comes with an ODBC-Driver. It is installed when you install the Database. So you are not aware of this. Your hypothetical MetaDatabase would need its own ODBC-Driver. So essentially the question is:

    How to write an ODBC-Driver in Perl?

      Good point.

      From here: ODBC is a call-level interface, not an application, network layer, or database driver.

      So, if you want to use ODBC (which is probably what your existing clients are able to talk to), you'll need to build a Windows DLL and/or a Unix shared library that implements the ODBC driver calling interface, and turns that into some kind of network traffic to your virtual server.

      You could take a look at OpenLink's Multi-Tier product for a commercial example of this type of solution. (In fact, you might be able to decode the network protocol they're using, which would let you reuse the existing drivers they've compiled for various platforms.)

      There also seem to be some free solutions that you might be able to piggyback on, like Boeireep. Googling for "odbc driver generic network protocol" should find some more candidates.

      Good call. I'm actually a little emberassed, now, but yes: this is exactly why I was getting confused. Thanks.

      That doesn't answer the question, however; it just helps me to phrase it properly.

      ------------ :Wq Not an editor command: Wq
Re: Resources for building database *server* interface
by dragonchild (Archbishop) on Nov 12, 2004 at 18:08 UTC
    Why go through all that trouble when you could just build a SOAP / XML-RPC server that handles business-level requests and passes them onto the correct database?

    Being right, does not endow the right to be rude; politeness costs nothing.
    Being unknowing, is not the same as being stupid.
    Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
    Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

      Let me try to explain it like this (maybe?) a little better... this is what I want to be able to do (in ASCII art):
      ???? DBI DBD /---------\ [] /---------\ [] [] <===> dbX |3rd party| ODBC [] |perl DB | [] [] |software | <========> [] |multiplex| [] [] <===> dbY \---------/ [] |logic | [] [] [] \---------/ [] [] <===> dbZ ^^^^^^^^^^^^^^^^^^^^^^ ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^ no real control over ^^ already exists, basically ^^ ^^ This is the part that I'm trying to get a handle around / maybe get some help with
      ------------ :Wq Not an editor command: Wq
Re: Resources for building database *server* interface
by erix (Prior) on Nov 12, 2004 at 18:13 UTC

    It sounds interesting. However I for one would like to get some more of that context.

    The only thing it reminds me of at the moment is the pulling together of several cgi frontends (=databases) to a single frontend. I've seen some examples of that. But I suppose you want something more, eh, detached?

      OK... more context, then (just for fun, though... doesn't really have anything to do with the question):

      We've got a system that has a large number of identically structured databases. Because they are identically structured, then you can imagine a sort of "virtual database" consisting of the sum of all the constituent databases. For example, if you had a table FOO in each of the databases like:

      ( FOOID number, FOOTYPE varchar, AMOUNT number
      for example. Thus, in the virtual database, there would exist a virtual table FOO like:
      ( FOOID number, FOOTYPE varchar, AMOUNT number,
        DATABASEID  number

      So, if you took a query like:

      select sum(amount), footype from foo group by footype
      and ran it against the virtual database, what would actually happen is that the query
      select sum(amount), footype from foo group by footype
      would get run in parallel through all of the identical (actual) DBs, and they would each return their result sets to the virtual DB, which would then roll up the individual result-sets into one result set. It's conceptually as though the query were executed like:
      select sum(sum_amount), footype from ( select sum(amount) as sum_amount, footype from group by footype UNION ALL select sum(amount) as sum_amount, footype from group by footype UNION ALL ... UNION ALL select sum(amount) as sum_amount, footype from group by footype ) group by footype
      Only with the "UNION"s being executed in parallel.

      To get an even better idea, there's stuff like:

      select sum(amount), footype from foo where databaseid in (1,2,3) group by footype
      This would get pulled apart, such that the "where databaseid in (...)" was removed from the query, and was used, instead, to control the set of databases over which to run the query.

      Anyway, none of that is what I want help with. It's already a done deal... I'm trying to figure out if there are any tools, etc, on how to build an ODBC server interface... I've already got the server to back it up, I just need the ability to slap an ODBC access method onto it.

      ------------ :Wq Not an editor command: Wq