Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

What's the best module to stop mixing SQL with Perl?

by Withigo (Friar)
on Sep 08, 2006 at 19:56 UTC ( [id://572079]=perlquestion: print w/replies, xml ) Need Help??

Withigo has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,
I am encountering the well known problem of mixing SQL and Perl in a huge codebase, and am looking for some advice about some alternatives to simplify or abstract my code.

Let's say I have a thousand subs, each of which has multiple handcrafted SQL queries which are doing complex joins and subselects on 4 or more other tables. It's a nightmare to alter to say the least.

But I would like to avoid the performance hits that Class::DBI and DBIx::Class impose. (I'm assuming DBIx::Class is now more popular/prefered to Class::DBI?). I like that plain DBI is 10-50 times faster.

And after accounting for the complex joins, subselects and where conditions I don't think representing each table as a single class will model very well to my code. I think for the OO approach to fit my existing code, I would need a class that represents the composition of several tables, but I may be overlooking the full capabilities of DBIx::Class etc.

I am looking into using an SQL phrasebook to reduce and separate the SQL from perl, but there are several such modules on CPAN to choose from. Can anyone recommend the best modules that might be appropriate for my situation and make my code better?

I'm preferably looking for the modules that are the most popular, most used and most recent. I looked at these modules:
Class::Phrasebook
Data::Phrasebook::SQL
DBIx::Phrasebook
SQL::Catalog
Thanks!

Replies are listed 'Best First'.
Re: What's the best module to stop mixing SQL with Perl?
by jZed (Prior) on Sep 08, 2006 at 20:15 UTC
    One of the differences between the various modules is how they store the SQL. SQL::Catalog stores the SQL in an RDBMS; DBIx:POS stores it in Plain Old SQL, a POD-like format; Class::Phrasebook::SQL stores it in XML.

    Personally I have a preference for storing the SQL in plain text files for easy editing. I use SQL comments to name the snippets and read the names and snippets into a hash with a simple parse on /;\n\n+/. One advantage of doing it that way is that the files are nothing but pure SQL and so can be run through dbish or other SQL-file processors specific to various RDBMSs and can be easily read by DBAs with no knowledge of how the SQL is used in an application or even what language the app is written in.

Re: What's the best module to stop mixing SQL with Perl?
by jimt (Chaplain) on Sep 08, 2006 at 23:20 UTC

    I've gotta agree with the stored procedure/view solution, despite the fact that I despise it in practice (since it when I've seen it suggestied, it's to pull the power away from the developers and hands it to the DBA), but it does have genuine speed benefits and keeps all the SQL out of your code.

    I know that Basset's persistent object can map multiple tables into a single object, and I think it does it pretty well and efficiently. But, as I've said in another thread, I'm biased (being the author and all). It's heavily supported, but not widely used, so you may find it difficult to work with. If so, I'd love to hear about it. Suggestions very welcome.

    Simple SQL::Abstract might be useful, too. Instead of having sql statements all over the place, you'd be using it to construct queries for you. You end up with the same stuff, but it's much easier to abstract out the table or column names to central locations. Yeah, yeah, you can do it with plain old statements, but it'd probably be a bit cleaner by using SQL::Abstract with a few custom brewed containers around it.

      I've gotta agree with the stored procedure/view solution, despite the fact that I despise it in practice (since it when I've seen it suggestied, it's to pull the power away from the developers and hands it to the DBA), but it does have genuine speed benefits and keeps all the SQL out of your code.

      I have the (dis?)advantage that I'm both the developer and the DBA. And the few times that I've had a dedicated DBA that wasn't me, they were still willing to let me write all of the code. (I just had to go through them to make changes).

      The main advantages to views and stored procedures (at least in Oracle), is that I can tune the SQL with hints on indexes to use (or not use) and force the method of joins (eg, a sort-merge is typically better for getting all records, but a correlated join may look faster to the user if I'm paginating it)

      I specifically don't like like the various modules to remove my access to the SQL, as they haven't given me the level of control that I want for tuning. (yes, I know I'm picky, but in one project, it was the difference between 0.62 sec to generate a webpage, and 0.10 sec, partially due to optimizing the table joins and use of materialized views but also removing some ineffective indexes, pinning tables in memory and creating some multi-column indexes for the most queried fields).

Re: What's the best module to stop mixing SQL with Perl?
by jhourcle (Prior) on Sep 08, 2006 at 20:19 UTC

    Have you considered a database-side solution?

    Depending on what exactly you're doing, you may be able to use a combination of views and/or stored procedures to deal with repetitition in your queries.

Re: What's the best module to stop mixing SQL with Perl?
by chromatic (Archbishop) on Sep 08, 2006 at 20:27 UTC
    I would like to avoid the performance hits that Class::DBI and DBIx::Class impose.

    Is this measurable? I assume that a decently-written application will spend a lot more time contacting the database and waiting to prepare, calculate, and retrieve results than it will executing Perl ops. IO is slow; network IO even slower.

      That is true, but in some cases the way the tools use SQL itself is inefficient compared to what you would write by hand. Rose::DB::Object and DBIx::Class avoid this for the most part though. See this benchmark for some comparisons.
      I am basing my assumptions on this; which I originally found linked in a Rose::DB thread on here.
      It depends on the application. The reason is not because of the construction of the SQL, but because of using one query per command. That can be damn slow: if you try to add a lot of data (ten thousands of records) into the database, using such a module can take 15 minutes where a plain native import takes 10 seconds. That is 2 orders of magnitude slower. Plain DBI is somewhere between these two extremes.

      For single record access, it'll hardly make a difference, it's just a matter of milliseconds.

Re: What's the best module to stop mixing SQL with Perl?
by mreece (Friar) on Sep 09, 2006 at 18:13 UTC
    if you just want to move the sql out of the way, what about Ima::DBI?
Re: What's the best module to stop mixing SQL with Perl?
by bibliophile (Prior) on Sep 11, 2006 at 14:22 UTC
    What I've done in a similar situation was to move ALL of the SQL out of the app and into a DB-abstraction module. The app just calls (for instance) a "get me a client record" routine, and the abstraction module handles all the database jiggery-pokery.

    It's (I think) a good first step toward the SQL phrasebook stuff you're looking at. And (as in my case) you may find that you don't need to take it any further. The big bonus is that maintenance is *way* easier than what you started with....

    Just my $0.02CDN

    -- WARNING: You are logged into reality as root.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://572079]
Approved by Joost
Front-paged by grinder
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others surveying the Monastery: (7)
As of 2024-03-28 08:26 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found