Re: Would you use SQLite?

[SwellJoe]

Hi Matt,

tilly referred me to SQLite in a couple of threads I've posted (Performance quandary and Berkeley DB performance, profiling, and degradation...) regarding a BerkeleyDB based program I'm developing...

So, in the spirit of desparately trying stuff until something works so that I can stop working on this darned program once and for all, I've just converted it from using BerkeleyDB to DBI::SQLite, and it seems to be working great. So the answer to your question is, "Yep. I would use SQLite." It seems to be working quite nicely, except for one quirk that I think is likely due to my ignorance of the DBI (more on that in a moment). The test database is about a quarter million sub-4k objects, with the real database topping 2 million--I haven't run it on the big database but on the test db it works great! The old BerkeleyDB version was roughly equal or a little faster in the case of a few thousand objects, but performance declined rapidly due to some pathological handling code in my program (which using a relational db allows me to eliminate completely).

Anyway, it seems like an ideal solution for me, as I have to provide a simple to use set of RPMs for my programs and SQLite is a lot easier to package up and use than MySQL or PostGreSQL. I'll be spending the next few hours on converting the support tools and testing with my small db before rolling it out to a client site for more serious work (don't worry, I can keep an eye on things and deal with any bugs that pop up--I know it's early code). Anything I ought to be watching for? In a couple of posts, here and at use.perl, I think you've mentioned suspected memory leaks--still suspicious?

Now to my problem...I have the following line:

$dbh->do("INSERT INTO objects (md5, url) VALUES ('$md5', '$key')");

Which produces the following error:

DBD::SQLite::db do failed: near "'http://www.gnome.org/images/banner-c
+ontribute": syntax error at ./logtaild.pl line 113, <OBJECT> line 1.

But only on a few (very few) entries into the db. I got about 50 errors like this during the full 250,000 object insertion run. The md5 field is a CHAR(32) while the url field is a VARCHAR(8192)...Am I doing something wrong here? I don't quite know what to make of the extra "'" in the entries that lead to an error--there is nothing particularly different about these few entries that I can see. Searching the Monastary and Google doesn't bring me any enlightenment, so it doesn't seem to be a DBI general issue, but I may have missed it.

Anyway, thanks for the SQLite DBD--it's very cool, and worked right out of the box. One issue, it doesn't seem that the NDEBUG option is disabled in the module build...the SQLite author pretty strongly recommends doing so, at least when performance matters.

[broquaint]

Is this possibly because there are single quotes in the $key value? So your SQL may in fact look like this (minus the formatting :-)

INSERT INTO objects (md5, url)
VALUES (
    '$md5',
    ''http://www.gnome.org/images/banner-contribute'
--   ^ note the extra single quote!!!
)

If so, it would be a good case to use those ever handy DBI placholders

$dbh->do("INSERT INTO objects (md5, url) VALUES (?,?)"
         undef,
         $md5, $key);


HTH

broquaint

[Matts]

Also if you do a prepare outside your loop followed by the execute with placeholders inside the loop, you'll see an eensy weeny tiny bit of a performance boost, because I don't have to keep re-parsing the SQL.

But no, all memory leaks appear to be cleared up now.

And I'll look into the NDEBUG issue - thanks!

[SwellJoe]

Already have the prepare outside the loop (I tend to read lots before I try something new--SQL and DBI were entirely new to me, so I read all 5 or so posts by you on the subject as well as all of the SQLite page ;-).

The NDEBUG thing is here: Database Speed Comparison

The specific line is:

SQLite version 2.0-alpha-2 was compiled with -O2 optimization and with the -DNDEBUG=1 compiler switch. Setting the NDEBUG macro is very important in SQLite version 2.0. SQLite 2.0 contains some expensive "assert()" statements in the inner loop of its processing. Setting the NDEBUG macro makes SQLite 2.0 run nearly twice as fast.

This may already be set by default now, but everything I could find in the makefiles indicates it is not. So, DBI::SQLite might get a nice little speed boost for next to nothing. But even if not, it is already plenty fast so far for my application. The client side CGI script has gotten slower, by quite a lot (as it has to connect to the DB and whatever else goes into starting up). It takes it a good three seconds to begin working, but once it is producing results it goes at a very zippy pace. So no complaints here.

Thanks for the additional info on the possibility of leaks. And thanks a ton for the module--it works great, and was very easy to install and program (I guess DBI gets the credit for programmability--but you did a great job packaging SQLite into a module/DBD).

[SwellJoe]

Thanks broquaint!

That seems to have done it. I had tried that already, but I didn't get the syntax quite right so got strange errors (I left out the 'undef'--why is that there/required?).

[broquaint]

> I left out the 'undef'--why is that there/required?
It certainly is required :-) It's used to "pass driver specific hints", and as yet, I've found no use for it (but that's because I mainly use MySQL which doesn't make use of these hints (as far as I know ...)).
HTH

broquaint