Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: DBD::SQLite tuning

by dragonchild (Archbishop)
on Mar 16, 2005 at 18:04 UTC ( #440067=note: print w/replies, xml ) Need Help??


in reply to DBD::SQLite tuning

It appears, from cursory examination, that commits on SQLite are implied to be slow.

From the DBD::SQLite documentation:

SQLite is fast, very fast. I recently processed my 72MB log file with it, inserting the data (400,000+ rows) by using transactions and only committing every 1000 rows (otherwise the insertion is quite slow), and then performing queries on the data. (Emphasis added)

From http://www.sqlite.org/cvstrac/wiki?p=PerformanceConsiderations, it says:

When doing lots of updates/inserts on a table it is a good idea to contain them within a transaction, . . . This will make SQLite write all the data to the disk in one go, vastly increasing performance. (Emphasis added)

Apparently, what's happening is there is a disk flush of the actual changes to the table upon commit. This apparently is to allow for preservation of committed data if the database crashes. Oracle and MySQL (using InnoDB) have REDO logs which capture the commands that made the change, then actually apply the changes when it's convenient or necessary. (MySQL, using MyISAM tables, doesn't have this guarantee. This is why you can have corruption of MyISAM tables, but not InnoDB tables.)

This constant flushing of the tables (and the attendant indices) looks to be why you have the performance issues.

Note: I don't have proof this is what's happening, but it seems like a reasonable guess.

Being right, does not endow the right to be rude; politeness costs nothing.
Being unknowing, is not the same as being stupid.
Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

Replies are listed 'Best First'.
Re^2: DBD::SQLite tuning
by perrin (Chancellor) on Mar 16, 2005 at 21:20 UTC
    As I said in the intro, leaving out all commits makes it go about three times as fast, but then it wouldn't work for sharing the data between multiple processes. At least I'm assuming that other processes can't see uncomitted data.
      I understand all of that. You were asking "Why is this slow?" (which is the necessary precursor to "How do I make this faster?"). I replied "Commits are implied to be slow in SQLite."

      I suspect that you are using SQLite correctly, but not for its intended purpose. It's like using Oracle for an online forum or BDB for a data warehouse. You could do that, but it wouldn't be recommended as best.

      The person who claimed Foo is Nx faster than Bar is correct, for his/her needs. And, for your needs, Bar is Nx faster than Foo. We've all seen situations where a semi-tuned MySQL 4.1 database on a single CPU blew away an tuned Oracle 9.2 database running on a quad-Xeon. Yet, we're not going to claim that MySQL 3.x is faster than Oracle in all situations.

      For your usage, BDB is clearly better. SQLite is probably better than BDB if the following is true:

      • Reads * LargeNumber >= Writes
      • Reads involve complex queries
      • Different reads may use different indices
      Otherwise, it's probably best to use BDB.

      Of course, I'm still wondering why SQLite is good when embedded MySQL exists. (Though the weird license is probably part of it ...)

      Being right, does not endow the right to be rude; politeness costs nothing.
      Being unknowing, is not the same as being stupid.
      Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
      Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://440067]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (5)
As of 2022-01-24 21:10 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:












    Results (65 votes). Check out past polls.

    Notices?