Ladies and gentlemen, please forgive my “brashness” in advancing what may at first well appear to be “an irrelevant, sideways, Sundialsvc4-typically-off-the-topic guess ... and ... well ... hear me out on this one.
SQLite is not slow. ... ... except ... ... except ... ... in one very counter-intuitive case that just happens to be tremendously well-known to me, because for a time it absolutely show-stopped an equally important project of mine that also used this database. I was shocked at how hideous SQLite’s performance was! Then, one day, as I was amusing myself with the “activity monitor” while waiting for my script to grind on ... I noticed something. My script was doing a truly-extraordinary number of disk operations. The entirety of what was bringing this script to its most-improbable knees was ... disk I/O.
It was only-then that I looked for, and found (somewhere ...) a discussion of what SQLite actually does and why it does it. This now-forgotten page discussed how SQLite re-reads what is in memory and how it re-reads after it writes, and how “transactions” are a Magickal Cure for all of that. I had never encountered a file-based application before that did that. I tried it for myself and ... well ... “holy sh*t!” All of the sudden, this thing took-off like the proverbial bat.
Hence ... my guess. My guess that “too slow” is in fact due to a cause that I had not suspected, either, when I encountered it and was show-stopped by it for the first time. My guess that “threading, or the lack/presence thereof,” most likely has nothing to do with it. I therefore injected this admittedly out-of-band comment, to suggest a completely-different stone that should be turned-over first.
Before getting started with multi-threading this thing ... try this, first. Wrap every activity that writes or that reads to this database in a transaction. (If you are issuing a large number of statements, periodically commit.) But, do not do anything meaningful outside of one. Reserve judgment about “what to do with this program” until you have tried this first. (And if you already have done this, just send me to bed once again without any supper.)
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.
| & || & |
| < || < |
| > || > |
| [ || [ |
| ] || ] ||