Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

Re^4: Writing a database lookup tool

by elef (Friar)
on Jan 18, 2013 at 11:54 UTC ( #1014040=note: print w/replies, xml ) Need Help??

in reply to Re^3: Writing a database lookup tool
in thread Writing a database lookup tool

"FYI - current versions include FTS (Full text search) capabilities."
Thank you for that info. Full text search looks like it was designed for precisely the type of queries I'd be using. Based on the descriptions I found, it would add a lot of functionality and a lot of speed. Now, the main question is: do I get FTS with the perl database modules? The DBI::DB and DBD::SQLite cpan pages don't mention FTS, but it seems to be a pretty old feature so it should have trickled down, right?

Edit: I digged around a bit more and found out that FTS is supported by DBD::SQLite:

"However, I got the impression you wanted a stand-alone (not web-server based) solution."
Yes. The database and the lookup tool would be on the "client" machine, and ideally the whole thing should be reasonably compact and self-contained. One not too large (hopefully <50MB without the data) download, one not too complex installation.

Replies are listed 'Best First'.
Re^5: Writing a database lookup tool
by elef (Friar) on Jan 20, 2013 at 09:02 UTC
    Update: I started playing with DBD::SQLite. I imported 800,000 records into a db, and quickly realized that by default, most of my searches are indeed carried out as sequential searches, which makes them hopelessly slow. With FTS enabled, average lookup times fell from 4 seconds to 0.04 seconds. If this scales in a roughly linear fashion and I get 0.5 sec lookups on 10 million records, I'll be very happy with the speed.
    Thanks again.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1014040]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (4)
As of 2017-03-29 21:49 GMT
Find Nodes?
    Voting Booth?
    Should Pluto Get Its Planethood Back?

    Results (353 votes). Check out past polls.