Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Re^4: Writing a database lookup tool

by elef (Friar)
on Jan 18, 2013 at 11:54 UTC ( #1014040=note: print w/ replies, xml ) Need Help??


in reply to Re^3: Writing a database lookup tool
in thread Writing a database lookup tool

"FYI - current versions include FTS (Full text search) capabilities."
Thank you for that info. Full text search looks like it was designed for precisely the type of queries I'd be using. Based on the descriptions I found, it would add a lot of functionality and a lot of speed. Now, the main question is: do I get FTS with the perl database modules? The DBI::DB and DBD::SQLite cpan pages don't mention FTS, but it seems to be a pretty old feature so it should have trickled down, right?

Edit: I digged around a bit more and found out that FTS is supported by DBD::SQLite: http://blogs.perl.org/users/adam_kennedy/2012/05/next-dbdsqlite-to-be-released-in-early-june.html

"However, I got the impression you wanted a stand-alone (not web-server based) solution."
Yes. The database and the lookup tool would be on the "client" machine, and ideally the whole thing should be reasonably compact and self-contained. One not too large (hopefully <50MB without the data) download, one not too complex installation.


Comment on Re^4: Writing a database lookup tool
Re^5: Writing a database lookup tool
by elef (Friar) on Jan 20, 2013 at 09:02 UTC
    Update: I started playing with DBD::SQLite. I imported 800,000 records into a db, and quickly realized that by default, most of my searches are indeed carried out as sequential searches, which makes them hopelessly slow. With FTS enabled, average lookup times fell from 4 seconds to 0.04 seconds. If this scales in a roughly linear fashion and I get 0.5 sec lookups on 10 million records, I'll be very happy with the speed.
    Thanks again.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1014040]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (7)
As of 2014-10-25 19:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (148 votes), past polls