Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?

Re^2: search a large text file

by creamygoodness (Curate)
on Feb 09, 2011 at 22:24 UTC ( #887307=note: print w/replies, xml ) Need Help??

in reply to Re: search a large text file
in thread search a large text file

I suspect that KinoSearch would work about as well as a database like SQLite or PostgreSQL for this. It's actually a decent conceptual match -- inverted indexers like KinoSearch, Lucene, Xapian, etc. are optimized for many reads and fewer inserts, as opposed to the typical B-tree indexes on databases which handle inserts a little better. The only thing that's odd is that the original poster doesn't seem to need the relevance-based ranking that inverted indexes do well.

Regardless, the problem is straightforward and there are lots of good options for solving it.

Replies are listed 'Best First'.
Re^3: search a large text file
by erix (Parson) on Feb 10, 2011 at 13:46 UTC

    PostgreSQL does indeed have btree indexes, but also inverted indexes (GIN), and the excellent GIST index type. (it seems to me the btree type does well enough in this case; if you see my example below, where searching in a 223-million+ rows table takes a tenth of a millisecond).

    PostgreSQL index-type docs here.

    I'm just reacting to the juxtaposition of sqlite and postgres; really: SQLite, handy as it often is, can not be compared with a powerful database system like postgresql.

    (And I should really try & compare Your Mother's example with KinoSearch, and see if he is right; maybe in the weekend... )

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://887307]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (2)
As of 2020-06-06 12:04 GMT
Find Nodes?
    Voting Booth?
    Do you really want to know if there is extraterrestrial life?

    Results (41 votes). Check out past polls.