Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re^6: How do I read from a compressed SQLite FTS4 database with DBD::SQLite?

by elef (Friar)
on Nov 30, 2013 at 09:57 UTC ( [id://1065043]=note: print w/replies, xml ) Need Help??


in reply to Re^5: How do I read from a compressed SQLite FTS4 database with DBD::SQLite?
in thread How do I read from a compressed SQLite FTS4 database with DBD::SQLite?

Thanks for the suggestion.
I use ActivePerl on Widows and IO::Compress::Xz is not in PPM. I tried to install it from the cpan shell but running the script fails with Can't locate auto/Compress/Raw/Lzma/autosplit.ix in @INC. Maybe I will try to get the module installed in a linux VM and see how it performs.
I do have IO::Compress::Zip so I tried that but the compressed db was much larger than the an uncompressed one with the same data... It's starting to look like I'm in a dead end.
Again, I think the compression is failing this badly because each column in each record is compressed separately by the FTS engine and I have text here in small chunks (sentences). But I don't think it would be feasible to structure the db differently because of the way the data is used (you search for a term or phrase and the program shows you each sentence that it occurs in, along with its translation in a different language).
  • Comment on Re^6: How do I read from a compressed SQLite FTS4 database with DBD::SQLite?
  • Download Code

Replies are listed 'Best First'.
Re^7: How do I read from a compressed SQLite FTS4 database with DBD::SQLite?
by taint (Chaplain) on Nov 30, 2013 at 19:54 UTC
    Ahh, I hadn't thought Windows wouldn't have it. Sorry, my bad.
    I seem to remember that LZMA had an "off by one" issue, a few years ago. Maybe that's why it isn't yet in the ActiveState edition.

    Yea, I think the only way to accomplish your goal most effectively. Would be to compress the contents of the rows/tables, without compressing the tables/rows, themselves. But, as you seem to have stated, it's probably not feasible.

    Best wishes, elef.

    --Chris

    #!/usr/bin/perl -Tw
    use Perl::Always or die;
    my $perl_version = (5.12.5);
    print $perl_version;

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1065043]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others browsing the Monastery: (3)
As of 2024-04-20 09:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found