Beefy Boxes and Bandwidth Generously Provided by pair Networks Frank
Perl Monk, Perl Meditation
 
PerlMonks  

Re: Using the unicode61 tokenizer in DBD::SQLite

by Anonymous Monk
on Jan 06, 2014 at 02:28 UTC ( #1069445=note: print w/ replies, xml ) Need Help??


in reply to Using the unicode61 tokenizer in DBD::SQLite

I just updated DBD::SQLite to 1.40 and made sure I have SQLite version 3.7.17.

But what does DBD::SQLite have, what is sqlite_version? update: I see you got it #define SQLITE_VERSION "3.7.17"

Was SQLite compiled without enabling the unicode61 tokenizer? (Some sources mention compiling sqlite with SQLITE_ENABLE_FTS4_UNICODE61 in order to get this functionality.)

I see that that conditional is in the source /* fts3_unicode2.c (functions generated by parsing unicode text files) */ #ifdef SQLITE_ENABLE_FTS4_UNICODE61 but since you get an error message, it probably wasn't compiled with that turned one ( nothing turned on in source)

Do I have any options here?

I think recompile with   perl Makefile.PL DEFINE=-DSQLITE_ENABLE_FTS4_UNICODE61 and give it a shot

update: I just recompiled, all tests passed except some in t/51_table_column_metadata.t , same with  cpanp -z ISHIGAKI/DBD-SQLite-1.41_03.tar.gz

same with  perl Makefile.PL "DEFINE=-DSQLITE_ENABLE_FTS4_UNICODE61 -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS "

Seems to work

$ perl -Mblib -S dbish dbi:SQLite:testgonernow DBI::Shell 11.95 using DBI 1.628 WARNING: The DBI::Shell interface and functionality are ======= very likely to change in subsequent versions! Connecting to 'dbi:SQLite:testgonernow' as ''... @dbi:SQLite:testgonernow> /table_info TABLE_CAT,TABLE_SCHEM,TABLE_NAME,TABLE_TYPE,REMARKS,sqlite_sql undef,'main','sqlite_master','SYSTEM TABLE',undef,undef undef,'temp','sqlite_temp_master','SYSTEM TABLE',undef,undef [2 rows of 6 fields returned] @dbi:SQLite:testgonernow> CREATE VIRTUAL TABLE mytable USING fts4 (tok +enize=unicode61); [0E0 rows affected] @dbi:SQLite:testgonernow> /table_info TABLE_CAT,TABLE_SCHEM,TABLE_NAME,TABLE_TYPE,REMARKS,sqlite_sql undef,'main','mytable_segdir','INDEX',undef,undef undef,'main','sqlite_master','SYSTEM TABLE',undef,undef undef,'temp','sqlite_temp_master','SYSTEM TABLE',undef,undef undef,'main','mytable','TABLE',undef,'CREATE VIRTUAL TABLE mytable USI +NG fts4 (tokenize=unicode61)' undef,'main','mytable_content','TABLE',undef,'CREATE TABLE \'mytable_c +ontent\'(docid INTEGER PRIMARY KEY, \'c0content\') ' undef,'main','mytable_docsize','TABLE',undef,'CREATE TABLE \'mytable_d +ocsize\'(docid INTEGER PRIMARY KEY, size BLOB)' undef,'main','mytable_segdir','TABLE',undef,'CREATE TABLE \'mytable_se +gdir\'(level INTEGER,idx INTEGER,start_block INTEG ER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(le +vel, idx))' undef,'main','mytable_segments','TABLE',undef,'CREATE TABLE \'mytable_ +segments\'(blockid INTEGER PRIMARY KEY, block BLOB )' undef,'main','mytable_stat','TABLE',undef,'CREATE TABLE \'mytable_stat +\'(id INTEGER PRIMARY KEY, value BLOB)' [9 rows of 6 fields returned] @dbi:SQLite:testgonernow> /exit Disconnecting from dbi:SQLite:testgonernow. $ rm testgonernow


Comment on Re: Using the unicode61 tokenizer in DBD::SQLite
Select or Download Code
Re^2: Using the unicode61 tokenizer in DBD::SQLite
by elef (Friar) on Jan 06, 2014 at 10:10 UTC
    Thanks, Anonymonk.
    I have no idea about any of the steps required for recompiling sqlite for DBD::SQLite. I don't even know where the executable or the source code is stored. Can you give me some pointers? I'm on Windows BTW.
      Um, I'm on windows too .... if you don't recognize cpanp -z... perl Makefile... you probably want to read A Guide To Installing Modules ... among others
        I seem to have missed that bit. Are you saying that the unicode61 tokenizer works out of the box in 1.41_03?
        Yes, according to DBD::SQLite it should work out of the box.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1069445]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (6)
As of 2014-04-18 02:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (460 votes), past polls