Re: DBI::SQLite slowness (DBD::SQLite bulk insert woes)

[Anonymous Monk]

See DBD::SQLite bulk insert woes and use http://metacpan.org/module/DBD::SQLite#Transactions, so

## begin transaction
## foreach loop
## end transaction

see also Re^5: Index a file with pack for fast access/Sauntering Coder: Fastest bulk import into sqlite which says

.echo ON

.read create_table_without_pk.sql

PRAGMA cache_size = 400000;
PRAGMA synchronous = OFF;
PRAGMA journal_mode = OFF;
PRAGMA locking_mode = EXCLUSIVE;
PRAGMA count_changes = OFF;
PRAGMA temp_store = MEMORY;
PRAGMA auto_vacuum = NONE;

.separator "\t"
.import a_tab_seprated_table.txt mytable

BEGIN;
.read add_indexes.sql
COMMIT;

.exit

sqlite3 mydb.db < commands.txt

#~

http://www.sqlite.org/pragma.html#pragma_cache_size
http://www.sqlite.org/pragma.html#pragma_synchronous
http://www.sqlite.org/pragma.html#pragma_journal_mode
http://www.sqlite.org/pragma.html#pragma_locking_mode
http://www.sqlite.org/pragma.html#pragma_count_changes
http://www.sqlite.org/pragma.html#pragma_temp_store
http://www.sqlite.org/pragma.html#pragma_auto_vacuum
Command Line Shell For SQLite

[Anonymous Monk]

See also SQLite Optimization FAQ (old but informative) and SQLite Slow insert (use transactions)

[marinersk]

I can attest to efficiency improvements due to at least one of the notes in SQLite Optimization FAQ. Used it at a customer site once, too long ago to remember the numbers but I recall the difference was dramatic. Sample of the commands needed:

C:\Steve>sqlite test.db

C:\Steve>call sqlite-STEVEM2 test.db

C:\Steve>"C:\App\SQLite\sqlite3.exe" test.db
SQLite version 3.3.5
Enter ".help" for instructions
sqlite> pragma default_cache_size;
2000
sqlite> pragma default_cache_size=200000;
sqlite> pragma default_cache_size;
200000
sqlite> .exit

C:\Steve>sqlite test.db

C:\Steve>call sqlite-STEVEM2 test.db

C:\Steve>"C:\App\SQLite\sqlite3.exe" test.db
SQLite version 3.3.5
Enter ".help" for instructions
sqlite> pragma default_cache_size;
200000
sqlite> .quit

C:\Steve>