in reply to Re: Optimise file line by line parsing, substitute SPLIT
in thread Optimise file line by line parsing, substitute SPLIT
That's true, but not always.
Sometimes you don't need process even 1% of the data.
You just read it, split, and drop 99.9% of lines where field1 <> 'abcd' (that's where SQL can help)
Or you read webserver logs into memory hash (grouped by IP) and then you do scoring of new site visitors in real time (and you need access only records related to particular IP) (and SQL will be slower)
Or maybe you read list of files from text file, and read filelisting from disk, and then compare it in memory (no use for SQL)
Or general case - you read data from text files (1M lines) and skip all records which are not already in another memory hash (there are 10K lines)
|
---|
Replies are listed 'Best First'. | |
---|---|
Re^3: Optimise file line by line parsing, substitute SPLIT
by BrowserUk (Patriarch) on Jun 03, 2013 at 14:11 UTC | |
by vsespb (Chaplain) on Jun 03, 2013 at 14:32 UTC | |
by BrowserUk (Patriarch) on Jun 03, 2013 at 14:54 UTC | |
by vsespb (Chaplain) on Jun 03, 2013 at 15:09 UTC | |
by BrowserUk (Patriarch) on Jun 03, 2013 at 16:03 UTC | |
Re^3: Optimise file line by line parsing, substitute SPLIT
by hdb (Monsignor) on Jun 03, 2013 at 14:17 UTC | |
by vsespb (Chaplain) on Jun 03, 2013 at 14:34 UTC |