Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

Re: Optimise file line by line parsing, substitute SPLIT

by Tux (Abbot)
on Jun 03, 2013 at 14:13 UTC ( #1036764=note: print w/replies, xml ) Need Help??

in reply to Optimise file line by line parsing, substitute SPLIT

If your data is easy to split on TAB's, perl's split is exceptionally fast. Building on BrowserUk's example - which is a good example - look at this test:

$ perl -E'say join"\t",qw(1 aap 2 noot 3 mies 4 wim 5 zus 6) for 1..2_ +000_000' > test.tsv $ wc -l test.tsv 2000000 test.tsv $ cat use 5.016; use warnings; use Benchmark qw( cmpthese ); use Text::CSV_XS; my $csv = Text::CSV_XS->new ({ sep_char => "\t", auto_diag => 1 }); cmpthese (3, { none => sub { open my $fh, "<", "test.tsv"; while (<$fh>) { } }, splt => sub { open my $fh, "<", "test.tsv"; while (<$fh>) { my @f = split m/\t/ => $_, -1; } }, tcsv => sub { open my $fh, "<", "test.tsv"; while (my $f = $csv->getline ($fh)) { # use $f->[1] instead of $f[1]; } }, tcsvbc => sub { open my $fh, "<", "test.tsv"; my @f = ("") x 11; $csv->bind_columns (\(@f)); while ($csv->getline ($fh)) { } }, }); $ perl s/iter tcsv tcsvbc splt none tcsv 5.26 -- -32% -36% -96% tcsvbc 3.57 48% -- -5% -93% splt 3.38 56% 6% -- -93% none 0.233 2156% 1429% 1347% -- $

where vsespb is missing the point, is that when determining if a record is to be split or not, one still has to look at the record. That is exaclty what BrowserUk was pointing at. Looking at the first character of a record is a cheap operation, not needing split at all, but if you want to decide processing on the content of the 11th field, you have to split before you can decide.

All of the above will start changing when the fields can be quoted and have (valid) TAB characters inside the field, or if fields are allowed to have embedded newlines or when the fields are (much) longer. Compare to e.g.:

$ perl -E'@x=("x" x 100) x 11;say join"\t",@x for 1..2_000_000' > test +.tsv $ perl s/iter tcsv tcsvbc splt none tcsv 23.3 -- -32% -74% -92% tcsvbc 15.8 47% -- -61% -89% splt 6.13 280% 158% -- -71% none 1.78 1206% 788% 244% --

Enjoy, Have FUN! H.Merijn

Replies are listed 'Best First'.
Re^2: Optimise file line by line parsing, substitute SPLIT
by sundialsvc4 (Abbot) on Jun 04, 2013 at 23:33 UTC

    ... and maybe simply looking for the string-of-interest anywhere in the original-string might be enough of a filter.   Decide whether the string exists anywhere, before you spent the time needed to decide if it’s the 11th field, instead of identifying the 11th field before you consider what it contains.   If split is deemed to be pricey, split only what you (guess) you (might) need to.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1036764]
marto mad corrections to one around 18 months ago
[ovedpo15]: my code checks the value after that comma, if it isn't valid it will remove it. so I would like to remove the substring after that comma meaning getting the string before comma.
[hippo]: Are you sure that Text::CSV_XS isn't already installed at your work? It's such a useful module that it might well be there.
[marto]: pointing out the advantages of cpan modules is well worth in, as both developer and $client/$company benefit greatly
[Tux]: $src =~ m{^(.*),(.*)$/ and $2 !~ $valid and $src = $1;
[Tux]: s,/,},

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (9)
As of 2018-05-27 10:51 GMT
Find Nodes?
    Voting Booth?