Re: selecting columns from a tab-separated-values file

[Kenosis]

in reply to selecting columns from a tab-separated-values file

Consider slicing a split to get only the elements you need, instead of splitting the entire line, as it benchmarks significantly faster:

use strict;
use warnings;
use Benchmark qw(cmpthese);

my $line = "FIRST\tMIDDLE\tLAST\tSTRNO\tSTRNAME\tCITY\tSTATE\tZIP" . "
+\tFOO" x 42;

sub trySplit {
    my @capture = split /\t/, $line;
}

sub trySplitSlice {
    my @capture = ( split /\t/, $line )[ 0, 2, 5 ];
}

sub trySplitSliceLimit {
    my @capture = ( split /\t/, $line, 7 )[ 0, 2, 5 ];
}

cmpthese(
    -5,
    {
        trySplit           => sub { trySplit() },
        trySplitSlice      => sub { trySplitSlice() },
        trySplitSliceLimit => sub { trySplitSliceLimit() }
    }
);

Results:

                       Rate         trySplit    trySplitSlice trySplit
+SliceLimit
trySplit           110337/s               --             -46%         
+      -84%
trySplitSlice      204730/s              86%               --         
+      -71%
trySplitSliceLimit 708158/s             542%             246%         
+        --

Update: Have added choroba's trySplitSliceLimit() option to the benchmarking.

Update II: Thanks to AnomalousMonk, have appended "\tFOO" x 42 to the original string to create a string with 50 tab-delimited fields. This effectively shows the speed increase using trySplitSliceLimit().

Update III: Changed splitting on ' ' to \t. Thanks CountZero.

[choroba]

Limiting the splice makes it even faster:

sub trySplitSliceLimit {
    my @capture = (split ' ', $line, 7)[ 0, 2, 5 ];
}

[Kenosis]

Yes--excellent! Will modify my posting to include this option.

[ibm1620]

This turns out to make the biggest difference, far and away. Of course, it depends on what fields are chosen - in particular, where the rightmost one is.

[AnomalousMonk]

How about using a 'record' string with more fields to better show the effect of a split limit? Just append something like  "\tFOO" x 42 to the existing eight-field string.

[Kenosis]

Yes--impressive! Doing so makes a huge difference in the benchmarking. Will update the update...

[ibm1620]

This is nice. Couple of questions:

What's the first arg to split()? It appears to be a single blank char. How does that work to split upon tab chars?

In trySplitSliceLimit, wouldn't it be better to set LIMIT to 3, or in general to the number of fields you expect to extract?

And one observation for the record: the indices can appear in any order. To extract LAST, FIRST, and CITY, you'd write [2, 0, 5]

[Kenosis]

What's the first arg to split()? It appears to be a single blank char. How does that work to split upon tab chars?

It's a space enclosed within single quotes. It tells split to split on whitespace, e.g., \t, \n, space.

In trySplitSliceLimit, wouldn't it be better to set LIMIT to 3, or in general to the number of fields you expect to extract?

It should be set to the number of fields plus one that are needed to get the fields you want. For example, using your original string:

"FIRST\tMIDDLE\tLAST\tSTRNO\tSTRNAME\tCITY\tSTATE\tZIP"
  1       2      3      4       5       6     7 ---->

my @capture = ( split /\t/, $line, 7 )[ 2, 0, 5 ];

You want LAST FIRST CITY and CITY is the sixth field. Setting the LIMIT to seven will return the first six fields and the remainder of the string is the seventh. The slice is then used on those seven to get only the three you want.

And one observation for the record: the indices can appear in any order. To extract LAST, FIRST, and CITY, you'd write [2, 0, 5]

You're correct!

Update: Changed splitting on ' ' to \t. Thanks CountZero.

[CountZero]

What's the first arg to split()? It appears to be a single blank char. How does that work to split upon tab chars?

It's a space enclosed within single quotes. It tells split to split on whitespace, e.g., \t, \n, space.

That is a dangerous thing to do. it works for your example data, but it will break on real word data where you will have LAST names like Van Winkle and CITYs called New York.

[Kenosis]

Deleted--replied to myself. Time for sleep...