Re: cut vs split (suggestions)

[tlm]

I'd be very surprised if a pure-Perl script beat a native utility like cut.

If cutting out of the columns of these large files is a bottleneck, your options include:

• As you mentioned, piping the output of cut to your script;
• Running cut as a pipe from within your script:

  open my $cutter, "-|", 'cut', '-d,', '-f"1-15"', $mongo
    or die "Fork failed: $!\n";
  go_to_town( $_ ) while <$cutter>;
  close $cutter;
  

  See perlipc.
• Write a Perl extension module (XS, Inline::C, Swig) to extract the rows.

I like the second option best.

[BrowserUk]

Once you read each line of output from cut via the piped open, you are still going to have to split it to an array in order to utilise the fields, so I think most if not all the performance advantage of using cut will be lost, though spitting 15 fields cut from 200 rather than all 200 may help.

[tlm]

The internal pipe approach is about 1.5X faster than the pure Perl approach (though still a far cry from cut):

% time perl -le 'open IN, q( cut -d, -f"1-15" numbers.csv| ); \
print join ",", ( chomp and @F = split /,/ ) while <IN>' > /dev/null
19.49s user 0.00s system 96% cpu 20.289 total

Update: But keep in mind that the numbers above are for a relatively fast cut command. The improvement with sk's cut will be more modest; it'd be interesting to see the actual numbers.