Mabooka % head -1 numbers.csv
462,393,252,996,663,603,344,439,139,259,879,766,545,192,477,986,317,77
+,611,303,79,742,190,556,538
Mabooka % wc -l numbers.csv
500000 numbers.csv
Mabooka %
Mabooka % time perl -lanF, -e 'print join ",", @F[0..4];' numbers.csv
+> f1
27.820u 0.100s 0:27.92 100.0% 0+0k 0+0io 320pf+0w
Mabooka % time cut -d, -f"1-5" numbers.csv > f2
1.860u 0.100s 0:01.96 100.0% 0+0k 0+0io 100pf+0w
Mabooka % diff f1 f2
Mabooka %
So it's clear what to use (if it's a bottleneck problem rather than an academic disput).
Now, back to the original problem (sum up columns):
on my system, for 500,000 it's negligible, so I tried with 5,000,000 x 25 cols:
Mabooka % time perl -nle'my($a,$b,$c,$d,$e)=split /,/;$ta+=$a, $tb+=$b
+, $tc+=$c, $td+=$d, $te+=$e;END{print join " ", $ta,$tb,$tc,$td,$te}'
+ numbers.csv
2499084140 2499188390 2500073650 2497725180 2495867770
45.270u 0.200s 0:45.44 100.0% 0+0k 0+0io 322pf+0w
Mabooka %
Mabooka % time sum5.cut_n_awk
2499084140 2499188390 2500073650 2497725180 2495867770
18.520u 0.490s 0:12.52 151.8% 0+0k 0+0io 575pf+0w
Mabooka %
, where:
Mabooka % cat sum5.cut_n_awk
#
cat numbers.csv | cut -f1,2,3,4,5 -d, |awk -F, '{s1 += $1; s2 += $2; s
+3+= $3; s4+=$4; s5+=$5} END {printf ("%.0f %.0f %.0f %.0f %.0f\n", s
+1, s2,s3,s4,s5)}'
3-4 times difference isn't bad. Maybe this would help...
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.