Benchmarking chop/substr/split

[larryl]

I tried a few benchmarks in response to discussion around Big Willy's Frequency Analyzer. The code I tested and the results can be summarized by:

• chop: 909.92/s
  

  while ($_ ne '') {
     $letters{lc chop}++;
  }
  

• split_0: 519.41/s
  $letters{$_}++ for split//, lc;
• split_1: 481.23/s
  $letters{lc $_}++ for split//;
• substr: 475.51/s
  

  while ($_ ne '') {
     $letters{lc substr($_,-1)}++;
     substr($_, -1) = "";
  }
  

I have two questions:

1. I'm not that familiar with the for syntax used in the split_[01] tests. Can someone help me mentally parse those, and possibly explain why one would be faster than the other?
   
   
2. It seems odd that chop() benchmarks so much faster than substr($_,-1). I've seen the same in other stuff I've benchmarked. Wouldn't you think the compiler would treat substr($_,-1) as a special case?
(Ok, that was really just one question and one musing...)

[Albannach]

Deparse is cool!:

perl -MO=Deparse -e "$letters{$_}++ for split //, lc;" gives

foreach $_ (split(//, lc $_, 0)) {
    ++$letters{$_};
}

Which I imagine you can understand.

--
I'd like to be able to assign to an luser

[larryl]

Excellent! Thanks, Brother Albannach!

[tye]

Well, I bet it would be possible to teach Perl to optimize substr($str,-1) (when not being assigned to and when the "-1" is a literal constant) into chop. But then, why? That is, would the bit of bloat (and the potential for introducing bugs) be worth allowing people to write chop long-hand without the performance penalty?

BTW, they are all close enough to the same speed that I don't think I'd ever worry about the speed difference. (:

        - tye (but my friends call me "Tye")

[larryl]

True, it might not be worth the bloat for something not frequently used. I'm just in a fuss because I read that gnat wants to take chop() out of the core.  I like chop.  chop is my friend.  I don't want to have to type  $a = substr($_,-1); substr($_,-1) = '' when what I want is  $a = chop.

Pout.

Besides, it's at least twice as fast every place I've tested it. If you need to check a few thousand credit cards a day, it makes a noticeable difference.

[tye]

Yeah, I like chop too. FYI, you don't have to repeat the substr: $a= substr($_,-1,1,""); Though this doesn't work in older versions of Perl.

        - tye (but my friends call me "Tye")

[MeowChow]

If efficiency is your primary concern, then use an array instead of a hash:

$letters[ord chop]++ while $_;

I also intentionally left out the lc function. Frequency counts can be consolidated at the end of the loop, instead of performing an lc upon every iteration. Save yourself the O(n) step, and get more detailed frequency information while you're at it.

   MeowChow                                   
               s aamecha.s a..a\u$&owag.print

[larryl]

Update: A new data point based on tye's suggestion


• substr_1: 714.54/s
  

  while ($_ ne '') {
     $letters{lc substr($_,-1,1,'')}++;
  }