As for the internals of the next method, I haven't run benchmark, but the basic approaches I would consider are the substr approach, splitting the string into an internal array you shift off, and using a /(.)/gs pattern match. (Warning on the last. Use of $' etc anywhere will slow that down. This may be a good reason to avoid no matter what benchmarking says.) I assume you have tried all of them? (Probably but it doesn't hurt to check.)

An incidental note. Your goto can be removed from next with perceivable performance change. Make the if condition be empty, put an else around throwing the exception, and then move the BUFFERED_READ section after the decision logic. This should also be marginally faster because Perl doesn't have to spent time figuring out where the goto goes. (That shouldn't be the common path though, so the change should be marginal.)

I am betting that 5.005 performance is not a priority of yours. But if you are allowing the logic to skip going to next anyways, you can replace a lot of the 5.005 logic with something like the following (untested):

my $len = $n >= 0xFC ? 5
    : $n >= 0xF8 ? 4
    : $n >= 0xF0 ? 3
    : $n >= 0xE0 ? 2
    : $n >= 0xC0 ? 1
    : throw XML::SAX::Exception::Parse(
        Message => sprintf("Invalid character 0x%x", $n),
            ColumnNumber => $self->column,
            LineNumber => $self->line,
            PublicId => $self->public_id,
            SystemId => $self->system_id,
      );

if ($len <= length($self->[BUFFER])) {
    $current .= substr($self->[BUFFER], 0, $len, '');
    $self->[CURRENT] = substr($self->[BUFFER], -1);
}
else {
    $len -= length($self->[BUFFER]);
    $current .= $self->[BUFFER];
    $self->[BUFFER] = '';
    while (-1 < --$len) {
        next($self);
        $current .= $self->[CURRENT];
    }
}
[download]

If you are aiming for performance, my first question would be whether you need next and nextchar to be overridable. Write them as function calls and you will get a significant speedup.

Actually that's more of a myth than a truism. I did try it, but the speedup wasn't significant, though it varies from perl to perl.

However I think there may be some value in the buffer check being moved to nextchar. I was originally thinking it needed to be in next() because sometimes next() is called on its own (for the encoding detection routines which need a byte-by-byte view), but that read()s in character by character anyway, so it might be a reasonable optimisation.

You're right, I have tried all of the various "give me a character" methods, and substr() comes out on top. Which is a pain in the ass really - it's one point where Perl loses out to python where you can do string[0] to get the first character, just like you can in C.

I'll leave the goto as is for now. It's not as bad as people make out - it's only bad when it's used for all flow control, and I think it's intention is quite clear here. Plus given a 1024 buffer, it's only part of the path 70 times in the parse of this 70K XML file.

I do like that last optimisation though - that coupled with moving the buffer test into nextchar() might make a big difference (though maybe not as I don't think the particular test file in question has any UTF-8 characters in it). I'll try it and come back and let you know.

YMMV, but when I tested it on my machine the popular myth was correct, a method call was massively slower than a function call. (Of course if you do anything interesting in your function...)

Speaking of functions, I am kind of wondering what the purpose of next is. It seems from the code I see in it that it was intended to keep track of things like line numbers. But I don't see the rest of the code that would be needed to do that. (Sign of a change in design?) If that is the case, then what next is really providing is buffering.

But isn't buffering exactly what read is supposed to do for you? OK, its speed is highly platform (and compilation option) dependent. But it seems to me that either you are better off using read, or else next should remove the additional buffering by using sysread.

dave hj~

I thought so too, until I added up the percentages.

Hola, Matts, all.

This probably isn't as big an improvement as tilly's above, but you can avoid a branch if you check $] at compiletime using a BEGIN block and play a little symref magic to have two different nextchar routines.

Also, though using lexical variables is generaly a win because of maintainablity, if you're trying to get the last ounce of performance, try declaring your 'my' variables at the widest scope you can (file scope), and seeing how much that helps. IIRC, creating a lexical is expensive.

Try running a line-level profiler, or even running a profiler on perl instead of your program.

For comparison, here are the pre-optimization figures for perl-5.6.1, XML::SAX::PurePerl-0.08 from CPAN:

$ perl -d:DProf t/16large.t
1..3
ok 1
ok 2
parsed 80085 bytes in 13 seconds
ok 3
$ dprofpp -O 6
Total Elapsed Time = 11.05672 Seconds
  User+System Time = 10.67672 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 36.9   3.940  3.892 115004   0.0000 0.0000  XML::SAX::PurePerl::Reade
+r::match_nonext
 14.8   1.590  1.547  86892   0.0000 0.0000  XML::SAX::PurePerl::Reade
+r::Stream::next
 10.8   1.160  8.197 114995   0.0000 0.0001  XML::SAX::PurePerl::Reade
+r::match
 9.55   1.020  7.152 115004   0.0000 0.0001  XML::SAX::PurePerl::Reade
+r::match_nocheck
 9.27   0.990  2.450  86891   0.0000 0.0000  XML::SAX::PurePerl::Reade
+r::nextchar
 9.05   0.966  8.389  17607   0.0001 0.0005  XML::SAX::PurePerl::Reade
+r::consume
[download]

Same machine, same perl, cvs version of fifteen minutes ago:

$ perl -d:DProf t/16large.t
1..3
ok 1
ok 2
parsed 80085 bytes in 16 seconds
ok 3
$ dprofpp -O 6
Total Elapsed Time = 13.08672 Seconds
  User+System Time = 11.91672 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 37.5   4.480  4.442 115004   0.0000 0.0000  XML::SAX::PurePerl::Reade
+r::match_nonext
 13.2   1.580  1.537  86892   0.0000 0.0000  XML::SAX::PurePerl::Reade
+r::Stream::next
 10.6   1.270  9.037 114995   0.0000 0.0001  XML::SAX::PurePerl::Reade
+r::match
 10.2   1.220  7.882 115004   0.0000 0.0001  XML::SAX::PurePerl::Reade
+r::match_nocheck
 9.02   1.075  9.239  17607   0.0001 0.0005  XML::SAX::PurePerl::Reade
+r::consume
 8.48   1.010  2.460  86891   0.0000 0.0000  XML::SAX::PurePerl::Reade
+r::nextchar
[download]

After Compline,
Zaxo

I also wonder if it would help to emulate walking through a string with a char pointer as in C -- instead of substr, keep around a variable holding the position in the buffer.

If you think either of these could help, I'd be happy to grab the CVS and play around a bit.

#!/usr/bin/perl
use strict;
use warnings;

use Benchmark qw/cmpthese/;

print "== Single subroutine argument ==\n";

sub b1_shift  { my $x = shift; $x . ''; }
sub b1_list   { my ($x) = @_;  $x . ''; }
sub b1_direct { $_[0] . ''; }

cmpthese( -5, {
    'shift'           => sub { b1_shift ('X' x rand 5000) },
    'list assignment' => sub { b1_list  ('X' x rand 5000) },
    'direct'          => sub { b1_direct('X' x rand 5000) }
});

print "== Two subroutine arguments ==\n";

sub b2_shift  { my $x = shift; my $y = shift; $x . $y; }
sub b2_list   { my ($x, $y) = @_;  $x . $y; }
sub b2_direct { $_[0] . $_[1]; }

cmpthese( -5, {
    'shift'           => sub { b2_shift ('X' x rand 5000, 'Y' x rand 5
+000) },
    'list assignment' => sub { b2_list  ('X' x rand 5000, 'Y' x rand 5
+000) },
    'direct'          => sub { b2_direct('X' x rand 5000, 'Y' x rand 5
+000) }
});


__END__
== Single subroutine argument ==
Benchmark: running direct, list assignment, shift, each for at least 5
+ CPU seconds...
    direct:  5 wallclock secs ( 5.10 usr +  0.02 sys =  5.12 CPU) @ 11
+0247.85/s (n=564469)
list assignment:  5 wallclock secs ( 5.14 usr +  0.03 sys =  5.17 CPU)
+ @ 89861.90/s (n=464586)
     shift:  6 wallclock secs ( 5.72 usr +  0.03 sys =  5.75 CPU) @ 70
+120.00/s (n=403190)
                    Rate           shift list assignment          dire
+ct
shift            70120/s              --            -22%            -3
+6%
list assignment  89862/s             28%              --            -1
+8%
direct          110248/s             57%             23%              
+--
== Two subroutine arguments ==
Benchmark: running direct, list assignment, shift, each for at least 5
+ CPU seconds...
    direct:  5 wallclock secs ( 5.46 usr +  0.00 sys =  5.46 CPU) @ 57
+434.25/s (n=313591)
list assignment:  6 wallclock secs ( 5.30 usr +  0.02 sys =  5.32 CPU)
+ @ 46535.53/s (n=247569)
     shift:  6 wallclock secs ( 5.00 usr +  0.01 sys =  5.01 CPU) @ 42
+526.95/s (n=213060)
                   Rate           shift list assignment          direc
+t
shift           42527/s              --             -9%            -26
+%
list assignment 46536/s              9%              --            -19
+%
direct          57434/s             35%             23%              -
+-
[download]

2;0 juerd@ouranos:~$ perl -e'undef christmas'
Segmentation fault
2;139 juerd@ouranos:~$
[download]