comment on

Does anyone know what the performance penalty of utf8 hash keys -- even if they contain only ASCII chars -- is?

Benchmarking script:

use strict;
use warnings;
use Encode qw( _utf8_on );
use Benchmark::Timer;
use Devel::Peek;

my $timer = Benchmark::Timer->new;

my $latin1 = "foo";
my $utf8   = "foo";
_utf8_on($utf8);
my %latin1_hash = ( $latin1 => 1 );
my %utf8_hash   = ( $utf8   => 1 );
warn "===================================\n";
warn "Latin-1 hash key:\n----------------------------------\n";
Dump \%latin1_hash;
warn "===================================\n";
warn "UTF-8 hash key:\n-----------------------------------\n";
Dump \%utf8_hash;
warn "===================================\n";

for my $iter ( 1 .. 10 ) { 
    my $total = 0;
    $timer->start("Latin-1 hash key, Latin-1 probe");
    $total += $latin1_hash{$latin1} for 1 .. 1_000_000;
    $timer->stop("Latin-1 hash key, Latin-1 probe");

    $total = 0;
    $timer->start("Latin-1 hash key, UTF-8 probe");
    $total += $utf8_hash{$latin1} for 1 .. 1_000_000;
    $timer->stop("Latin-1 hash key, UTF-8 probe");

    $total = 0;
    $timer->start("UTF-8 hash key, Latin-1 probe");
    $total += $latin1_hash{$utf8} for 1 .. 1_000_000;
    $timer->stop("UTF-8 hash key, Latin-1 probe");

    $total = 0;
    $timer->start("UTF-8 hash key, UTF-8 probe");
    $total += $utf8_hash{foo} for 1 .. 1_000_000;
    $timer->stop("UTF-8 hash key, UTF-8 probe");
}

warn scalar $timer->reports . "\n";
[download]

Results for vanilla custom-compiled Perl 5.10.0 on Mac OS X:

marvin@smokie:~/perltest $ perl hash_key_bench.pl 
===================================
Latin-1 hash key:
----------------------------------
SV = RV(0x8503ac) at 0x8503a0
  REFCNT = 1
  FLAGS = (TEMP,ROK)
  RV = 0x84df80
  SV = PVHV(0x809048) at 0x84df80
    REFCNT = 2
    FLAGS = (PADMY,SHAREKEYS)
    ARRAY = 0x200f60  (0:7, 1:1)
    hash quality = 100.0%
    KEYS = 1
    FILL = 1
    MAX = 7
    RITER = -1
    EITER = 0x0
    Elt "foo" HASH = 0x238678dd
    SV = IV(0x85036c) at 0x850370
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 1
===================================
UTF-8 hash key:
-----------------------------------
SV = RV(0x8503ac) at 0x8503a0
  REFCNT = 1
  FLAGS = (TEMP,ROK)
  RV = 0x84df40
  SV = PVHV(0x80905c) at 0x84df40
    REFCNT = 2
    FLAGS = (PADMY,SHAREKEYS,HASKFLAGS)
    ARRAY = 0x200f80  (0:7, 1:1)
    hash quality = 100.0%
    KEYS = 1
    FILL = 1
    MAX = 7
    RITER = -1
    EITER = 0x0
    Elt "foo" [UTF8 "foo"] HASH = 0x238678dd
    SV = IV(0x85038c) at 0x850390
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 1
===================================
Results:
-----------------------------------
10 trials of Latin-1 hash key, Latin-1 probe (2.272s total), 227.159ms
+/trial
10 trials of Latin-1 hash key, UTF-8 probe (2.248s total), 224.823ms/t
+rial
10 trials of UTF-8 hash key, Latin-1 probe (3.100s total), 309.985ms/t
+rial
10 trials of UTF-8 hash key, UTF-8 probe (1.893s total), 189.301ms/tri
+al
[download]

In reply to Re: Unexpected utf8 in hash keys by creamygoodness
in thread Unexpected utf8 in hash keys by kappa

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


XP is just a number
	PerlMonks