Re^2: How to find out, why my perl code is slow.

[Anonymous Monk]

Thanks for your recommendations !!

I modified a lot in my code, it is now about 6 times faster !: * use vec to get the byte from the file string. * removed all "." or ".=" operators in often used places * coded position bytes new (use "&" instead of regex,binary strings and no loops) * pre calculated "for loop of crc" and stored it into a look up table (256 Bytes).

I modified step by step. But: My version with arrays instead of using strings and vec is faster (e.g crc-look up table, no change of array size over the program). Could it be that this is different in win, which I use, and linux ?

In the Profile of -MDevel::NYTProf I can see a big time at the last statement of a subroutine. e.g. 3.8 seconds at the last statement ($fpos++;) of readByte() @ 1.700.000 calls. This time is nearly independent of the statement itself. Inserting a return makes it slower: 4s for the return. The statement is than rated as fast(200ms). Where does this time come from ? Following is not clear for me: I tried to insert the content of the subroutine into the code instead using this subroutine. This version is in the profiler 15% faster, but comparing this two versions without profiler, it shows less than 1% speed difference. So not worth doing it. Any idea what happens here ?

[haukex]

the last statement ($fpos++;) of readByte() ... Any idea what happens here ?

You haven't shown us the code, so we don't know... could you provide a Short, Self-Contained, Correct Example that we could run ourselves? (e.g. with some sample input data)

[Anonymous Monk]

Please find the (simplified) program below. The file is simulated with an array. It is running itself.

#!/usr/bin/perl 
use strict;
use warnings;
$\="\n"; $,=", ";

#------------------------------------------------
my($f, $fpos, $fposlen, @data, $byte1, @byte2, $crc, @crcval); 
initCRC();
ReadDatFile();
exit;

#------------------------------------------------
sub ReadDatFile
{ # simulate file read:
  # three datasets:  0,1,2,3,4,5,6,7,8,9  and  10,11,2,3,4,5,6,7,8,9 a
+nd 20,11,2,3,4,5,6,7,18,19
  my @file=(0x00, 0xc0,0xff,0x80,1,2,3,4,5,6,7,8,9,0x8e,   10,0x80,0x8
+0,11,0x9d,   20,0xc0,0x01,0x80,18,19,0x1c,   0xff);
  $f.=chr($_)|'' foreach (@file);
  
  $crc=255; $fposlen=length($f); $fpos=0;
  @byte2=(0,0,0,0,0,0,0,0);
  readByte(); $data[0]=$_;                              # first data
  
  do
  { readByte(); $byte1=$_;                              # BYTE1: $_
    if ($byte1 & 0x80) {readByte(); $byte2[0]=$_; } 
    if ($byte1 & 0x40) {readByte(); $byte2[1]=$_; }
    if ($byte1 & 0x20) {readByte(); $byte2[2]=$_; }
    if ($byte1 & 0x10) {readByte(); $byte2[3]=$_; } 
    if ($byte1 & 0x08) {readByte(); $byte2[4]=$_; }
    if ($byte1 & 0x04) {readByte(); $byte2[5]=$_; }
    if ($byte1 & 0x02) {readByte(); $byte2[6]=$_; }
    if ($byte1 & 0x01) {readByte(); $byte2[7]=$_; }

    do_byte2( 0) if ($byte1 & 0x80); 
    do_byte2( 8) if ($byte1 & 0x40); 
    do_byte2(16) if ($byte1 & 0x20);
    do_byte2(24) if ($byte1 & 0x10); 
    do_byte2(32) if ($byte1 & 0x08); 
    do_byte2(40) if ($byte1 & 0x04); 
    do_byte2(48) if ($byte1 & 0x02); 
    do_byte2(56) if ($byte1 & 0x01); 
    
    readByte(); if ($crc>0) {print "ERROR: CRC (POS:$fpos)"; exit; }  
+ # crc check
    $crc=255;     

    print @data;           # ---- USE DATA ---------
    
    readByte(); $data[0]=$_;                         # next dataset
  } while ($_<255);
  
}

sub do_byte2
{ my $bpos=$_[0];                                     # pos in data fr
+om byte0
  my $pp=$byte2[$bpos>>3];
  if ($pp & 0x80) {readByte(); $data[$bpos+1]=$_;}    # take value int
+o data array
  if ($pp & 0x40) {readByte(); $data[$bpos+2]=$_;}
  if ($pp & 0x20) {readByte(); $data[$bpos+3]=$_;}
  if ($pp & 0x10) {readByte(); $data[$bpos+4]=$_;}
  if ($pp & 0x08) {readByte(); $data[$bpos+5]=$_;}
  if ($pp & 0x04) {readByte(); $data[$bpos+6]=$_;}
  if ($pp & 0x02) {readByte(); $data[$bpos+7]=$_;}
  if ($pp & 0x01) {readByte(); $data[$bpos+8]=$_;}
}

sub readByte
{ if ($fposlen==$fpos) {print "Last data structure not complete !"; ex
+it;}
  $_=vec($f,$fpos,8); 
  $crc=$crcval[$crc^$_];         # CRC     
  $fpos++;  
}

sub initCRC
{ for my $v (0..255)
  { $_=$v;
    for my $r (1..8)
    { if (($_&1)>0) {$_=($_>>1)^0x8c;}
      else          {$_=$_>>1;}  
    }
    $crcval[$v]=$_;  
} }

[Corion]

Have you looked at Digest::CRC? It has the CRC routine itself written in C, so if you can supply the data to it in large chunks (instead of reading byte by byte), it should be much faster than Perl alone.

[swl]

I've found similar differences in the past between profiling and benchmarking. I assume it is because profiling is run under the debugger and so some optimisations are disabled or there is more bookkeeping involved. Running both profiler and benchmarking is always a sensible process.

As for the last statement taking time, I think this is because it includes the time for the various bookkeeping operations run at the end of the subroutine (e.g. memory cleanup). If you end the sub with "1;" then you should see the actual time taken for the increment (although it might break your program).

Hopefully others will be better able to comment on or clarify the above.