Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

Re^2: How to find out, why my perl code is slow.

by Anonymous Monk
on Nov 05, 2018 at 18:31 UTC ( #1225282=note: print w/replies, xml ) Need Help??

in reply to Re: How to find out, why my perl code is slow.
in thread How to find out, why my perl code is slow.

Thanks for your recommendations !!

I modified a lot in my code, it is now about 6 times faster !: * use vec to get the byte from the file string. * removed all "." or ".=" operators in often used places * coded position bytes new (use "&" instead of regex,binary strings and no loops) * pre calculated "for loop of crc" and stored it into a look up table (256 Bytes).

I modified step by step. But: My version with arrays instead of using strings and vec is faster (e.g crc-look up table, no change of array size over the program). Could it be that this is different in win, which I use, and linux ?

In the Profile of -MDevel::NYTProf I can see a big time at the last statement of a subroutine. e.g. 3.8 seconds at the last statement ($fpos++;) of readByte() @ 1.700.000 calls. This time is nearly independent of the statement itself. Inserting a return makes it slower: 4s for the return. The statement is than rated as fast(200ms). Where does this time come from ? Following is not clear for me: I tried to insert the content of the subroutine into the code instead using this subroutine. This version is in the profiler 15% faster, but comparing this two versions without profiler, it shows less than 1% speed difference. So not worth doing it. Any idea what happens here ?

  • Comment on Re^2: How to find out, why my perl code is slow.

Replies are listed 'Best First'.
Re^3: How to find out, why my perl code is slow.
by haukex (Canon) on Nov 05, 2018 at 18:48 UTC
    the last statement ($fpos++;) of readByte() ... Any idea what happens here ?

    You haven't shown us the code, so we don't know... could you provide a Short, Self-Contained, Correct Example that we could run ourselves? (e.g. with some sample input data)

      Please find the (simplified) program below. The file is simulated with an array. It is running itself.

      #!/usr/bin/perl use strict; use warnings; $\="\n"; $,=", "; #------------------------------------------------ my($f, $fpos, $fposlen, @data, $byte1, @byte2, $crc, @crcval); initCRC(); ReadDatFile(); exit; #------------------------------------------------ sub ReadDatFile { # simulate file read: # three datasets: 0,1,2,3,4,5,6,7,8,9 and 10,11,2,3,4,5,6,7,8,9 a +nd 20,11,2,3,4,5,6,7,18,19 my @file=(0x00, 0xc0,0xff,0x80,1,2,3,4,5,6,7,8,9,0x8e, 10,0x80,0x8 +0,11,0x9d, 20,0xc0,0x01,0x80,18,19,0x1c, 0xff); $f.=chr($_)|'' foreach (@file); $crc=255; $fposlen=length($f); $fpos=0; @byte2=(0,0,0,0,0,0,0,0); readByte(); $data[0]=$_; # first data do { readByte(); $byte1=$_; # BYTE1: $_ if ($byte1 & 0x80) {readByte(); $byte2[0]=$_; } if ($byte1 & 0x40) {readByte(); $byte2[1]=$_; } if ($byte1 & 0x20) {readByte(); $byte2[2]=$_; } if ($byte1 & 0x10) {readByte(); $byte2[3]=$_; } if ($byte1 & 0x08) {readByte(); $byte2[4]=$_; } if ($byte1 & 0x04) {readByte(); $byte2[5]=$_; } if ($byte1 & 0x02) {readByte(); $byte2[6]=$_; } if ($byte1 & 0x01) {readByte(); $byte2[7]=$_; } do_byte2( 0) if ($byte1 & 0x80); do_byte2( 8) if ($byte1 & 0x40); do_byte2(16) if ($byte1 & 0x20); do_byte2(24) if ($byte1 & 0x10); do_byte2(32) if ($byte1 & 0x08); do_byte2(40) if ($byte1 & 0x04); do_byte2(48) if ($byte1 & 0x02); do_byte2(56) if ($byte1 & 0x01); readByte(); if ($crc>0) {print "ERROR: CRC (POS:$fpos)"; exit; } + # crc check $crc=255; print @data; # ---- USE DATA --------- readByte(); $data[0]=$_; # next dataset } while ($_<255); } sub do_byte2 { my $bpos=$_[0]; # pos in data fr +om byte0 my $pp=$byte2[$bpos>>3]; if ($pp & 0x80) {readByte(); $data[$bpos+1]=$_;} # take value int +o data array if ($pp & 0x40) {readByte(); $data[$bpos+2]=$_;} if ($pp & 0x20) {readByte(); $data[$bpos+3]=$_;} if ($pp & 0x10) {readByte(); $data[$bpos+4]=$_;} if ($pp & 0x08) {readByte(); $data[$bpos+5]=$_;} if ($pp & 0x04) {readByte(); $data[$bpos+6]=$_;} if ($pp & 0x02) {readByte(); $data[$bpos+7]=$_;} if ($pp & 0x01) {readByte(); $data[$bpos+8]=$_;} } sub readByte { if ($fposlen==$fpos) {print "Last data structure not complete !"; ex +it;} $_=vec($f,$fpos,8); $crc=$crcval[$crc^$_]; # CRC $fpos++; } sub initCRC { for my $v (0..255) { $_=$v; for my $r (1..8) { if (($_&1)>0) {$_=($_>>1)^0x8c;} else {$_=$_>>1;} } $crcval[$v]=$_; } }

        Have you looked at Digest::CRC? It has the CRC routine itself written in C, so if you can supply the data to it in large chunks (instead of reading byte by byte), it should be much faster than Perl alone.

Re^3: How to find out, why my perl code is slow.
by swl (Hermit) on Nov 06, 2018 at 00:54 UTC

    I've found similar differences in the past between profiling and benchmarking. I assume it is because profiling is run under the debugger and so some optimisations are disabled or there is more bookkeeping involved. Running both profiler and benchmarking is always a sensible process.

    As for the last statement taking time, I think this is because it includes the time for the various bookkeeping operations run at the end of the subroutine (e.g. memory cleanup). If you end the sub with "1;" then you should see the actual time taken for the increment (although it might break your program).

    Hopefully others will be better able to comment on or clarify the above.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1225282]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (3)
As of 2018-12-18 23:31 GMT
Find Nodes?
    Voting Booth?
    How many stories does it take before you've heard them all?

    Results (83 votes). Check out past polls.