Re: Tokenising a 10MB file trashes a 2GB machine

[Anonymous Monk]

use re 'debug'; and see if it sheds any light, this replicates your results

perl -Mre=debug -e"$f = q~a b c~ x 1;$g = [split m{\p{IsSpace}}ms, $f 
+];" 


perl -e"die 1E7"

perl -Mre=debug -e"$f = q~a b c~ x 1E7;$g = [split m{\p{IsSpace}}ms, $
+f ];" 2>2

[PetaMem]

perl -Mre=debug -e"$f = q~a b c~ x 1E4;$g = [split m{\p{IsSpace}}ms, $
+f ];" 2>2

The sheer size of the debugger output makes it impossible to run with the 1E7 multiplier and although I still do not know how to interpret the output, maybe someone here knows.

Multiplier	Size of debugger output
1	14KiB
10	21KiB
100	137KiB
1E3	5,7MiB
1E4	507MiB
1E5	50GiB

Therefore I predict output of the debugger would be (at least) about 5TiB for 1E6. The size comes from the fact, that there is always a printout of the complete dataset that will be matched against, which is every time the regexp matches shortened by one token. Therefore the numbers mentioned above halve if we have e.g. q{1234 } instead q{a b c}.

In between these printouts there is always the same output:

Matching REx "\p{IsSpace}" against "1234 1234 1234 1234 1234 1234 1234
+ "
Matching stclass "ANYOF[{unicode}\11-\15 \302\205\302\240...+utf8::IsS
+pace]" against "1234 1234 1234 1234 1234 1234 1234 "
  Setting an EVAL scope, savestack=6
49969 < 1234> < 1234 1>    |  1:  ANYOF[{unicode}\11-\15 \302\205\302\
+240...+utf8::IsSpace]
49970 <1234 > <1234 12>    | 13:  END
Match successful!
Matching REx "\p{IsSpace}" against "1234 1234 1234 1234 1234 1234 "
Matching stclass "ANYOF[{unicode}\11-\15 \302\205\302\240...+utf8::IsS
+pace]" against "1234 1234 1234 1234 1234 1234 "
  Setting an EVAL scope, savestack=6
49974 < 1234> < 1234 1>    |  1:  ANYOF[{unicode}\11-\15 \302\205\302\
+240...+utf8::IsSpace]
49975 <1234 > <1234 12>    | 13:  END
Match successful!
Matching REx "\p{IsSpace}" against "1234 1234 1234 1234 1234 "
Matching stclass "ANYOF[{unicode}\11-\15 \302\205\302\240...+utf8::IsS
+pace]" against "1234 1234 1234 1234 1234 "
  Setting an EVAL scope, savestack=6
49979 < 1234> < 1234 1>    |  1:  ANYOF[{unicode}\11-\15 \302\205\302\
+240...+utf8::IsSpace]
49980 <1234 > <1234 12>    | 13:  END
Match successful!
Matching REx "\p{IsSpace}" against "1234 1234 1234 1234 "

(this is taken from near the end of the debugger output to keep the size of the data sections small)

So unfortunately I do not see much from this output that could give me a hint for the additional memory consumption. Except probably the "savestack=6", but I guess that is on every other perl interpreter the same. I'll try to compile Perl conservatively with an old GCC and generic CPU architecture (maybe the new gcc does some wasting alignments for Core2 architecture).