Re: Parsing 12GB Entourage database in pieces...

[betterworld]

There might be a completely different solution. Using Sys::Mmap, you can map the entire file to a single string, and then brute-force through that string with something like while ($string =~ m/.../g) {...} (like you hinted in your post).

However, there are some caveats:

• Not all operating systems support mmap;
• you'd need a 64 bit system to fit those 12GB into your program's address space;
• I don't know how regular expressions perform with such a huge string. I've just tried it on a mmapped 200MB file (most of which consisted of \0) and it did quite well, but that's much smaller than your file.

[ikegami]

I don't know how regular expressions perform with such a huge string.

Some uses of "*" are equivalent to "{0,32767}", so you might have problems.

>perl -Mre=debug -we"qr/^(.)(\1*)\z/"
...
   9:   CURLYX[1] {0,32767}(14)
...

Be sure to prevent backtracking using (?>...) or (in 5.10.0+) the possessive quantifier.

Update: "/\0\0MSrc.{16}((?>[^\0]*))(?=\0)/s" looks safe.

[betterworld]

Interesting...

perl5.8.8 -wle '$s = "x" x 40_000;
    $s =~ /^(.)(\1*)/ and print length $2'

# (segfaults)

perl5.10.0 -wle '$s = "x" x 40_000;
    $s =~ /^(.)(\1*)/ and print length $2'
Complex regular subexpression recursion limit (32766) exceeded at -e l
+ine 1.
32767

Well, at least it seems to warn when this limitation affects the result...

[ikegami]

I believe p5p is working on a patch to change that warning to a die.