Re^4: In need of a Dumper that has no pretentions to being anything else.

[BrowserUk]

Okay. Try this:

#! perl -slw
use strict;
use Data::Dumper;

my %h;
$h{ $_ } = [ 1 .. 10 ] for 'aaaa' .. 'zzzz';
print Dumper \%h;

Add whatever Dumper options you like. Prior to the Dump, this hash with somewhat under 500,000 keys and a smallish array for each value consumes ~ 177 MB of ram.

Attempting to dump it pushes that memory consumption (transiently on Win32) to well over 700 800 MB (and still going and consumption still climbing after 1/2 3/4 hour!).

My real hash has close to a million keys and nested arrays. It consumes over 500 MB to start with. Trying to dump it blows 2GB of virtual memory before it crashes Perl--and the time taken even before swapping starts is measured in the half-lifes of Plutonium. I'd like to avoid both. I just need to be able to dump the structure to a file. Preferably in a reasonably compact format.

[fergal]

Try the following patch to Data/Dumper.pm . Also, turn off Deepcopy as in my first post. It should make a huge difference. I'll file a bug and submit the patch.

--- ./ok/Data/Dumper.pm.orig    2005-02-22 20:17:13.000000000 -0800
+++ ./ok/Data/Dumper.pm    2005-02-22 20:16:47.000000000 -0800
@@ -405,7 +405,7 @@
 
     my $ref = \$_[1];
     # first, catalog the scalar
-    if ($name ne '') {
+    if ($s->{deepcopy} and ($name ne '')) {
       ($id) = ("$ref" =~ /\(([^\(]*)\)$/);
       if (exists $s->{seen}{$id}) {
         if ($s->{seen}{$id}[2]) {

Updated And do $Data::Dumper::Useperl = 1;

[BrowserUk]

Hmm. I made the one-line change and turned Deepcopy off. The result was that it still crashed (using the example above). The difference was that the memory consumption grew much more quickly though it may have topped out at a lower or the same point, but it crashes at this point with:

The instruction at 0x2808756d caused an access violation attempting to access memory at 0x000000004

The code at the reported location disassembles as:

Disassembly of Function perl58.dll!Perl_my_socketpair + 0x0F90 (0x2808
+756D)
+ 0x2808756D: MOV      DWORD PTR [EDI+0x4],EAX            
  0x28087570: MOV      DWORD PTR [EDI],ECX                
  0x28087572: MOV      DWORD PTR [EDI+0x8],ESI            
  0x28087575: MOV      DWORD PTR [ECX+0x4],EDI            
  0x28087578: MOV      EAX,DWORD PTR [ESI]                
  0x2808757A: MOV      ECX,ESI                            
  0x2808757C: CALL     DWORD PTR [EAX+0x10]               
  0x2808757F: LEA      EAX,DWORD PTR [EDI+0xC]            
  0x28087584: RET      0x4

though I suspect that the failure is caused through Perl not detecting that it had been refused a further allocation of memory. I don't seem to get the "Out of memory" message any more? (Since 5.8.4 and/or my moving from NT4 to XP--I am not sure which is responsible?)

At the pointer of failure, though the stack could have been corrupted by this point, the stack contents are:

PID: 988 TID: 332 - Stack Contents for 0x2808756D
0x2808756D: perl58.dll:Perl_my_socketpair + 0x0F90
0x2800F40A: perl58.dll:Perl_hv_fetch_ent + 0x091C
0x2800E9EC: perl58.dll:Perl_hv_store + 0x002C
0x10003325: 
0x100028A6: 
0x10003014: 
0x10001A4D: 
0x28040F9F: perl58.dll:Perl_sv_compile_2op + 0x7D9C
0x2805F811: perl58.dll:Perl_runops_standard + 0x000C
0x2808A3B1: perl58.dll:RunPerl + 0x0086
0x00401012: perl.exe+0x00001012
0x77E814C7: GetCurrentDirectoryW + 0x0044

And the registers are:

PID: 332 TID: 332 - Register Contents at EIP: 0x2808756D
EAX: 0x00223FFC
EBX: 0x00200000
ECX: 0x525DC288
EDX: 0x00220608
EDI: 0x00000000
ESI: 0x00223FF8
ESP: 0x0140F2FC
EBP: 0x0140F34C
EIP: 0x2808756D
Flags: 0x00010246 o D i s Z a P c

If you want the same information for the crash with the pre-patched Data::Dumper let me know.

Update: Golly gee willickers this is fun! Try to be helpful and some miserable toerag who hasn't the guts to stand up and argue his case, vents by downvoting.

Well get this. If your a saint, it'll only take you another 1000 days to wipe me out. If you use all your votes on downvoting me, every day. And if noone else ever opvotes me. So, you got 2.7 years to waste?

[fergal]

Sorry, I should have said that for that patch do have any affect at all you need also to do $Data::Dumper::Useperl = 1;. Otherwise you're probably running the C version which still has the bug.