Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re^4: In need of a Dumper that has no pretentions to being anything else.

by BrowserUk (Pope)
on Feb 23, 2005 at 03:20 UTC ( #433560=note: print w/ replies, xml ) Need Help??


in reply to Re^3: In need of a Dumper that has no pretentions to being anything else.
in thread In need of a Dumper that has no pretentions to being anything else.

Okay. Try this:

#! perl -slw use strict; use Data::Dumper; my %h; $h{ $_ } = [ 1 .. 10 ] for 'aaaa' .. 'zzzz'; print Dumper \%h;

Add whatever Dumper options you like. Prior to the Dump, this hash with somewhat under 500,000 keys and a smallish array for each value consumes ~ 177 MB of ram.

Attempting to dump it pushes that memory consumption (transiently on Win32) to well over 700 800 MB (and still going and consumption still climbing after 1/2 3/4 hour!).

My real hash has close to a million keys and nested arrays. It consumes over 500 MB to start with. Trying to dump it blows 2GB of virtual memory before it crashes Perl--and the time taken even before swapping starts is measured in the half-lifes of Plutonium. I'd like to avoid both. I just need to be able to dump the structure to a file. Preferably in a reasonably compact format.


Examine what is said, not who speaks.
Silence betokens consent.
Love the truth but pardon error.


Comment on Re^4: In need of a Dumper that has no pretentions to being anything else.
Download Code
Re^5: In need of a Dumper that has no pretentions to being anything else.
by fergal (Chaplain) on Feb 23, 2005 at 04:19 UTC
    Try the following patch to Data/Dumper.pm . Also, turn off Deepcopy as in my first post. It should make a huge difference. I'll file a bug and submit the patch.
    --- ./ok/Data/Dumper.pm.orig 2005-02-22 20:17:13.000000000 -0800 +++ ./ok/Data/Dumper.pm 2005-02-22 20:16:47.000000000 -0800 @@ -405,7 +405,7 @@ my $ref = \$_[1]; # first, catalog the scalar - if ($name ne '') { + if ($s->{deepcopy} and ($name ne '')) { ($id) = ("$ref" =~ /\(([^\(]*)\)$/); if (exists $s->{seen}{$id}) { if ($s->{seen}{$id}[2]) {
    Updated And do $Data::Dumper::Useperl = 1;

      Hmm. I made the one-line change and turned Deepcopy off. The result was that it still crashed (using the example above). The difference was that the memory consumption grew much more quickly though it may have topped out at a lower or the same point, but it crashes at this point with:

      The instruction at 0x2808756d caused an access violation attempting to access memory at 0x000000004

      The code at the reported location disassembles as:

      Disassembly of Function perl58.dll!Perl_my_socketpair + 0x0F90 (0x2808 +756D) + 0x2808756D: MOV DWORD PTR [EDI+0x4],EAX 0x28087570: MOV DWORD PTR [EDI],ECX 0x28087572: MOV DWORD PTR [EDI+0x8],ESI 0x28087575: MOV DWORD PTR [ECX+0x4],EDI 0x28087578: MOV EAX,DWORD PTR [ESI] 0x2808757A: MOV ECX,ESI 0x2808757C: CALL DWORD PTR [EAX+0x10] 0x2808757F: LEA EAX,DWORD PTR [EDI+0xC] 0x28087584: RET 0x4

      though I suspect that the failure is caused through Perl not detecting that it had been refused a further allocation of memory. I don't seem to get the "Out of memory" message any more? (Since 5.8.4 and/or my moving from NT4 to XP--I am not sure which is responsible?)

      At the pointer of failure, though the stack could have been corrupted by this point, the stack contents are:

      PID: 988 TID: 332 - Stack Contents for 0x2808756D 0x2808756D: perl58.dll:Perl_my_socketpair + 0x0F90 0x2800F40A: perl58.dll:Perl_hv_fetch_ent + 0x091C 0x2800E9EC: perl58.dll:Perl_hv_store + 0x002C 0x10003325: 0x100028A6: 0x10003014: 0x10001A4D: 0x28040F9F: perl58.dll:Perl_sv_compile_2op + 0x7D9C 0x2805F811: perl58.dll:Perl_runops_standard + 0x000C 0x2808A3B1: perl58.dll:RunPerl + 0x0086 0x00401012: perl.exe+0x00001012 0x77E814C7: GetCurrentDirectoryW + 0x0044

      And the registers are:

      PID: 332 TID: 332 - Register Contents at EIP: 0x2808756D EAX: 0x00223FFC EBX: 0x00200000 ECX: 0x525DC288 EDX: 0x00220608 EDI: 0x00000000 ESI: 0x00223FF8 ESP: 0x0140F2FC EBP: 0x0140F34C EIP: 0x2808756D Flags: 0x00010246 o D i s Z a P c

      If you want the same information for the crash with the pre-patched Data::Dumper let me know.

      Update: Golly gee willickers this is fun! Try to be helpful and some miserable toerag who hasn't the guts to stand up and argue his case, vents by downvoting.

      Well get this. If your a saint, it'll only take you another 1000 days to wipe me out. If you use all your votes on downvoting me, every day. And if noone else ever opvotes me. So, you got 2.7 years to waste?


      Examine what is said, not who speaks.
      Silence betokens consent.
      Love the truth but pardon error.
        Sorry, I should have said that for that patch do have any affect at all you need also to do $Data::Dumper::Useperl = 1;. Otherwise you're probably running the C version which still has the bug.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://433560]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (5)
As of 2014-08-02 04:41 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Who would be the most fun to work for?















    Results (54 votes), past polls