Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Re^3: In need of a Dumper that has no pretentions to being anything else.

by fergal (Chaplain)
on Feb 23, 2005 at 02:19 UTC ( #433550=note: print w/replies, xml ) Need Help??

in reply to Re^2: In need of a Dumper that has no pretentions to being anything else.
in thread In need of a Dumper that has no pretentions to being anything else.

That sounds like a serious bug in DD. In order to catch circularity it only needs to keep a 4 byte hash key and a short string for every reference in the structure. Unless your structure is full of almost empty arrays, hashes and scalar refs, this should take less memory than your structure.

Update: Just dug in DD and I see it's storing a 2 element arrayref for each ref it find. That's still not very much and unless you have an unusual structure, it should be negligible. The only other thing is that it stores a copy of the hash key in that 2 element array. If your keys are very big then that could be a problem however you'd still only be at most doubling things.

Update again It was a DD bug, see below

  • Comment on Re^3: In need of a Dumper that has no pretentions to being anything else.

Replies are listed 'Best First'.
Re^4: In need of a Dumper that has no pretentions to being anything else.
by BrowserUk (Pope) on Feb 23, 2005 at 03:20 UTC

    Okay. Try this:

    #! perl -slw use strict; use Data::Dumper; my %h; $h{ $_ } = [ 1 .. 10 ] for 'aaaa' .. 'zzzz'; print Dumper \%h;

    Add whatever Dumper options you like. Prior to the Dump, this hash with somewhat under 500,000 keys and a smallish array for each value consumes ~ 177 MB of ram.

    Attempting to dump it pushes that memory consumption (transiently on Win32) to well over 700 800 MB (and still going and consumption still climbing after 1/2 3/4 hour!).

    My real hash has close to a million keys and nested arrays. It consumes over 500 MB to start with. Trying to dump it blows 2GB of virtual memory before it crashes Perl--and the time taken even before swapping starts is measured in the half-lifes of Plutonium. I'd like to avoid both. I just need to be able to dump the structure to a file. Preferably in a reasonably compact format.

    Examine what is said, not who speaks.
    Silence betokens consent.
    Love the truth but pardon error.
      Try the following patch to Data/ . Also, turn off Deepcopy as in my first post. It should make a huge difference. I'll file a bug and submit the patch.
      --- ./ok/Data/ 2005-02-22 20:17:13.000000000 -0800 +++ ./ok/Data/ 2005-02-22 20:16:47.000000000 -0800 @@ -405,7 +405,7 @@ my $ref = \$_[1]; # first, catalog the scalar - if ($name ne '') { + if ($s->{deepcopy} and ($name ne '')) { ($id) = ("$ref" =~ /\(([^\(]*)\)$/); if (exists $s->{seen}{$id}) { if ($s->{seen}{$id}[2]) {
      Updated And do $Data::Dumper::Useperl = 1;

        Hmm. I made the one-line change and turned Deepcopy off. The result was that it still crashed (using the example above). The difference was that the memory consumption grew much more quickly though it may have topped out at a lower or the same point, but it crashes at this point with:

        The instruction at 0x2808756d caused an access violation attempting to access memory at 0x000000004

        The code at the reported location disassembles as:

        Disassembly of Function perl58.dll!Perl_my_socketpair + 0x0F90 (0x2808 +756D) + 0x2808756D: MOV DWORD PTR [EDI+0x4],EAX 0x28087570: MOV DWORD PTR [EDI],ECX 0x28087572: MOV DWORD PTR [EDI+0x8],ESI 0x28087575: MOV DWORD PTR [ECX+0x4],EDI 0x28087578: MOV EAX,DWORD PTR [ESI] 0x2808757A: MOV ECX,ESI 0x2808757C: CALL DWORD PTR [EAX+0x10] 0x2808757F: LEA EAX,DWORD PTR [EDI+0xC] 0x28087584: RET 0x4

        though I suspect that the failure is caused through Perl not detecting that it had been refused a further allocation of memory. I don't seem to get the "Out of memory" message any more? (Since 5.8.4 and/or my moving from NT4 to XP--I am not sure which is responsible?)

        At the pointer of failure, though the stack could have been corrupted by this point, the stack contents are:

        PID: 988 TID: 332 - Stack Contents for 0x2808756D 0x2808756D: perl58.dll:Perl_my_socketpair + 0x0F90 0x2800F40A: perl58.dll:Perl_hv_fetch_ent + 0x091C 0x2800E9EC: perl58.dll:Perl_hv_store + 0x002C 0x10003325: 0x100028A6: 0x10003014: 0x10001A4D: 0x28040F9F: perl58.dll:Perl_sv_compile_2op + 0x7D9C 0x2805F811: perl58.dll:Perl_runops_standard + 0x000C 0x2808A3B1: perl58.dll:RunPerl + 0x0086 0x00401012: perl.exe+0x00001012 0x77E814C7: GetCurrentDirectoryW + 0x0044

        And the registers are:

        PID: 332 TID: 332 - Register Contents at EIP: 0x2808756D EAX: 0x00223FFC EBX: 0x00200000 ECX: 0x525DC288 EDX: 0x00220608 EDI: 0x00000000 ESI: 0x00223FF8 ESP: 0x0140F2FC EBP: 0x0140F34C EIP: 0x2808756D Flags: 0x00010246 o D i s Z a P c

        If you want the same information for the crash with the pre-patched Data::Dumper let me know.

        Update: Golly gee willickers this is fun! Try to be helpful and some miserable toerag who hasn't the guts to stand up and argue his case, vents by downvoting.

        Well get this. If your a saint, it'll only take you another 1000 days to wipe me out. If you use all your votes on downvoting me, every day. And if noone else ever opvotes me. So, you got 2.7 years to waste?

        Examine what is said, not who speaks.
        Silence betokens consent.
        Love the truth but pardon error.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://433550]
LanX recommends inner immigration, work less hard and enjoy pay roll
[MidLifeXis]: ahh, yeah, didn't look through nodes yet today.
[erix]: haha
[MidLifeXis]: Ahh well, time to get back to it, so there isn't a reason to take that choice out of my hands :-)
[LanX]: Inner emigration

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (12)
As of 2017-03-23 12:53 GMT
Find Nodes?
    Voting Booth?
    Should Pluto Get Its Planethood Back?

    Results (286 votes). Check out past polls.