Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

Re: Tokenising a 10MB file trashes a 2GB machine

by BrowserUk (Patriarch)
on Jul 16, 2008 at 09:10 UTC ( [id://697905]=note: print w/replies, xml ) Need Help??


in reply to Tokenising a 10MB file trashes a 2GB machine

Be aware that using Devel::Size::total_size() itself can consume prodigous amounts of memory.

In the process of examining the structure of the anonymous array you are creating, it builds a 'tracking hash' to allow it to avoid counting duplicate references to data or internal magic etc. Whether that's the cause of your memory consumption here is not clear, but it would be worth eliminating Devel::Size from the equation.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^2: Tokenising a 10MB file trashes a 2GB machine
by PetaMem (Priest) on Jul 16, 2008 at 09:23 UTC

    It should be obvious, that I added Devel::Size AFTER I found my script trashing the comp. I wanted to write in my original post that Devel::Size acts completely neutral in this case but found that information redundant.

    Bye
     PetaMem
        All Perl:   MT, NLP, NLU

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://697905]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others examining the Monastery: (5)
As of 2024-04-24 10:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found