Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Re: memory usage and leakage

by Sewi (Friar)
on Jun 23, 2011 at 11:06 UTC ( #911057=note: print w/replies, xml ) Need Help??


in reply to memory usage and leakage

The VM value is the address space allocated by an application, not the memory used. It doesn't give any reasonable result at all.

RSS is the memory actually used and this is not shared across instances. The sum of RSS of all processes is the memory (including swap) used on your computer. Add kernel memory, disk cache and free and you got all of your memory.

If you want to share some memory, "use" all required modules and start threading, but you need to check if all modules are thread-safe and ensure your code to be thread-safe.

If you care about CPU or memory usage, remove PEO and Moose from your project as they eat up half of the scripts memory.

Replies are listed 'Best First'.
Re^2: memory usage and leakage
by BrowserUk (Pope) on Jun 23, 2011 at 12:15 UTC
    If you want to share some memory, "use" all required modules and start threading,

    Unless the application requires all the modules in all the threads, that is bad advice. It ensures that every thread will get it own copy of everything in every module, whether it ever uses it or not. And if the application does require all of those in every thread, it is badly designed and probably won't work anyway.

    If the OP structures his application properly, and requires only those modules he needs in each thread, it will reduce memory consumption conspicuously.

    But you're right that the simplest route to reducing memory consumption is to avoid behemoth frameworks that force you to load everything before you can do the simplest thing, and a shit-load more every time you want to do something extra.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      You're right as long as the threads do different things.

      But he said, he'd need many instances, my suggestion was to make every "instance" one "thread".

        But he said, he'd need many instances,

        He actually said:

        I'll need to run several instances of this process

        And in general, it is a Bad Idea to use threads as (direct replacements for) processes. He would almost certainly save memory by using fork assuming that he's running on a platform with native fork and COW. The savings won't be as dramatic as many people expect, but depending upon the nature and detail of the application, they could be significant relative to running multiple instances of the application within threads.

        But the biggest saving would likely come from restructuring the application to make best use of threading. This generally means only threading those small parts of the application that really benefit from concurrency, rather than throwing huge lumps or whole applications into each thread.

        For example, people have a tendency to want to run objects across threads. That is, they create objects in their main application, and then when they need the thread to do something that will take a long time and they don't want to wait, their instinct is to pass the object to a thread to do it. Whilst this can be made to work, and can even be convenient for one-off's, it requires a lot of duplication (of memory) and tends to lead to the use of large numbers of short-lived threads, which is expensive.

        A better approach is to move the threading inside the class. So, instead of passing a single instance of a MyClass() object to a new thread every time it need to perform the longRunning() method, the class creates a single thread with input and output queues to perform the _longRunning() procedure, and the longRunning() method simple queues the relevant instance data to that thread and immediately returns a promise to the caller. When the _longRunning() procedure has finished processing the instance, it places the results on the output queue. And when the caller asks for the promise to be fulfilled, the result is read from the queue and returned. The caller can also poll the promise periodically to check if it is ready to be fulfilled.

        Though this strategy is more complex to program, it advantages are myriad.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re^2: memory usage and leakage
by szabgab (Priest) on Jun 23, 2011 at 11:10 UTC
    Thanks, I especially liked your sig :)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://911057]
help
Chatterbox?
[Discipulus]: found translate only good from or possibly to english.. in any other case i use double way translation

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (11)
As of 2018-06-19 07:38 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?



    Results (111 votes). Check out past polls.

    Notices?