Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Why it takes so much time here?

by PerlOnTheWay (Scribe)
on Dec 28, 2011 at 03:05 UTC ( #945271=perlquestion: print w/ replies, xml ) Need Help??
PerlOnTheWay has asked for the wisdom of the Perl Monks concerning the following question:

BEGIN { print time . "\n"; } use List::MoreUtils qw/any/; BEGIN { print time . "\n"; } print time . "\n"; my @a = any {$_== 92} 1..100000000; print "\n"; print time; print "\n"; print @a;
The output is:
1325041203 1325041203 1325041218 1325041218 1

What is taking the 15 seconds here?

I guessed it was caused by initializing the array 1..100000000 ,and I tested it in C ,it's really less than 1 second:

int *array = malloc(100000000 * sizeof(int)); int i; for(i = 0; i < 100000000; i++){ array[i] = i + 1; }
Please pay close attention to the output given above(or you can try it yourself), seems no one has noticed the output ...

I printed the time before and after the iteration, the result is both 1325041218:

print time . "\n"; my @a = any {$_== 92} 1..100000000; print "\n"; print time;

So the time is not taken by the iteration..

Comment on Why it takes so much time here?
Select or Download Code
Re: Why it takes so much time here?
by ikegami (Pope) on Dec 28, 2011 at 03:19 UTC

    You've run out of memory and resorted to using virtual memory.

    100,000,000 scalars x 16 bytes = 1.5 GB just in data. Then there's the overhead of 100,000,000 memory blocks, and there's possibly waste resulting from alignment issue.

      My system has 7GB memory,how can it run out of memory for 1.5GB?

      [anony@dev-test test]$ free -g total used free shared buffers cac +hed Mem: 7 1 6 0 0 + 0 -/+ buffers/cache: 1 6 Swap: 1 1 0

        Ok, what about pure CPU speed limitations?

        15/100,000,000 s per iteration
        = 0.00000015 s per iteration
        = 150 ns per iteration

        Is that unreasonable?

        My system has 7GB ... (6GB free)

        On my system, creating a list of 10_000_000 scalars on Perl's stack already takes roughly 700MB. So, as you have 10 times as many, 7GB is pretty close...

        Try this

        $ perl -MList::MoreUtils -e'any {<>} 1..10_000_000'

        and check its memory usage with top or ps (abort with ^C).

Re: Why it takes so much time here?
by davido (Archbishop) on Dec 28, 2011 at 03:44 UTC

    In C you're allocating about 381MB of memory assuming 32bit integers, and iterating over it. In Perl, you're allocating 100,000,000 scalars, and passing them on the call stack.

    Your C code and what Perl is doing are far from comparable. You could see at least part of what's going on by browsing the XS source file for List::MoreUtils, here: MoreUtils.xs.

    But that doesn't show how the list is being built by Perl, nor how it's being passed on "The Stack."

    Just keep in mind that you're dealing with one hundred million Perl scalars, not one hundred million ints.


    Dave

Re: Why it takes so much time here?
by BrowserUk (Pope) on Dec 28, 2011 at 03:56 UTC

    The time (assuming you are running a 64-bit Perl in your 7GB), is taken in two stages:

    1. Allocating 6.8GB to hold the 100 million scalars on the stack;
    2. Cleaning up 100 million scalars from the stack

    If you want your Perl code to perform the same thing but more quickly, try this:

    #! perl -slw use strict; use Time::HiRes qw[ time ];; use List::MoreUtils qw/any/; print time(); my @a; $_== 92 and push @a, $_ and last for 1..10000000; print time; print @a; __END__ C:\test>junk38 1325044379.166 1325044379.17017 92

    That took just 0.00417 seconds on my machine.

    Note that it is different from your C code in that like List::MoreUtils::any(), it stops looking as soon as it has seen a value that meets your test criteria.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

Re: Why it takes so much time here?
by BrowserUk (Pope) on Dec 28, 2011 at 04:06 UTC

    You can avoid half of the time with your original code by skipping the clean-up using POSIX_exit():

    #! perl -slw use strict; use time::HiRes qw[ time ]; use POSIX qw[ _exit ]; use List::MoreUtils qw/any/; print time; my @a = any {$_== 92} 1..100000000; print time; print @a; print time; _exit(0);

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

      Tried, the result is the same,please try the exact code I provided above...
        ,please try the exact code I provided above...

        No.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?

Re: Why it takes so much time here?
by flexvault (Parson) on Dec 29, 2011 at 15:29 UTC

    Hello PerlOnTheWay,

    I read your question yesterday, and my first response was "why would anyone do that?". Then I read the responses and skipped to another RAT.

    But after thinking about your question, I realized that your trying to do "C" like things in Perl. I did the same from 1995 to 2001 (approx). But sometime after that I started to "think in Perl". I only speak one language, but I have heard from others that if you want to understand a spoken language, you have to think in that language.

    Well, I believe that's true of programming languages as well. If you want to be a Perl programmer, you have to think of how to best solve the problem in Perl.

    In 2015, I'll be a programmer for 50years. Most of that time without Perl. My first program was written on a computer with 4,096 words and only 16 instructions and 2 registers. No multiply and no division and no branch(now 'goto'). But you would be amazed at the things we got done.

    Why I'm commenting, is that you're got both 'C' and 'Perl' and 7GB of memory. Perl's strengths are numerous. Did you look at 'vec'? Maybe your problem could have been solved differently?

    But what I'm trying to explain; Is don't compare Perl to the strengths of 'C'. Solve real problems with Perl and use the strengths of Perl for your benefit.

    Now off my soapbox, some comments about your code:

    • start using 'use Time::HiRes qw( gettimeofday usleep ... );'

    If your code looked like this (not tested):

    use Time::HiRes qw( gettimeofday ); my $stime = gettimeofday; . . . print "Time taken: ",gettimeofday - $stime," seconds\n";
    No one would have asked you "...how did you get that time?", it would have been obvious. Try having a skeleton program with key modules and commented examples. Each time you solve something unique, add anything new to your skeleton program. Years from now, it'll save hours of looking for something you know you did before.

    And the extra benefit, you'll look like a hero to future management when they comment on "...how quickly he gets the work done!".

    Good Luck and keep learning about Perl.

    "Well done is better than well said." - Benjamin Franklin

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://945271]
Approved by ikegami
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (9)
As of 2014-09-18 18:26 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (120 votes), past polls