Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

Re^3: Perl is dying

by Jenda (Abbot)
on Jul 16, 2006 at 08:15 UTC ( #561528=note: print w/replies, xml ) Need Help??

in reply to Re^2: Perl is dying (hyperbole)
in thread Perl is dying

Any examples?

Replies are listed 'Best First'.
Better sorting than the ST (was: Perl is dying)
by Aristotle (Chancellor) on Jul 16, 2006 at 08:42 UTC

    Generally I use an extra array for extracted keys and then use it to sort indices.

    my @sorted = do { my @key = map { lc substr $_->[ 4 ], 0, 3 } @a_of_a; my @idx = sort { $key[ $a ] cmp $key[ $b ] } 0 .. $#key; @a_of_a[ @idx ]; };

    Thatís much easier to follow than the ST. Additionally, it uses orders of magnitude less memory for large datasets because it doesnít create zillions of tiny anonymous arrays. (I love Perlís effortless data structures and what they enable, I really do, but roughly 100 bytes of overhead for an empty arrayÖ talk about wasteful.) Itís also faster for the same reason. In fact, itís a speed demon Ė as fast as it gets without a GRT, or possibly faster if the transform you need for your GRT is costly.

    Whatís not to like? I see very little reason to use a classic ST.

    Makeshifts last the longest.

        Yeah, those would be attempts to formalise the GRT. Thatís worthwhile to attempt to put into a module Ė GRT transforms can be extremely cryptic. A hard to learn module is still a better alternative to that, trust me. :)

        Makeshifts last the longest.

      I'm wondering if being concerned about the 100 byte overhead is a form of premature optimisation. Hypothetically perl should be able to determine if the arrays are going to have their size changed, or the arrays will be preserved or whatever, and if possible use a something more efficient.

      Actually here is where Perl5 could steal a trick from Python and add a Tuple data type, that is an array like data type that is of constant size and contains constant values. Having such a data type would make it possible to write ST's and be MUCH more efficient, since the constantness of the structure would allow the structure to be represented in a much more efficient way. Much of the size issues of perls data structures are due to its dynamic nature, and the fact that we optimise for speed and not space.


        How can perl know the extent of of the arrays? I can see from looking at a sample optree that the size of the input array can be known at compile time. I can also see sometimes sort's function can be known to be "simple" and won't increase the extent of the array. Lastly, I can also see that the expression used by the outer, unpacking map is simple. If I thought this were an interesting thing to chase down I'd probably go write some prolog to match stuff in our optree and if we were lisp instead of perl I'd write a macro to automatically optimize the code. We're not and we won't get that goodness til Perl 6 so it'd be a waste to do this to Perl 5.

        perl -MO=Concise -e 'print map $_->[1],sort{$a->[3]<=>$b->[3]}map[lc,$ +_],1,2,3' -e syntax OK r <@> leave[1 ref] vKP/REFC ->(end) 1 <0> enter ->2 2 <;> nextstate(main 2 -e:1) v ->3 q <@> print vK ->r 3 <0> pushmark s ->4 k <|> mapwhile(other->l)[t8] lK/1 ->q j <@> mapstart lK/2 ->k 4 <0> pushmark s ->5 - <1> null lK/1 ->5 ;; You can see the ->[1] access and you already know there's only two +elements. p <2> aelem sK/2 ->k n <1> rv2av[t1] sKR/1 ->o m <1> rv2sv sKM/DREFAV,1 ->n l <$> gv(*_) s ->m o <$> const(IV 1) s ->p ;; You can see the entire sort block got optimized into something nati +ve (I guess. It's certainly not in perl anymore) i <@> sort lKMS* ->j 5 <0> pushmark s ->6 - <1> null sK/1 ->6 - <@> scope sK ->(end) - <0> ex-nextstate v ->- - <2> ncmp[t4] sK/2 ->- - <2> aelem sK/2 ->- - <1> rv2av[t2] sKR/1 ->- - <1> rv2sv sKM/DREFAV,1 ->- - <$> gv(*a) s ->- - <$> const(IV 3) s ->- - <2> aelem sK/2 ->- - <1> rv2av[t3] sKR/1 ->- - <1> rv2sv sKM/DREFAV,1 ->- - <$> gv(*b) s ->- - <$> const(IV 3) s ->- b <|> mapwhile(other->c)[t7] lK/1 ->i a <@> mapstart lK/2 ->b 6 <0> pushmark s ->7 - <1> null lK/1 ->7 ;; You can see the array get allocated and can see it has only two ele +ments. h <1> srefgen sK/1 ->b - <1> ex-list lKRM ->h g <@> anonlist sKRM/1 ->h c <0> pushmark s ->d e <1> lc[t6] sK/1 ->f - <1> ex-rv2sv sK/1 ->e d <$> gvsv(*_) s ->e - <1> ex-rv2sv sK/1 ->g - <1> ex-rv2sv sK/1 ->g f <$> gvsv(*_) s ->g 7 <$> const(IV 1) sM ->8 8 <$> const(IV 2) sM ->9 9 <$> const(IV 3) sM ->a

        ⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊

        Optimisation, yes; premature, how? Please show me how maintenance or verifiability suffers in any way from picking this algorithm over the ST. Tuples would help the ST, but even if its key container overhead were much smaller, itíd still be O(n), whereas itís O(1) for this algorithm.

        Makeshifts last the longest.

      In another node, you mention this came from APL. I comment here since this is the prettier version than in Re: An APL trick for the Schwartzian Transform.

      Googling this has become Hard (not NP hard). Know of any off hand references?

        I have no idea whether this comes from APL, Iím afraid. I originally figured out this approach on my own and only afterwards learned that itís been known in the Perl community for at least as long as the ST. (Oddly, it does not seem to get promoted much, despite its considerable advantages in the general case.) I have no idea about its history beyond that.

        Makeshifts last the longest.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://561528]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (13)
As of 2017-03-24 14:12 GMT
Find Nodes?
    Voting Booth?
    Should Pluto Get Its Planethood Back?

    Results (302 votes). Check out past polls.