Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Faster regex to split a string into runs of similar characters?

by BrowserUk (Pope)
on Nov 21, 2016 at 09:59 UTC ( #1176234=perlquestion: print w/replies, xml ) Need Help??
BrowserUk has asked for the wisdom of the Perl Monks concerning the following question:

Given a string of bytes, I want to split the string into chunks of the same byte value, as efficiently as possible. Ie. Given 'aaabcccdeeefffggg', get 'aaa','b','ccc','d','eee','fff','ggg'. The string of bytes can contain *any* values 0 .. 255.

I thought this would be easy, but has actually proved to be quite hard. The fastest, and in fact only way I've found is shown below. Test b checks the cost of the /m modifier, but is just a placeholder for something better...

$s = join'', map{ chr( 65+rand(26) ) x rand( 100 ) } 1 .. 1000;; cmpthese -1,{ a=>q[ 1 while $s =~ m[((?=(.))\2+)]g; ], b=>q[ 1 while $s =~ m[((?=(.))\2+)]mg; ] };; Rate a b a 203/s -- -1% b 205/s 1% --

Anyone see a better, ie. more efficient way?


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re: Faster regex to split a string into runs of similar characters?
by dave_the_m (Prior) on Nov 21, 2016 at 12:46 UTC
    Looking for blocks of the same char speeds it up a bit:
    $s = join'', map{ chr( 65+rand(26) ) x rand( 100 ) } 1 .. 1000;; cmpthese( -1,{ a=>q[ 1 while $s =~ m[((?=(.))\2+)]g; ], b=>q[ 1 while $s =~ m[((?=(.))\2+)]sg; ] , c=>q[ 1 while $s =~ m/((.)\2*)/sg; ], d=>q[ 1 while $s =~ m{ ( (?=(.)) (?: (\2\2\2\2\2\2\2\2) \3* | (\2\2\2\2) \4* | (\2\2) \5* | \2+ )+ ) }sgx; ], }); Rate b a c d b 193/s -- -0% -3% -35% a 194/s 0% -- -2% -35% c 198/s 3% 2% -- -33% d 296/s 54% 53% 50% --

    Dave.

      Never would have thought of that! It yeilds another nice chunk of savings in the real application:

      C:\test>\perl22\bin\perl 1176081.pl -WIDTH=1000 -HEIGHT=1000 yr() took 2.551594 buk() took 1.068262 buk2() took 0.681494 buk3() took 0.167000 dave() took 0.127978

      Thanks.

      On a related note: Any ideas why this:

      while( substr( $$str, $y * $WIDTH, $WIDTH ) =~ m[((.)\2*)]mg ) { ...

      Works (runs to completion, produces the desired results) on 5.10.1, but silently loops forever in 5.22.0?

      This works in 5.22.0:

      my $ref = \substr( $$str, $y * $WIDTH, $WIDTH ); while( $$ref =~ m[((.)\2*)]mg ) {

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
      In the absence of evidence, opinion is indistinguishable from prejudice.
        On a related note: Any ideas why this:
        while( substr( $$str, $y * $WIDTH, $WIDTH ) =~ m[((.)\2*)]mg ) { ...
        Works (runs to completion, produces the desired results) on 5.10.1, but silently loops forever in 5.22.0?
        Looks like a bug fix. expr =~ /.../g attaches position magic to the LHS after each match, to record the current pos() for that expression. If expr is a var or similar this works well; if expr returns something new each time, the old pos magic gets lost.

        This similarly loops forever:

        sub f { "abc" } 1 while f() =~ /./g;

        Dave.

Re: Faster regex to split a string into runs of similar characters?
by Eily (Parson) on Nov 21, 2016 at 13:27 UTC

    Depending on how you want to use the information this may help:

    use Benchmark qw( cmpthese ); $s = join'', map{ chr( 65+rand(26) ) x rand( 100 ) } 1 .. 1000;; push @first, $1 while $s =~ /((.)\2*)/gs; $s2 = " $s" ^ $s; # XORing the string with a shifted copy of itself, s +o that you have a series of 0s for identical characters push @second, $1 while $s2 =~ /(.\o{0}*)/gs; $\ = $/ x 2; print pack "(A4)*", map length, @first; print pack "(A4)*", map length, @second; cmpthese -1,{ a=>q[ 1 while $s =~ m[((?=(.))\2+)]g; ], b=>q[ 1 while $s =~ m[((.)\2*)]sg; ] , c=>q[$s3 = " $s" ^ $s;; 1 while $s3 =~ /(.\o{0}*)/gs], };;
    This first prints the length of the strings found by the two methods (I have removed most of the lines, which don't add any more information):
    55 97 65 7 87 60 53 98 2 71 35 68 67 58 12 19 17 22 + 5 28 63 96 30 18 32 6 37 27 47 68 79 97 2 9 60 + 75 87 31 15 82 62 78 33 69 10 35 4 82 61 33 63 82 +96 68 140 88 59 67 87 78 98 14 3 6 52 59 74 86 79 49 44 28 76 + 25 83 99 66 42 67 73 3 46 55 97 65 7 87 60 53 98 2 71 35 68 67 58 12 19 17 22 + 5 28 63 96 30 18 32 6 37 27 47 68 79 97 2 9 60 + 75 87 31 15 82 62 78 33 69 10 35 4 82 61 33 63 82 +96 68 140 88 59 67 87 78 98 14 3 6 52 59 74 86 79 49 44 28 76 + 25 83 99 66 42 67 73 3 46 1
    So the second method does give the correct length (plus an extra character because of the shift).

    And the benchmark is much faster:

    Rate a b c a 445/s -- -4% -80% b 465/s 5% -- -79% c 2228/s 401% 379% --

    It does not provide all the information of other methods directly (you still have to get a character in the original string to know what a substring exactly is), but might be useful depending on what you actually need.

      That's a nice piece of lateral thinking. I'll have to see if I can find a way of using it without throwing away those significant gains getting the other required info back. Watch this space...


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
      In the absence of evidence, opinion is indistinguishable from prejudice.

        Well if instead of the length of the strings you save the pos, getting the information from the first string is straightforward (and O(1) if you only need one character). It removes the need to copy the substrings altogether, since you can access them directly in the input string. It does look like you still get a significant gain when copying all the substrings into the array:

        cmpthese -1,{ a_copy=>q[ @array = (); push @array, "$1" while $s =~ m[((.)\2*)]sg; ] , a_pos=>q[ @array = (); push @array, pos() while $s =~ m[(.)\1*]sg; ] , b_cow=>q[ # There might be a COW mechanism because of the call to +substr @array = (); $s3 = " $s" ^ $s;; push @array, substr($s, pos(), length + $1) while $s3 =~ /(.\o{0}*)/gs ], b_copy=>q[ # Force copy, to avoid delayed penalty of COW @array = (); $s3 = " $s" ^ $s;; push @array, "".substr($s, pos(), l +ength $1) while $s3 =~ /(.\o{0}*)/gs ], b_pos=>q[ @array = (); $s3 = " $s" ^ $s;; push @array, pos() while $s3 =~ /.\o{ +0}*/gs ], };; __DATA__ Rate a_copy a_pos b_copy b_cow b_pos a_copy 383/s -- -20% -49% -61% -80% a_pos 478/s 25% -- -36% -51% -75% b_copy 747/s 95% 56% -- -23% -60% b_cow 971/s 154% 103% 30% -- -49% b_pos 1888/s 393% 295% 153% 94% --

Re: Faster regex to split a string into runs of similar characters?
by Eily (Parson) on Nov 21, 2016 at 10:26 UTC

    If you want to match all 256 byte values, you'll need the /s modifier. I thought this would just be more correct, but it also happen to be slightly faster (4% on my computer), certainly because it's faster to match anything rather than check that the character is different from "\n".

    Beside that, why are you using look-ahead assertions? Isn't /((.)\2+)/g stricly identical to your regex? I don't see a performance difference with that though, so maybe perl optimizes away the look-ahead.

      If you want to match all 256 byte values, you'll need the /s modifier.

      Yes. /s not /m; I always mix those two up.

      why are you using look-ahead assertions? Isn't /((.)\2+)/g stricly identical to your regex?

      No. That won't match a single character 'run'.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
      In the absence of evidence, opinion is indistinguishable from prejudice.
        No. That won't match a single character 'run'.

        Right. /((.)\2*)/g does though, and this time it is faster (by 12% on my computer).

Re: Faster regex to split a string into runs of similar characters?
by Anonymous Monk on Nov 21, 2016 at 12:04 UTC
    So some kind of run-length encoding? Maybe use existing fast C code?
      So some kind of run-length encoding?

      No. It relates to this problem.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
      In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1176234]
Approved by Ratazong
Front-paged by Corion
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (7)
As of 2017-12-16 00:40 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What programming language do you hate the most?




















    Results (447 votes). Check out past polls.

    Notices?