glob quietly fails

[QM]

I've been tinkering with combinations of strings from different arrays, and discovered that the ubiquitous glob solution breaks for long input strings.

Update: The below example is for Strawberry Perl 5.16.1 on Windows. On Ubuntu 10.4, Perl 5.10.1 (i486-linux-gnu-thread-multi), it first breaks at 4027.

Here's a trivial example to show the effect:

#!/usr/env/perl

use strict;
use warnings;

my $q = (shift or 129); 
my @x = ( 'A' x $q, 'B' x $q );

my $glob = '{' . join(',', @x) . '}';

while (my $g = glob($glob)) {
    print "$g\n"; # no output if $q > 129
}

##### returns nothing with 129 ####

What I really want to do is more like this, but glob also fails when longer strings are used:

my @x = (1..10);
my @y = qw(one two three four five six seven eight nine ten);
my @z = qw(Amy Bob Cal Deb Eck Fay Gin Hob Ian Jan);

my @xyz = (\@x, \@y, \@z);
my $glob = join( ',', map { '{' . join( ',', @$_ ) . '}' } @xyz );

Q1: Why does glob quietly fail?

Q2: Is there a built-in, core, or module that is not so limited?

Instead of that, I suspect I'll have to do something like the following. This is very simplistic, but does the job. (I'm sure this has been done before, but I haven't spent enough time searching.)

Q3: Any comments here?

sub make_combo_iter
{
    my $array_ref = shift;
    
    # last index of top level
    my $size = scalar( @{$array_ref} );
    
    # create array of last indices
    my @array_limits;
    for my $i (0..$size-1)
    {
        $array_limits[$i] = scalar( @{$array_ref->[$i]} ) - 1;
    }
    
    # one counter for each sub-array
    my @array_counter = (0) x $size;
    my $done = 0;
    my $iter = sub { 
        if ($done)
        {
            $done = 0;
            return undef;
        }
         
        my @next_combo;
        # compute @next_combo
        for my $i (0..$size-1)
        {
            push @next_combo, $array_ref->[$i][$array_counter[$i]];
        }
         
        # increment counters, set $done if the last one rolls over
        my $rollover = 0;
        for my $c (0..$size-1)
        {
            if (++$array_counter[$c] > $array_limits[$c])
            {
                $array_counter[$c] = 0; # reset
                $rollover = 1;          # carry
            }
            else
            {
                $rollover = 0;  # no carry
                last;           # no more increments
            }
        }
        
        # if the last array rolled over, set the done flag
        if ( $rollover )
        {
            $done = 1;
        }
         
        return \@next_combo;
    }; # anonymous iterator sub
    return $iter;
} # make_combo_iter

[daxim]

Q2: see Re: Hexamer combination from 3mers. about Algorithm::Combinatorics. Abusing glob for its permutation/variation/combination abilities is a terrible hack that needs to be extinguished.

[QM]

see Re: Hexamer combination from 3mers. about Algorithm::Combinatorics.

Understood. But that doesn't solve the "array of arrays, 1 from each" problem. A shorter version of the OP glob spec would be

@x = (1..2);
@y = qw(one two);
@z = qw(Amy Bob);

@xyz = (\@x, \@y, \@z);
$glob = join( ',', map { '{' . join( ',', @$_ ) . '}' } @xyz );
print "$glob\n";

# prints '{1,2},{one,two},{Amy,Bob}'

I don't know of a module that can emulate glob on that.

[martin]

Actually I think your iterator solution is not half bad. It could work very well where nested foreach loops are not an option.

I'd drop the redundant array_limits array, however.

[Anonymous Monk]

Q1: Why does glob quietly fail?

Recompile http://perl5.git.perl.org/perl.git/tree/HEAD:/ext/File-Glob with  DEFINE           => '-DGLOB_DEBUG', you might get a hint :)

[RichardK]

What are you expecting the first code example to do?

As it is, it creates an array of 2 strings each 129 characters long and then glob returns them. Which is working just fine for me ;)

UPDATE : Ah! your calling glob in scalar context! try a list context instead.

for my $g (glob(...) ) {
}

[remiah]

Hello RichardK.

glob fails silently as QM saids with my perl. How does this code say with your platform?

for (my $q =1; $q < 10000; $q++){
    my @x = ( 'A' x $q, 'B' x $q );
    my $glob = '{' . join(',', @x) . '}';

    my $g = join(',', glob($glob));
    if (length $g == 0){
        print "fail silently, q=$q\n";
        last;
    }
}

Mine stops saying

fail silently, q=511

perl -v

This is perl 5, version 12, subversion 2 (v5.12.2) built for i386-freebsd-thread-multi-64int

regards.

[QM]

As remiah and Anonymous Monk have stated, I expected while and for to behave the same way, and they do for me, on Strawberry Perl v5.16.1:

#!/usr/env/perl

use strict;
use warnings;

my $q = (shift or 129); # 128 is the largest working value
my @x = ( 'A' x $q, 'B' x $q );

my $glob = '{' . join(',', @x) . '}';

while (my $g = glob($glob)) {
    print "while: $g\n"; 
}

for my $g ( glob( $glob ) ) {
    print "for: $g\n"; 
}

__EOF__
#################
# result:
#
while:
for:

Running this with a command line argument of 128 gives the expected output of 2 very long strings for while and 2 for for.

[Anonymous Monk]

UPDATE : Ah! your calling glob in scalar context! try a list context instead.

Both are supposed to work