Listening on multiple ports using IO::Select

[JPaul]

Greetings monks;
I'm trying to get a non-forking perl TCP server (at its simplest) to listen to multiple local ports bound to the same address and am having a time making it work. What I'm doing appears logical... but just doesn't want to do its thing.
I use IO::Select to watch the first port, and figured it'd be as simple as add()ing another server handle to the Select handle's list... apparently not.
The below code produces a working server on port 5000, however when I connect to port 5050 the script terminates with a 'Broken pipe' error, closing the telnet connection rather impolitely.

my $server = IO::Socket::INET->new(LocalPort => 5000,
                                   Listen    => 10,  
                                   Reuse     => 1) 
        or die "Can't make server socket: $@\n";
my $serverB = IO::Socket::INET->new(LocalPort => 5050,
                                   Listen    => 10,   
                                   Reuse     => 1) 
        or die "Can't make server socketB: $@\n";    

my $select = IO::Select->new(($server, $serverB));

while (1) {
        foreach my $client ($select->can_read(1)) {
                if ($client == $server) {
                        $client = $server->accept();
                        $select->add($client);      
                        print $client "Howdy\n";
                } else {
                        print $client "G'bye\n";
                        $select->remove($client);     
                        close($client);          
                        }              
                }
        }

If I switch the Select->new statement, I get the same results, so I know its not in the way I add the handle to the Select.
ie:

my $select = IO::Select->new();
$select->add($server);  $select->add($serverB);

What fundamental fact am I missing here that prevents this from working like I want it to? (Sounds suspiciously like "Do what I want, not what I say")
Thanks all;

JP,
-- Alexander Widdlemouse undid his bellybutton and his bum dropped off --