I agree that it is a bit unexpected and unfortunate that there are libraries of some kind that cause a socket to lose its file descriptor number.
I do understand, that there might be buggy libraries out there that close sockets, when they shouldn't. I also understand, that it is trivial matter to change behavior of IO::Select so, that it does not depend on presence of file descriptor. But at the end of the day, it is bug of those libraries, not bug of IO::Select. So it would be more appropriate to fix those libraries, not IO::Select.
What is good about fixing those things that are not really broken, and letting really broken stuff to function further? Personally I avoid using libraries that do unexpected stuff. They must die :)