No, it is a serious bug in IO::Select. At the end of a perfect day, it doesn't matter that IO::Select silently does stupid stuff for a case that one shouldn't end up in when in a perfect world. In the real world, there are lots of ways to easily end up in that case and so it is quite simply unacceptable for IO::Select to skip trivial work that would avoid silently doing stupid stuff in that common case.
Well, I live in real world and in 10 years of network programming I haven't run up into situation where the behavior of IO::Select would be a problem for me. And so far neither you nor BrowserUK has provided practical example of such situation. Only theoretical.
Really, removing socket from IO::Select is the smallest thing that should be done before socket is closed. In practical life there's whole bunch of other stuff that should be done as well. For example: release data associated with the connection, notify the user that connection is closed (log it at least), possibly schedule establishment of new connection and so on. If you as developer are using some library that is not friendly enough to allow you to do all of this stuff, then you've made wrong choice for the library.
It is quite normal for 2 libraries to be incompatible, but it is strange and confusing when in this case one of the libraries is called "broken". Wouldn't "asking for new feature" be more appropriate here?