You have:
if ( ! exists $our_cache_imap_object ) {
there. exists checks if a hask or array element does exist. You should rather check whether the object is defined, e.g.:
if ( ! defined $our_cache_imap_object ) {
Mail::IMAPTalk's constructor supports being passed a reference to a Socket. You can do some magic with that. You might want to take a look at Socket::PassAccessRights, which you could use to pass sockets to an external daemon, which would ensure that the connection is not lost and get the sockets back agian.