I agree 100% with this assessment: you have a nasty logic-error in your code.
In addition to "Thread::Queue," you should also review what is contained in the Perldoc entry for threads::shared ... particularly the BUGS AND LIMITATIONS and the SEE ALSO sections.
Sharing of information among multiple threads does not work reliably, in the manner that you are now attempting to do so, even if it seems to do so in testing. (Hate to be the bearer of bad news, but then again, thread-related issues are often like that.) Your notion of “delaying garbage collection” will lead to even more problems; it is a blind alley.