As for adding the trace messages to threads.xs, not sure how to go about that. I searched for it in the perl installation folder for it but it was not there. Is it part of the threads.dll?

The easiest way would be to grab the latest version of threads from cpan, then build and install it manually. (Once you've checked you can still reproduce the error with the newly built version), you can then modify threads.xs, re-build and install.

Do you think any harm would be caused from doing something like my $val = eval{$join->join()}; as just a way to prevent the server from crashing.

If you are not doing anything with the return value from the join, it would probably be an okay temporary workaround. If you're only joining teh thread to make it go away, and it is going away of its own accord, in one sense, job done.

But in the long term, whatever the cause might just be a symptom of a deeper issue in your code; or in the threads module; or perl itself. As you have a recreatable scenario, it would be silly not to use it to help track down the underlying issue.