Thanks for shedding some light on semaphore case.
I'd really like to create a test case but there's a few obstacles that i'd have to overcome in order to do this. First of all the app is pretty large. In minimal version there are at least 4 threads involved:
- thread hosting rpc server
- thread preparing load balancing for sql data related to rpc request
- thread that sends request with data definition to remote host
- thread keeping track of available hosts
I can't pinpoint which thread is to blame (i can't guess it neither from log nor from the message itself).
Second of all, the app is tied to local sql database and i can't see any sane way to simulate that in a script.
I know about the threads::shared bug that was fixed (i reported it to J.D. Hedden ;) but downgrading seems to me like "out of the frying pan, into the fire" - type situation :) But i guess You are right - maybe i could spot this problem more easily with this.
Although i'm still in a deep dark wood some light can be seen ;) Thanks to You i know it's something related to locks. I'll try to mess around with the code which uses lock the most and test, test, test.... Maybe i create a bug report after all.