I couldn't find any combination of DESTROY and eval in Log::Dispatch but there must be one somewhere as without Log::Dispatch everything was working fine...
For some reason the cause of the problem was not in Log:.Dispatch, but in one of my classes and their destructors. I didn't use eval directly, but indirectly it was used somewhere within Log4perl, which was used to log some debug statements in the destructor. After I added local $@; before the logging statements error trapping worked again. Don't know why the problem was only triggered using Log::Dispatch, though.
Looks like I have to really switch to Devel::EvalError in the future, problems like this are very hard to find.