You nearly always want to use the low precedence or instead of || for this kind of statments. In your case, with precedence being what it is, you actually have :
%config = ($conf->getall || die "message");. That probably doesn't look like what you meant.
With or %config = $conf->getall or die; means (%config = $conf->getall) or die; so getall is called in list context.
That would still mean that you would die on an empty config (%config = ()). And instead of using or you could just add parenthesis when needed, but I find it clearer when the low precedence operator are used for control flow, and high precedence ones are used when the return value is used.
NB: your problem is, with a few meaningless differences, one of the exemples given in Logical or and Exclusive Or to show the meaning of both operators.