I like the approach of Athanasius, although I would throw in a couple of negative look-around assertions (as in the code of the OP) to eliminate a match against something like '9999-999-9999'.
$diff contains : -.
This statement in the OP is ambiguous and potentially a serious bug. If the assignment to $diff were indeed
my $diff = ' -.';
(one possible interpretation of the quoted statement), the '-' (hyphen) character would be treated as a range metacharacter in the [\d$diff] (or [\d -.] after interpolation) character class and the class would end up including all characters from ' ' (space, or \x20) through '.' (period) inclusive, i.e., [\d !"#$%&'()*+,-.] in the ultimate class, probably not what was intended.