Those are good points in regards to only changing the font size. See my other post for some more thoughts.
As for the threshold, since such a change would affect all older content as well, and AFAIK the vote counts were higher in the past (e.g. Selected Best Nodes), I suspect the threshold would have to be dynamic based on the number of votes cast. I don't think it's necessary to start at something like -3 or -4, since there are always some threads with simple disagreements in which hiding content might not be warranted. But then there are certainly nodes where the downvotes are overwhelming.
I'm just putting my two cents out there, I don't yet have a clear idea on what the best way to go about something like this might be.