Side note. Instead of your sort call, consider using Sort::Key's nkeysort_inplace:
use Sort::Key qw(nkeysort_inplace);
nkeysort_inplace { $_->[3] } @$points;
Read
Sort::Key's docs to find out how to sort on multiple keys if you want to secondarily sort on other values of your points.
The uniq function doesn't work for you because it treats everything as a string. Since you actually have a list of references (to arrays), not a list of simple scalars, this doesn't quite work. You will pretty much have to roll your own here. I'm sure there are ways to cheat, there always are, but it's probably more work than warranted. Something like setting up your points as objects that overload the q[""] operator to return whatever you want to determine uniqueness on - that might work. But I'm not sure about that. :-)
