The best way to store your data really depends on what you're doing with it afterwards. Also a top level hash might be better if your data points are sparse.
Assuming that you'll never use the horizontal coordinate (abscissa) without the vertical coordinate (ordinate), you might even store them as a concatenated value in a hash, thereby simplifying your data structure by removing one level of nested-ness:
my %notes; # notes is now a hash
#...
$notes{"$x;$y"}[0] = ...
or:
$notes{"$x;$y"}{...} = ...