Apart from that, maybe my hint was too subtle, but I think your sample code behaves quite differently from the OP's code. Shouldn't you have set the auto_deref option to true for your ArrayRef and HashRef accessors?
Yes, you are correct, but even that would not work exactly the same. The OP's code (as well as your refactoring) would return an array count in scalar context (or that odd fraction looking string that you get when you do scalar %foo that tells you about the buckets and keys of the HV), where auto_deref in Moose would return an array reference or hash reference. Honestly, this to me is just another reason to use Moose since this behavior is sane, documented and tested, all a user would need to do would be to check those docs. Using the hand-rolled code, a simple accident of context (a common gotcha for Perl beginners) could lead to a very confusing debugging session and non-obvious perldoc hunt. I am sure even you would agree that seeing something like "HASH(0xDECAF)" is easier to figure out then "3/12".
One of the things about Moose that I suspect you are not seeing is that it brings consistency to Perl OO and that is a good thing. If 100 programmers write a Perl classes using Moose and use auto_deref, then any of those 100 programmers could look at any of those 100 classes and know exactly what auto_deref means and how it will behave. Additionally all these 100 classes would be (within reason) compatible with one another since they would all be using the same underlying object system making it easier for people to reuse them.
If just 10 programmers write a Perl class using their hand-rolled object system the odds that any of those 10 systems would be compatible with one another are not very good. The odds are very good that at least one of those hand-rolled object systems would have issues with inheritance and some of them will make mistakes like the OP and do something like regenerate accessors in the constructor. Writing an object system in Perl is easy and fun, but writing a robust and flexible object system in Perl is hard and is only fun if your a masochist.