What you want to do is change your visitor attribute to be something like:
has 'visitors' => (
is => 'ro',
writer => '_set_visitors',
init_arg => undef,
Then inside your BUILD method, you will want to do:
$self->_set_visitors( ... );
The init_arg being set to "undef" will ensure that no one can do:
MyClass->new( visitors => ... )
But all this said, you should really spend some time reading the Moose::Manual. Several of the mistakes you have made seem to be caused by the fact you are guessing and/or making assumptions of your own about how Moose works, rather then actually taking the time to learn how it actually works. You also might find some of the presentations on http://moose.perl.org
helpful as well, there are several excellent introductory ones from recent conferences.
You really too should give a look to MooseX::SimpleConfig as it solves this problem for you and the code has been battle tested already.