I was thinking of just rsync'ing the following directories
Sounds like a reasonable plan (if the machines are binary-compatible). But note that if modules depend on 3rd-party shared libraries (installed in different locations), you'd have to distribute them too... For the latter case, some packaging/distribution tool might be easier to use.
In case you intend to do larger cpan installations, in particular of modules that might risk destabilizing any system tools that rely on the Perl installation as it ships, you might want to think about setting up a second separate Perl installation for your own use (e.g. in /usr/local/perl/...), into which you then install the CPAN modules (and rsync them...).