Did you know Math::GMP is already in Ubuntu as the package libmath-gmp-perl? For that matter Net::SSH is libnet-ssh-perl.
No, I did not. However, while I've tried to consistently use ubuntu and debian packages for most additions, Perl I've installed from source (in order to get the most recent version, which is never available in Debian stable) and CPAN modules I always first try to install with the cpan shell.
I think your Darwin travails are due to an older version of gcc miscompiling libgmp3. Can you try with 3.4 or higher?