That works - thanks a bunch. For anyone else reading, it's easy to pass this through 'o conf':
o conf makepl_arg 'LIBS="-lz -L/usr/sfw/lib/64 -lssl -lcrypto"'
Long story: turns out that -lssl -lcrypto are being left out of EXTRALIBS and LDLOADLIBS in the Makefile (no idea why - Makefile.PL comes with LIBS => [q{-lz -lssl -lcrypto -lssl32 -lssleay32 -leay32}]). So putting them in the Makefile and building again from the shell works too:
EXTRALIBS = -lz -lssl -lcrypto
LDLOADLIBS = -lz -lssl -lcrypto