I also faced similar issues in Windows perl distribution. I have solved by using
1. Install & use ActiveState Perl - Windows x86 version - All the modules builds properly & Work properly in x86 (including Win32::Serialport) in my laptop (Which is Windows 7 , X64 but I have x86 version of gcc to compile CPAN modules which are not available in PPM)
2. Purchased ActiveState PDK - There is utility called perlApp, which will convert your .pl to .exe (and includes all dependencies like .ico, .pm , .dll etc etc..) , Which works perfectly with Active perl.
3. Create x86 .exe version of tools and distribute. (It will work on x86 & x64)
4. Note: Yes, I know, creating .exe will increase size. But there is a option to create a dependent exe which uses installed perl without bundling perl in .exe. But I am fine with the size (usually ~3 MB).