Something is probably segfaulting within your module's C code (or you've found a bug in Perl/XS). If all tests are passing, I guess the segfault would be happening rather late in the execution--maybe even right at cleanup.
To debug, by far the most efficient method would be to run it through an interactive debugger (or your preferred method of debugging segfaults) directly on a FreeBSD system.
Since you maintain several XS modules, my advice would be to grab an appropriate ISO of FreeBSD from http://www.freebsd.org/where.html and run it in a virtual machine, such as VirtualBox. Assuming average skill with basic OS installs, you can be debugging your module with standard development tools in under an hour.
If that option seems like too much work, you could try a web search for hosted FreeBSD shell accounts with development environments. I prefer the VM route, so I don't know of any I'd recommend off-hand.