You should probably check out SoX -- it's a command-line tool that is easy to install on osx (except that I think you may need to rename a file in the distro, from "INSTALL" to something different, e.g. "INSTALL.txt", so that "make install" will do the right thing). I haven't used it on mp3 data, but I think the tool supports that... If not, just do everything with wav or raw files, then convert to mp3 afterwards.
The command-line syntax for sox is pretty strange, so making up a perl wrapper script to do what you want on a list of files will be very handy (just trying to keep the thread "on topic" as a perl question...)
UPDATE: In case you seriously want to pursue a "pure perl" solution, my first inclination would be to convert your mp3 files of "challenge" and "answer" utterances into uncompressed raw pcm streams, join them together with a suitable number of 16-bit zeros to make up the appropriate length of silence for the given sample rate and channel count, then convert back to mp3. (If the files are stereo and 22.05 KHz, you need 4410 16-bit zeros to create 0.1 seconds of silence.) Have fun with that.