I think that when you install a module, it stores the list of files it installs, somewhere in a text file. I forgot what it's called and I'm presently not at a computer where I can check. Sorry.
in reply to Re^2: help managing modules using CPAN.pm
in thread help managing modules using CPAN.pm
But to uninstall, just delete each file listed in the file. Except... if the installation replaced an older version of the same module, and I mean physically replace: placing the files on top of the old ones, well this will obviously not restore the old files.
I think you're then better off installing the old module over the new one, or at least have the archive ready in case it's an essential module to even be able to run CPAN.pm, so you can install them just using make and perl. Or install at a new location, delete the old files, and move the new files in place. Something like that.
Yes a process like this should be automated, IMO.
update I looked: the text file with the file listing is called ".packlist", and it resides somewhere under the "auto" root, in the module root where it's installed. For example, if the module is "Test::Pod" which is installed in "site/lib", then this file ought to be at "site/lib/auto/Test/Pod/.packlist".