If you wrote the module yourself, you are the one responsible for providing clean uninstall method for it.
If you used MakeMaker for creating a Makefile (I'm assuming that is what you did), it should create an uninstall target for you, so, if you installed your module with make install, you should be able to uninstall it with make uninstall from module's directory.
Anyway, you should provide more details (how did you install this test module? windoze ppm? perl -MCPAN -e "install Test::Module"? make install?