Using a temporary directory for test files is fine, until your tests need to do things that your user can't do. For example, my File::Find::Rule::Permissions needs to behave differently for root and for ordinary users, and the tests also need to create directories and files owned by all kinds of different users and groups.
If my tests are running as an ordinary user, I simply can't create all the structures I need. And if running as root, they *shouldn't*. Dicking about with users and group membership just so my tests can run would be a Really Bad Idea. So I mock up a bunch of filesystem functions so I can test as an ordinary user. Then if my tests are running as root *and* there are appropriate user/group relationships already, I then also create a bunch of test files and run the tests for real.