in reply to Faster searching
How about creating your index from the ftp server's log (assuming it has one)? Have a process that tail's the FTP server's log and updates the index whenever something is put. I did something similar with wuftpd a few years back and it worked briliantly. You could even skip this whole problem by generating the "log of changes" for the application guys from the FTP server's log.
In Section
Seekers of Perl Wisdom