in reply to Pipes vs. temporary files

This is somewhat off-the-cuff, but I would expect the pipe methodology to be faster in terms of wall clock time. Running the first script and redirecting to a temp file requires that process to run to completion, which I would imagine could take some time since you say it produces a large amount of data. Using a pipe allows you to process the data in your second script as it is produced by your first script.

Ivan Heffner
Sr. Software Engineer, Inc.