in reply to array overhead
This may sound facile, but have you considered installing more memory?
Another alternative is to do a little research into how long your script runs, and at what point it ".. starts to consume huge amounts of memory." Is this after 20 minutes? Five hours? Three days? Two weeks? If your script looks at the most recent four hours, and starts to bog down after three days, then presumably you could restart it every two days as a temporary solution.
Alex / talexb / Toronto
"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds