I usually use WWW::Mechanize for my scraping needs, but for my latest project, the memory usage is not acceptable. Yes, I do set stack_depth to 0. The problem is that Mech is storing the responses in memory, and then there's the much larger decoded_content if the response was compressed, and then the copy of the html data as it's passed to HTML::Form. Right now, my single process is at 200MB and I was planning on having multiple simultaneous processes.
I know I can save the response content to a file with :content_file, but then much of Mech's other functionality disappears, such as link and form parsing. Does a memory-efficient alternative exist that writes the content to files and incrementally parses the content?