I have a program that is giving me headaches. It stops in an obscure way and I haven't managed to figure out a way to identify the fault. I'd welcome any thoughts on how to narrow down the cause.
I have a perl program that runs for a couple of days and then crashes. It scans a database and creates various files for each row in some tables. One of the files is a graph, created using the Graphviz2 module. Graphviz2 runs a binary via IPC::Run to do the heavy lifting. The binary is /usr/bin/dot from the graphviz release, and IPC::Run reports that it failed with the message "Argument list too long". I don't understand where that error message comes from, because I didn't think IPC::Run invoked a shell.
But if I restart the program from that point in the databse, it creates the graph just fine, runs on for another couple of days or so and then crashes with the same error message on another seemingly random record.
I can't find any evidence of running out of memory or similar problems.
I'm really not sure where I should be looking to localise the problem, and would welcome any suggestions.