I have previously run into the same memory leak issue issue with Chart::Clicker. As you noted, it has a fairly heavy dependency chain. I didn't get very far trying to follow it. This was for a project at $WORK and I need to get things done.
My work around was exactly as you said: fork off children to do the work. This was just a one off for exploration, so I didn't spend much time trying to optimize it. I forked a child for each chart, and kept the total child count under $n for some appropriate value of $n.