I think you misunderstand what mod_perl
is about, or have not spent a lot of time configuring it properly. The parts of Apache that lead to mod_perl being activated are small and efficient, so "getting rid of Apache" seems an odd goal.
Please make it clearer about why you've ruled out mod_perl. "forking" under mod_perl is no more expensive than forking under your proposed daemon. And you don't "load 3-4mb (sic) of modules every time" you fork under mod_perl. If you preload in the Apache parent process, nothing needs to be loaded per fork.
P.S. 3-4mb is 3 to 4 millibits. That's pretty small. Get 250 to 333 of those together, and you have an entire bit!