Re^2: Reusing Compiled Perl or Opcode for Apache Server

[mlodato]

Thanks Rolf!

So we currently use mod_perl. We don't use fastcgi. I'm not exactly sure why but there's a comment mentioning "unconfirmed strange behavior". Basically, when we start up a server, it calls use on all the packages/modules so that they're in memory and available quickly when a response needs to be processed. I was wondering if there was a way to store the result of all these use calls in a way that would make it faster to load next time.

Does that make any more sense? Apologies if it doesn't...

[hippo]

I agree with RonW in that it is likely to be impossible FAPP. However, there are a number of alternative strategies which you could employ:

• Don't load any modules at server start. If server start-up time is more important to you than the response time of an initial request then just don't bother with all this preloading.
• Don't load all the modules, just the lighter and more frequently-used ones. This is the compromise option.
• Don't keep stopping and starting the server. Just let it run and use Apache2::Reload to pick up code changes.
• Live with it. How long does it take anyway? 3 seconds? 5? If more than that there may be other issues at play which you haven't disclosed. Apache (even with mod_perl2) isn't JBoss - it should only take a few seconds at most to start up.

[mlodato]

Thanks to you and RonW both! I did more digging and I agree that due to Perl's blurry compile-but-I'm-going-to-run-some-stuff-too phase, this just isn't feasible.

For most of us, response time of requests is more important than server start, but I just didn't know if this could be done in case a server died randomly or some other nonsense.

Good idea. I have been looking into how to only preload the modules needed for individual team workflows, so hopefully that will be fruitful. Just didn't know if there was an "easy" shortcut.

Definitely it would be best if servers never had to be restarted, but I guess I was trying to save people time in case servers died or they suddenly someone wanted to start up a server to demo or check something

It takes 15-20 minutes to start the servers. I'll try to do some profiling and see where the hot pockets are. I know that a non-preloaded server takes more like one minute and I know that static content and Perl modules are preloaded so I figured I'd ask you guys about the Perl half and try to do the front-end part myself because it's less abstract than whatever Perl does.

[hippo]

It takes 15-20 minutes to start the servers. I'll try to do some profiling and see where the hot pockets are. I know that a non-preloaded server takes more like one minute

This sounds like you have bigger problems. Neither of those delays (the 15 minute nor the 1 minute) is reasonable - as I said above they should be measurable in seconds on modern OS and hardware. If I were you I would increase the LogLevel in apache all the way up and see if that explains what the cause of the delay is. Otherwise, check the server for other problems: swapping could easily be one but also maybe bad disks or even a network misconfiguration as my anonymous brother suggested.

Good luck with your diagnosis.

[Anonymous Monk]

It takes 15-20 minutes to start the servers. I'll try to do some profiling and see where the hot pockets are. I know that a non-preloaded server takes more like one minute and I know that static content and Perl modules are preloaded so I figured I'd ask you guys about the Perl half and try to do the front-end part myself because it's less abstract than whatever Perl does.

Hi, that sounds excessive, like network lag resolving hostnames or some such, I'm real curious to see what you come up with

on my old laptop I can load 6086 modules in 86 seconds

[RonW]

I was wondering if there was a way to store the result of all these use calls in a way that would make it faster to load next time.

For a mod_perl application, I think it's impossible. Besides the many problems presented by Perl itself, mod_perl and Apache impose additional problems.

For a stand-alone Perl application, it's remotely possible, depending on what the modules loaded by use do and how. Also, how your Perl application uses those modules, what your application is doing and how it does it.

If any of those modules require shared libraries (also known as DLLs), probably impossible. There are also other things that use-ing a module can do that make loading a saved cache impossible.

The problems arise because in Perl use doesn't simply load a module. Modules can and often do have initialization code that has effects that can't be saved and reloaded. They have to be done every time the module is loaded.

Ideally, modules should be written to separate initialization from loading, but (1) initialization is often a big chuck of the "loading" time, (2) most people want the convenience of the automatic initialization and (3) too few people have a need to defer initialization that the extra logic to support a "load only" option in modules isn't worth the effort to maintain.

It's very likely to be impossible. I can understand that your employer may want you to try. You could look at https://metacpan.org/pod/distribution/B-C/script/perlcc.PL, but very doubtful anyone will be able to help you. Even if you were to write a new application completely on your own, it would still be very difficult to make the end result work correctly.

[LanX]

> Does that make any more sense?

Not really. Mod_Perl is already caching at startup and is fast for every http request. (If not then you are doing it wrong)

Frequently restarting Apache doesn't make sense, sorry.

This rather sounds like XY problem.