apache/perl caching problem

[ksublondie]

Monks,

I have a perl site on an apache2 webserver. While writing/editing my code, of course I periodically test it in my browser. However, I don't know what version of my perl code will show up. Usually, my html/javascript templates are the current version, but I could receive any version of my perl at any time.

I regularly reload apache and have tried restarting apache, but I still get the problem. I've even tried clearing my cookies & temp internet files multiple times (even though this is a server problem not a client problem, right?).

I've always had this problem from time to time, but it's now wearing on my last nerve. Does anyone have any answers for me?

[desemondo]

If you aren't specifying that your perl generated webpage should expire immeadiately, your browser may try to cache it. I don't have the code handy at the moment, but I'm pretty sure its a just a simple one liner you need to add to your code.

*Eg.

use CGI;
 print header(-type=>'image/gif',-expires=>'now',);


(Alternatively, If using IE, you can disable browser caching by Tools>Internet Options> General Tab > Browsing History Settings button > Select "Never" radio button.)

Pressing Ctrl+F5 should also force your browser to pull a new copy of the page from the server (pressing F5 doesn't necessarily refresh everything, it cheats a bit...)

*Updated: Added code snippit

[ig]

but I could receive any version of my perl at any time.

I guess you mean that at your browser you receive the output produced when your server runs your Perl scripts. If I have guessed wrong then what follows probably isn't relevant.

If you sometimes receive old content after having received new content then it isn't a simple caching problem.

Do you have a pool of caching proxy servers between your browser and server? If so, each proxy server might have cached a different version of response and, depending on which server the load balancer sends your latest request to, you might get various ages of responses somewhat randomly. Controlling expiry will solve this problem if the proxy servers are well behaved. In development I would bypass all proxy servers and query the source server directly, if at all possible, until I was sure the server was stable and correct, then test through the proxy servers to ensure that caching wasn't causing a problem.

If there are proxy servers between browser and origin server, can you inspect their logs to find out what requests they receive and what they do to respond to them?

this is a server problem not a client problem, right?

When you receive old content at your browser, does your server access log show that the request made it all the way to the server? Your content may be delivered in parts through several requests: a main document and many linked documents (images, css, javascript, etc.). If content is not being served from a cache then you should see a record of each part being accessed in your server's access log.

Setting the cache expiry on the document produced by your Perl script will not affect caching of other linked documents.

If you are going to control caching, it would be better to put the controls in the HTTP headers rather than in the HTML. You can't add meta tags to images, as far as I know, and I suspect not in your javascript either (though I could be wrong about the latter). There is a nice tutorial at http://www.mnot.net/cache_docs/ that explains some of the options and issues.

You may find the apache mod_expires module helpful for adding HTTP headers related to caching: http://httpd.apache.org/docs/2.0/mod/mod_expires.html

[ksublondie]

"...you sometimes receive old content after having received new content..."

That is EXACTLY what I'm getting...and it could be any version in between! I'm not sure on the proxy servers, but I doubt it. As far as I know this is a basic intranet server. I need to hear back from the guy that does the network architecture to know for sure...

I guess I should have been clearer on what I am getting. When I modify my HTML templates, the output is immediately correct. When I modify my perl script, I have to refresh several times to maybe see the change. ...and once I do see the change, I may get the older version at any time. The portion that is generated with the perl is seemingly random. Some crazy examples:

• Now-fixed errors getting generated & triggering a separate error page (in the perl) to display instead of the correct output page
• Email generation: either triggered by the example above and/or containing different content than the current version
• SQL queries & output may be different -- including now-fixed SQL errors still being generated, ex. incorrect server connection variables, query typos, etc
• Different variable values presented to the templates
I suppose the last 2 examples could be a previously cached local version, but to me the email screams "server problem". Frustrating, to say the least.

I haven't checked the access logs, but when I receive an error, it's in the error logs and of course the email error is generated. I can check the access log, of course it may take me a while to PURPOSELY recreate the problem!

...Yes, I've tried the ctrl+f5, clearing my cache multiple times, changing browser settings, blah blah blah, and anything else that would point to a local caching problem... I can also recall having end users test it for me that have NEVER been to the page, however they may get a previous version.

I just scanned those links you referred and haven't read them in depth yet, but wouldn't controlling the caching in the HTTP headers only effect the HTML and not the perl behind the scenes?

[ig]

Caching control in the HTTP headers controls what the clients (including intermediate proxy servers) do with the response and subsequent queries for the same URL. It doesn't matter whether the response came from a static HTML file or was produced by a script.

To be certain the responses are coming from your server, I would run a network sniffer on the server (e.g. wireshark) and observe the query and response.

After confirming the bad responses are coming from the server, I would investigate the server configuration. Given the unusual behavior, I would make no assumptions, so I would begin by determining what process is listening on the port that accepted the connection over which the request and bad response were exchanged. If this is an apache server process then the scope is narrowed, but perhaps there is some intervening software.

Is your script a standard CGI script? If it is, it will be read from disk and executed once for every request handled by the server. You might verify this by having your script log its start time, process ID, version and the full path and modification time of the file loaded. Log this every time your script handles a request. You can then correlate these logs with the requests to confirm that your script is running and producing the results you are seeing, and see exactly what is running each time. If you have plain CGI, you should see a different process ID each time and the version and modification times of your script should always be the latest. Your evidence suggests you will see something else. If you see the same process ID for several requests, then you should investigate what that process is, how it comes to be handling multiple CGI requests and how it is handling your script.

The mod_perl module was mentioned in a previous post. This and others can cause your script to be loaded and kept in memory, effectively becoming a subroutine that is executed over and over for each request, rather than running your script from disk for each request. This is caching of a different sort and seems a likely explanation for the behavior you have described. HTTP headers and meta tags in the produced HTML will not affect this sort of caching.

[Anonymous Monk]

mod_perl?

[jethro]

What about your browser cache? If your server delivers wrong expiry information for your pages, your browser might keep old versions of your webpages in the cache and you get the impression you see previous scripts at work.

[MidLifeXis]

• Different shbang lines
• Multi-node hosting service with different versions of perl on each
• ... profit???

[technojosh]

clear your cache. I think safari's developer tools allow you to explicitly "not cache" things to avoid this. other browsers may have similar switches...