Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Test your code under the same conditions as it will run in production

by hossman (Prior)
on Dec 23, 2004 at 07:52 UTC ( #417019=note: print w/replies, xml ) Need Help??

in reply to Greatest programming mistakes and what to learn from

My first big project at my first big job was an apache module that took in some search terms and some categories as input, opened some files on disk that cached some meta-data for the category, and then did a search on a search index using the user input and the meta-data.

The module worked fine, it ran fast, i had no problems in development or in qa, and we ever ran it through PMA to make sure you could respond quickly.

when we deployed it live, it worked fine for a little while and then the whole machine would seeze up and not do anything. It turns out i wasn't closing the meta-data files, And the apache "threads" (sub-processes) where long lived, and haning on to the open file handles untill they exited.

The problem didn't show up in QA because they didn't hit it with a lot of concurrent load. It didn't show up in dev (even though i specificly tried hammering it with lots of requests) because the dev server had a much shorter time to live for the thread/process pool. It didn't show up in PMA (even though we were hitting it hard, and it was configured just like production) because the client we were using to trigger load was testing lots of permutations of input was doing a lot of randomness on the search terms, but not the category ... so the processes were usually getting lucky and typically allready had the filehandle that they needed opened.

it actually turned out that we were occasionally hitting the problem in PMA, it was just blocking untill another process died and released some filehandles ... once i put in the appropriate "close" call, the average response time was cut in half because of the eliminated blocking.

  • Comment on Test your code under the same conditions as it will run in production

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://417019]
[hippo]: Thanks for that. I had rather supposed that Nvidia had moved on over the last 15 years but apparently not so much.

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (14)
As of 2017-05-26 14:17 GMT
Find Nodes?
    Voting Booth?