|laziness, impatience, and hubris|
Up front. I've done very little CGI/webserver stuff at all.
That said, I think that you are breaking the cardinal rule of presentation layer work, by mixing your presentation layer code and your 'application logic' code together in your CGI code.
(IMO) your application should be split into two distinct parts:
By disconnecting the production of the page from the connections to the webserver, you ensure that no matter how thick and fast the connections arrive you can deal with them in a timely manner whilst imposing minimal load on your critical, public-facing front end, because they are in effect just responding with a static page. The latest updated version that is available. It also ensure that you don't hammer your DB requesting the same information over and over for every page request that arrives.
Indeed, you could do away with the cgi completely and serve the webpage directly from the file-system though I don't know how the webserver would handle it if the file was in the process of being written when a request came in?
And by disconnecting the production of the page from inbound requests, it allows you to choose the frequency with which you update that page. Whether that be on the basis of every few seconds, or when a significant event occurs or whatever makes sense to your application. And you only need to maintain one copy of your complex data structure in memory.
With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.