Re^2: Scalable application with perl

[Your Mother]

Sounds like a fairly simple problem after all maybe. 12,000 users, but what are they doing during that 90 minutes? One GET and one POST each? Dozens? A hundred? Can users undo answers or backpage? Issues like this can change a simple problem into a terribly complicated one quickly.

Vanilla nginx can serve pre-built or cached GETs of form pages to the tune of 12K per second just fine if you have the bandwidth. The incoming POSTs have to go through the application though and you won't know how fast that is until you build or at least prototype it. uwsgi has deep and powerful controls for how many worker processes to run and can even make changes dynamically. nginx has an experimental perl embed module now too which could be amazing (I haven't tried it yet). Anyway if you have the RAM and a halfway decent CPU I imagine you'll be able to handle the use you're talking about unless a lot of the users are banging on submit the whole time.

Tips–

• Prebuild and cache (probably just as files) all forms. Serve them statically or from webserver memory; both are terribly fast in nginx. There are many Perl templating engines to accomplish this. Text::Xslate is a very fast one (the right choice for "live" service) and Template::Toolkit is a very expressive and well known, documented, and extended one.
• If the UI is complicated (users can backup and change answers, for example) save state in the browser with JS localStorage. It's a mild hassle to duplicate the form validation in the Perl and the JS but it will keep the webserverapp load zero till submit. If you split an app like this, good testing becomes crucial to avoiding nasty surprises.
• Investigate embedding the code in nginx but uwsgi probably has the dynamic load options you need.
• Extra clock savings: don't touch the DB at all during the exams. Dump the validated submissions as single field delimited strings to a flat file or NoSQL or similar. Whatever is fastest and puts lightest load on the machine. This is not my forté, I have no direct recommendation today. When the exam is over, transform and load the flat data to the DB. Then you can grade and run reports from there.

[tangent]

Extra clock savings: don't touch the DB at all during the exams. Dump the validated submissions as single field delimited strings to a flat file or NoSQL or similar.

Excellent tip. You could use a key/value store like DB_File to save each submission. If you can construct the form "action" attribute to give it a unique path (perhaps using Javascript on the client side) then you would not even have to parse the query arguments:

<form action="myapp/user1/question1">
<form action="myapp/user1/question2">
<form action="myapp/user2/question1">

Then use $ENV{'PATH_INFO'} as the key ('/user1/question1'), and read from STDIN directly to get the raw value to store.