in reply to Re^5: aXML vs TT2
in thread aXML vs TT2

The parser basically works in a two phase process, first it scans using a very fast non-backtracking regex, to see if it can find any opening tags it recognises. If it succeeds it marks the tag with a control character, then runs a looped slower regex which scans for a complete open and close tag set. When it has something it knows is valid, and does not contain a nested tag set (as determined by negating the control character), it then executes the relavent code and substitutes the return value into the document. The loop continues until there are no more matching sets to process.

The first phase negates the backslash to prevent it from picking up on and marking out the close tags so the tag you mentioned will probably be ignored by both phases and remain untouched in the final output. I'd have to check back on the actual regex used to be certain but I'm pretty sure that is correct.

Replies are listed 'Best First'.
Re^7: aXML vs TT2
by anneli (Pilgrim) on Oct 22, 2011 at 05:39 UTC

    Have you considered writing a proper state machine-based lexer/parser? It would positively fly, compared to using regular expressions, and probably end up more amenable to extension (if you wanted to actually support XML, for instance).

      I have made many attempts at building a character by character state-machine for it, all of which failed miserably and ended up breaking the functionality in some way or other.

      I do believe it is possible to construct such a thing, and I do believe that if I had time and inclination to do so that eventually I would solve the problem, but since I have now adopted Plack as the basis for the system to sit upon I'm already getting close to a thousand hits a second out of it using a Quad Core Phenom II as it stands. (16,000 hits in just over 18 seconds, last time I ran my stress testing script)

      Given that 24 and 32 processor server systems are readily available and 100+ processor core chips are already in production by companies such as tilera, I don't feel at this stage that there is much point to further optimisation as all the "low hanging fruit" has already been had by optimising the regexes and removing backtracking etc where possible.

        1000req/s is good, though what's the backend doing then? Also, what kind of concurrency have you got going with that benchmark? I've developed a few proprietary high-availability/high-throughput web servers, and you learn very quickly that what you thought was great performance can fall down around you when you start passing -c to ab.

        Testing one such thing I wrote now; it's used in production with months of uptime (i.e. it's very stable and feature-complete), and hooks into a variety of networks and network types.

        At 1000 concurrent requests, it's serving about 18,000req/s on my not-too-impressive Core i5-450M (notebook) -- and that's because it's doing nothing but serving a very simple static page (and because it's written in C). The most complicated thing it's doing is reading the request, dispatching it to the correct endpoint, and writing the response into a buffer to be shunted onto the network.

        My point (other than comparing apples and oranges) is that, being the underlying framework, you can never be too fast, as you don't know what someone's going to put on top of it. Serving tens of thousands of requests per second is a nice baseline. Frankly, so is serving nearly a thousand, but just a reminder not to settle for "good enough".