The Internet is full of lexical analyzers, as this was one of the
favorable hobbies before the arrival of html and windows widgets:
you are re-inventing the wheel. How about lex(1), flex(1), or even
a full parser like RecDescent (if you don't mind the slow speed.)
in reply to Lexical Analyzer with Actions : Continue Development?
Lots of analyzers out there. Many parsers can also act as lexers,
and some (rich) lexers like flex(1) can almost act like parsers.
There is also a perl binding to flex(1) through my cpan