If you really want to know how to write an engine you probably need to start with some compiler theory. The traditional text for this is the dragon book ISBN 0201100886
by Aho and others. Other treatments may or may not be more helpful. Try searching for 'compiler design' on your favorite on-line bookseller. These books explain parsing theory and their state machines.
None of the above is useful for writing regexes that work well. For that follow the advice offered earlier (especially the advice to read Friedl's book on regexes).
Update:Added isbn tag.