Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re^10: Block-structured language parsing using a Perl module?

by BrowserUk (Patriarch)
on Aug 18, 2012 at 10:16 UTC ( [id://988187]=note: print w/replies, xml ) Need Help??


in reply to Re^9: Block-structured language parsing using a Perl module?
in thread Block-structured language parsing using a Perl module?

I really don't see any need for the lexer to do all the machinations you're worrying with.

I don't think you read what I wrote ... or I wrote it badly. I'm not worrying about any machinations in the lexer.

I don't want (and assert, shouldn't have) to write my own lexer.

it's pretty straightforward,

For this language may be so, but imagine how much more straight forward it would be if you didn't have to write it.

By definition, the grammar contains all the terminal symbols, and how those terminal symbols can be combined.

It could produce your %keywords hash for you from the grammar, and in the process ensure that the grammar and the lexer's hash of tokens, remain in synchronisation.

But more than that, it also knows at each stage what token(s) are possible next in the language going forward from the point it is currently at, so it could inspect the next part of the data and very quickly determine whether what is there makes sense in context.

I assert, it not only could, it should.

and it doesn't worry about tokens other than the current one.

One example does not an (counter-) argument make.

More to the point: You extract the next token and pass it to the parser, and the parser rejects it.

What do you report? What can you report? About all you can say, given your lexer's lack of context, is:

[source.file:123:31] Numeric literal '123' not expected at this time.

Not so helpful.

Whereas the parser could report something like:

[source.file:123:31] parsing 'while', expecting '('; got '123'

Which would you prefer?

Writing character-by-character grammar rules to recognize numbers, keywords, strings, comments, etc. would be a pain in BNF. I don't really look forward to writing a zillion BNF productions to specify what tokens look like character by character. But that sort of thing is trivial for regexes, so I split the lexer out into a simple bit of regex code to create the tokens, and the grammar is relatively straightforward too.

But why would you? Why not supply your identifier syntax; literal syntax etc. to the parser as regex!

I don't anticipate changing anyone's mind immediately. I'm expressing my reasoning for rejecting the module, but that won't make it disappear from cpan, or stop anyone who want to use it from doing so.

The job I'm taking on that requires a real parser, is sufficiently long-term and complex, that it is worth my trying to avoid the duplication of effort, and parallel resource maintenance, that I see being required by using Marpa.

Even if that means writing my own parser than generates a lexer as a part of the grammar compile step.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

Replies are listed 'Best First'.
Re^11: Block-structured language parsing using a Perl module?
by roboticus (Chancellor) on Jul 20, 2013 at 16:30 UTC

    BrowserUk:

    I've been meaning to follow up on this post for some time, but was recently strapped for 'round tuits'. I don't think you wrote it badly, more like I just disagreed. The followup I was going to post was a set of reasons that it's a good idea to split the lexer and parser into different sections. However, with the new additions to Marpa, I'm rather hard pressed to think of a good reason. ;^)

    I don't know if you've been following Marpa::R2, but they've been going in the direction of putting tokenizer rules in the grammar, and it looks rather good. If I were to start over on my project, I think I'd do it that way. (In fact, I've been migrating toward the scanless grammar, I just haven't moved my tokenization in there yet. I'm currently working on getting some of the semantic actions worked out. (As you can tell from the time between posts, this is just a side project, and I've not been working terribly hard on it of late.)

    So if you're still working on your project where you need a good parser engine--or are going to start another--I'd encourage you play with Marpa for a couple of hours to see if it works well for you.

    ...roboticus

    When your only tool is a hammer, all problems look like your thumb.

      I don't know if you've been following Marpa::R2, but they've been going in the direction of putting tokenizer rules in the grammar, and it looks rather good.

      No, I haven't. From my brief interactions with the author I had no expectation that Marpa would ever change in any way that would cause me to reconsider it.

      From a cursory scan of the R2 docs it does seem that somewhere amongst the close to 2 dozen modules that make it up there might be something that starts to look like it might do the job. But, it is really hard to tell given that the two examples are:

      • an expression parser that uses about the same number of source lines as I did for my 6 function/3 precedence level expression parser -- on top of those 2 dozens modules -- to provide a 2 function/1 precedence level expression parser.
      • The other deals with one of those completely pointless, meaningless, "languages" that does absolutely nothing, that parser theorists are so enamored of.

      If I ever find a working example of a block-structured language done using Marpa, I might look again; but I won't hold my breath.

      Unfortunately, the documentation doesn't seem to have improved. It still spends an inordinate amount of time telling me haw clever the parser is; and almost none showing me how to use it to do something realistic and useful.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://988187]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chanting in the Monastery: (3)
As of 2024-03-29 01:37 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found