Writing a Programming Language in Perl

programmer99 has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Writing a Programming Language in Perl by BrowserUk (Patriarch) on Oct 25, 2011 at 19:30 UTC
See Exploring Programming Language Architecture in Perl by Bill Hails. for one of the most complete explorations of doing exactly what the title suggests. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re: Writing a Programming Language in Perl by Your Mother (Archbishop) on Oct 25, 2011 at 20:12 UTC
BUK's suggestion is quite good. IIRC, it's a fun manuscript. I've been meaning to suggest Marpa to you for awhile. While still marked experimental it should be excellent for this kind of thing. I've wanted to play with it myself for a long time but tuits being what they are... Marpa would be used, I'd expect, to convert a language spec/grammar to Perl and then you could use any number of strategies to freeze the resulting code (in memory, file, some other specialized cache) until the original source changed, at which point it could be "recompiled." Doing it this way should afford an easy path for testing and building up a grammar piece by piece. If you don't write tests as you go, you will be sorry down the road. If you do write tests, you'll be able to breeze through wrong turns and accidentally introduced bugs with confidence.	[reply]
Re^2: Writing a Programming Language in Perl by ikegami (Patriarch) on Oct 25, 2011 at 21:39 UTC
Marpa - Parse any Language You Can Describe in BNF Hopefully, it can do a bit more than that, because associativity cannot be described in BNF.	[reply]
Re^3: Writing a Programming Language in Perl by Your Mother (Archbishop) on Oct 25, 2011 at 21:47 UTC
Definitely not my forte but from the outside it looks wonderful. I'd love to hear your impressions of it if you're inclined and have time to take it though some of its paces.	[reply]
Re^4: Writing a Programming Language in Perl by ikegami (Patriarch) on Oct 25, 2011 at 22:15 UTC
Re^5: Writing a Programming Language in Perl by Jeffrey Kegler (Hermit) on Nov 14, 2011 at 17:28 UTC
Some notes below your chosen depth have not been shown here
Re^2: Writing a Programming Language in Perl by programmer99 (Novice) on Oct 25, 2011 at 20:36 UTC
Ok, I have a few questions (if you don't mind). What do you mean by grammar? How will Marpa help me? Thanks.	[reply]
Re^3: Writing a Programming Language in Perl by ikegami (Patriarch) on Oct 25, 2011 at 21:54 UTC
The grammar of a language defines its syntax. Parsers (e.g. Marpa) take a sequence of bytes, characters or tokens, check if it conforms to the grammar, and assigns meaning to the components of the sequence as per the grammar. For example, the job of the parser is to receive "a+b*c" and return "a multiplication of ( an addition of ( identifier a ) and (identifier b ) ) and ( identifier c )".	[reply]
Re^4: Writing a Programming Language in Perl by programmer99 (Novice) on Oct 25, 2011 at 22:30 UTC
Re^5: Writing a Programming Language in Perl by ikegami (Patriarch) on Oct 25, 2011 at 22:32 UTC
Some notes below your chosen depth have not been shown here
Re^3: Writing a Programming Language in Perl by Your Mother (Archbishop) on Oct 25, 2011 at 21:20 UTC
As I say, tuits in the round are the issue and without the time to cook up some working example I'd just be glossing the various documentation and BNF/ParseRec stuffage. I think a simple example can be cooked up but it's also not entirely trivial or my forte (hint, hint to others who might have something to show). This is quite interesting and I've been waiting to try some Marpa though. If I can grab an hour tonight I'll revisit this otherwise, the weekend, otherwise... uh, what were we talking about?	[reply]
Re: Writing a Programming Language in Perl by Corion (Patriarch) on Oct 25, 2011 at 22:10 UTC
In my opinion a very good book is Compiler Construction (pdf warning) by Niklaus Wirth. I've only used the German edition from 1986, but it introduces a small language and how to build a parser and compiler for it. It only teaches you the technique of recursive descent. Recursive Descent is a technique that is currently enjoying a comeback because it is easier with it to produce meaningful error messages than it is when using `lex` and `yacc` or other parsing tools. The German book has 115 pages, the English and more recent edition (from 1996) has 130 pages. This seems to me a good starting point in the sense that little previous knowledge is needed and it gets you a feel for implementing some language with Pascal-like syntax. Actually, in the English PDF, it's an Oberon-like syntax, but that's still close enough. When I first read the book, I skipped over the chapters 1 to 4, as they concern themselves with the theoretical background of the languages that the approach can handle. Looking back over the chapters, I'm not sure whether it was really necessary to skip over them, as they don't seem as dry to me as they did back then. On the other hand, I now have much more experience and also learned the theory of regular languages and context-free languages in university. Browsing through the PDF, I see that it also mentions bottom-up (`yacc`-style) parsing and gives a rough outline of it.	[reply] [d/l] [select]
Re^2: Writing a Programming Language in Perl by andal (Hermit) on Oct 26, 2011 at 08:16 UTC
The work of Crenshaw at http://compilers.iecc.com/crenshaw/ appears to be simpler. But YMMV.	[reply]
Re^2: Writing a Programming Language in Perl by programmer99 (Novice) on Oct 25, 2011 at 23:17 UTC
I scanned through the PDF and it looks really good. I'll get back to you on how good... ;-)	[reply]
Re: Writing a Programming Language in Perl by davies (Prior) on Oct 25, 2011 at 21:34 UTC
Have a look at Jonathan Worthington's slides (http://www.jnthn.net/papers/2011-yapc-russia-compiler.pdf). The code is next door on his web site. But as this was a public talk, there may be a recording of it on YAPC.tv. My spies tell me that the talk was outstanding, writing a complete language (well, almost) from next to nothing in under an hour. Regards, John Davies	[reply]
Re^2: Writing a Programming Language in Perl by programmer99 (Novice) on Oct 25, 2011 at 23:02 UTC
This is great! Tell your spies they were right ;-). Thanks!	[reply]
Re: Writing a Programming Language in Perl by ikegami (Patriarch) on Oct 25, 2011 at 21:36 UTC
If you're interested in parsing and compiling, consider getting your hands on Compilers: Principles, Techniques, and Tools. You won't find any Perl in it, though.	[reply]
Re^2: Writing a Programming Language in Perl by programmer99 (Novice) on Oct 25, 2011 at 23:05 UTC
Is the info in the older version of the book accurate. The price of the newer version is kind of steep...	[reply]
Re: Writing a Programming Language in Perl by JavaFan (Canon) on Oct 26, 2011 at 06:37 UTC
I'd start by first thinking what your new language should do. Is it going to be a domain specific language? Or a general purpose one? If the latter, what is its strength going to be? Is it going to be procedial? functional? OO? Block based? List based? Something else? Second, make some trade-offs. Do you want to write a simple compiler/interpreter? (That probably means a simple syntax, not many features in your language). Does the compiler have to be run fast? Or do you want to give much power to the user of the language (resulting in a more complex compiler)? Only then I'd worry about the implementation. Oh, and my step 0 would be is to ponder about "if I really haven't a fucking clue on how to do that, is my disdain to read a book justified"?	[reply]
Re^2: Writing a Programming Language in Perl by programmer99 (Novice) on Oct 26, 2011 at 09:37 UTC
In a way, I have already thought this out. The issue I am having now how to get started with the actual code. Thanks so much!	[reply]
Re^3: Writing a Programming Language in Perl by spx2 (Deacon) on Oct 26, 2011 at 09:45 UTC
well, just look over Parse::Yapp, that would be your entry point.	[reply]
Re^4: Writing a Programming Language in Perl by anneli (Pilgrim) on Oct 26, 2011 at 11:03 UTC
Re^3: Writing a Programming Language in Perl by JavaFan (Canon) on Oct 26, 2011 at 11:52 UTC
In a way, I have already thought this out. The issue I am having now how to get started with the actual code. So, your question really is, I know what I want to do, but I'm not telling you what it is, yet I ask from you how to get started? Is that how you ask for directions in an unknown city as well? You ask I want to go somewhere, in which direction do I go?	[reply]
Re: Writing a Programming Language in Perl by Fieldrunner (Initiate) on Oct 25, 2011 at 20:55 UTC
progammer99, Your question reminded me of the time my brother offered the solution "just rewrite command.com". The code script or snippet we create is compiled with our perl program (typically /usr/bin/perl) As a humble monk I just wish to point out, it is not a function of Perl to create a New language as it is one itself. I wish you the best on this truly amazing task and one day look forward to seeing it used globally.	[reply]
Re^2: Writing a Programming Language in Perl by anneli (Pilgrim) on Oct 25, 2011 at 22:03 UTC
As a humble monk I just wish to point out, it is not a function of Perl to create a New language as it is one itself. If this was true, then what would you make a new language in? We have only the languages we currently possess in which to make new ones.	[reply]
Re^3: Writing a Programming Language in Perl by ikegami (Patriarch) on Oct 25, 2011 at 22:29 UTC
Don't you know? Languages were created fully-formed by the divine! ;)	[reply]
Re^3: Writing a Programming Language in Perl by Fieldrunner (Initiate) on Oct 26, 2011 at 00:10 UTC
anneli, I am sure others are more sage, I believe that if you write in a language you create an application or program. So not matter what you wrote in Perl it would still be Perl. I believe you would need to create a new compiler with or without a scripting interface to pass instructions directly to the processor. Perl Programming is so more rewarding than SUDOKU, its $var is only (1..9)	[reply]
Re^4: Writing a Programming Language in Perl by Your Mother (Archbishop) on Oct 26, 2011 at 00:55 UTC
Re^4: Writing a Programming Language in Perl by anneli (Pilgrim) on Oct 26, 2011 at 01:24 UTC
Re^5: Writing a Programming Language in Perl by ikegami (Patriarch) on Oct 26, 2011 at 07:13 UTC
Some notes below your chosen depth have not been shown here
Re: Writing a Programming Language in Perl by spx2 (Deacon) on Oct 25, 2011 at 23:43 UTC
There's a lot of theory about "bla grammars bla bla translators bla ... context-free context-dependent bla bla etc". You can just throw that Dragon Book out the window and study the grammar of some small language like Basic or JavaScript's grammar to learn stuff (here you can find a lot of grammars to many many languages). This is not rocket science, it's engineering. So I totally agree with you about not wanting to read 9999 books on the subject, you need to build something. I'm not in any way a compiler/language expert but I do feel that you want to get down to business so.. here's Parse::Yapp, here's an article , here's another article , here's some benchmarks and comparisons between different parser generators for Perl, and another comparison , and another article. If you have further questions, I'm sure the monastery has lots of monks with knowledge of Parse::Yapp so your questions will receive an answer. Now go write a language and good luck !	[reply]
Re: Writing a Programming Language in Perl by sundialsvc4 (Abbot) on Oct 26, 2011 at 02:11 UTC
Creating a programming language from scratch is an interesting exercise. I not only did that, but sold umpteen thousand copies of a system that ~~used~~ uses it. (So there...) But I dunno if I would implement the entire language in Perl. If you do it, I’d love to see it.
Re: Writing a Programming Language in Perl by Anonymous Monk on Oct 26, 2011 at 12:07 UTC
Parsing Strings and Trees with `Parse::Eyapp` (An Introduction to Compiler Construction http://search.cpan.org/dist/HOP-Lexer/lib/HOP/Lexer/Article.pod Lexing Without Grammars: When Regular Expressions Suck http://hop.perl.plover.com/Examples/ALL/calculator http://hop.perl.plover.com/Examples/ALL/regex-parser http://hop.perl.plover.com/linogram/	[reply] [d/l]
Re: Writing a Programming Language in Perl by hbm (Hermit) on Oct 25, 2011 at 23:44 UTC
For ideas, you might look at Brainf*ck, which has a simple grammar of: `<>+-[],.`	[reply]
Re^2: Writing a Programming Language in Perl by Your Mother (Archbishop) on Oct 26, 2011 at 00:56 UTC
Speaking of which, this is worth revisiting for anyone interested or that missed it the first time around: Ook interpreter.	[reply]


go ahead... be a heretic
	PerlMonks