I write #perl6 summaries
. An AM recently said "This is nice. Also can you give a percentage progress of how much of rakudo matches the spec as it exists today."
I don't think we can (reasonably easily) do exactly as the AM requested. I am also concerned about encouraging the impression Perl 6 is not feature complete (for some users and use cases it is now sufficiently complete).
However, I would like to publish a metric that helps the AM and interested parties track Perl 6 development. I would appreciate constructive feedback to my response and thoughts below.
TL;DR As a metric to help track Perl 6 development, I favor test pass rates such as spectest passes over time or "64 modules passing 100% of their tests when compiled using the 2012.08 Rakudo compiler". Do you like these?
Any metric will almost certainly have to be based on data that's already being collected. I could see trying one or more of the following:
- Tracking completion of very high level features of compiler implementations. What do you think of Feature completion pie charts? This is drawn from the same data used for the compiler comparison matrix.
We could update the pie chart weekly.
This data currently tracks 143 very high level compiler "features", ones that have been mostly implemented in five years or so (Rakudo) or 2 years or so (Niecza) for an average implementation rate in the order of 20 - 40 per year.
- 143 entries is a small list. The rate of change is slow. Many weeks it wouldn't change. It will be non-trivial to expand this list in a way that significantly increases its utility for tracking Perl 6 development (as against comparison among compilers).
- Some important "features" are not included in this compiler feature list. Consider the (awesome!) debugger that was just added to the latest Rakudo Star bundle. The compiler features list didn't include a debugger. (As it happens, the spec didn't say much either -- "needs further specification", and the Rakudo roadmap didn't even mention it.)
I suspect the fact this feature list doesn't include the debugger reflects weakness for tracking what the AM cares about.
- There are non-essential items in the compiler features list. For example, junctions, hypers, and races are spec'd to optionally autothread (automatically provide transparent concurrency). Rakudo implements junctions and hypers but does not autothread.
Do we count Rakudo as having the junction and hyper features? If we do, folk are likely to complain that this is misleading because it doesn't yet do autothreading. Do we include the optional autothreading in the "features not yet implemented" count? If we do, folk are likely to complain that Rakudo isn't ready because it's not "feature complete".
- Tracking test passing percentages. Consider the Perl 6 test suite and the related perl6 pass rates.
This data generally changes at least a little from week to week.
I think this is about the best data we have that can quickly convey progress but publishing it has tended to elicit criticism ("Test count is a silly metric.").
In the past flussence has generated charts of spectest passes over time. He gave up on it recently, perhaps due to it closing in on 100%, or maybe due to the negative responses we've gotten from publishing such data, but I imagine it would be fairly simple to restart this.
There are module smoketesting results. We could list how many modules in the Perl 6 ecosystem are passing 100% of their tests (64 as of 2012-09-05 tested against the August Rakudo compiler.)
A couple days ago mst uploaded Perl 6 to cpan. That's going to bring in a lot of interesting platform test data. Maybe we track that?
- Tracking compiler roadmap edits. The Rakudo roadmap currently contains 60 items.
The roadmap is probably even less suitable as the subject of a weekly update than either of the previous two items. But I thought it worthy of mention here because it is a very important project driver.
A few weeks ago (on August 20th) I wrote How things are going compared to rakudo roadmap using jan 1 2012 as baseline.
Since I wrote the above gist there have been two roadmap changes. Removing (marking as done) the one star (very easy) task "sigilless variables" and removing the five star (very difficult) task "design, implement and switch to QAST".
(While we're here, let's summarize the remaining roadmap tasks. There is just one task marked as top priority: "basic Perl 5 interop (use, eval, etc.)". Interestingly it is not marked as being difficult. Then come 38 medium priority tasks (only two marked difficult); 17 low priority tasks (only one marked difficult); and one task of unspecified priority and difficulty ("Correct type smiley support (:U, :D, :T, etc.)").)
Do you particularly like or dislike any of the above? Any other ideas?
Edited to correct statements of roadmap task difficulty.