Minions is yet another OOP automation module, roughly similar to Moo, but which has the addtional goal of putting Encapsulation centre stage. I wrote it because after reading How Large Does Your Project Have To Be to Justify Using Moose? (especially the comments by tye and JavaFan), I became increasingly disillusioned with the Moo(se) way of OOP (essentially OOP with no Encapsulation).

package FixedSizeQueue;

use Minions
    interface => [qw( push pop size )],

    construct_with  => {
        max_items => {
            assert => { positive_int => sub { $_[0] =~ /^\d+$/ && $_[0
+] > 0 } },
        },
    },

    implementation => 'FixedSizeQueue::Default',
;

1;

package FixedSizeQueue::Default;

use Minions::Implementation
    has  => {
        q => { default => sub { [ ] } },

        max_size => {
            init_arg => 'max_items',
        },
    },
;

sub size {
    my ($self) = @_;
    scalar @{ $self->{$__q} };
}

sub push {
    my ($self, $val) = @_;

    log_info($self);
    push @{ $self->{$__q} }, $val;

    if ($self->size > $self->{$__max_size}) {
        $self->pop;
    }
}

sub pop {
    my ($self) = @_;

    log_info($self);
    shift @{ $self->{$__q} };
}

sub log_info {
    my ($self) = @_;

    warn sprintf "[%s] I have %d element(s)\n", scalar(localtime), $se
+lf->size;
}

1;
[download]

% reply -I lib
0> use FixedSizeQueue
1> my $q = FixedSizeQueue->new(max_items => 3)
$res[0] = bless( {
         '932db126-' => 'FixedSizeQueue::__Private',
         '932db126-max_size' => 3,
         '932db126-q' => []
       }, 'FixedSizeQueue::__Minions' )

2> $q->can
$res[1] = [
  'pop',
  'push',
  'size'
]

3> $q->push($_) for 1 .. 3
[Mon Jan 26 12:01:53 2015] I have 0 element(s)
[Mon Jan 26 12:01:53 2015] I have 1 element(s)
[Mon Jan 26 12:01:53 2015] I have 2 element(s)
$res[2] = ''

4> $q->pop
[Mon Jan 26 12:02:09 2015] I have 3 element(s)
$res[3] = 1

5> $q
$res[4] = bless( {
         '932db126-' => 'FixedSizeQueue::__Private',
         '932db126-max_size' => 3,
         '932db126-q' => [
                           2,
                           3
                         ]
       }, 'FixedSizeQueue::__Minions' )

6> $q->push($_) for 4 .. 6
[Mon Jan 26 12:02:55 2015] I have 2 element(s)
[Mon Jan 26 12:02:55 2015] I have 3 element(s)
[Mon Jan 26 12:02:55 2015] I have 4 element(s)
[Mon Jan 26 12:02:55 2015] I have 3 element(s)
[Mon Jan 26 12:02:55 2015] I have 4 element(s)
$res[5] = ''

7> $q
$res[6] = bless( {
         '932db126-' => 'FixedSizeQueue::__Private',
         '932db126-max_size' => 3,
         '932db126-q' => [
                           4,
                           5,
                           6
                         ]
       }, 'FixedSizeQueue::__Minions' )

8> $q->log_info()
Can't locate object method "log_info" via package "FixedSizeQueue::__M
+inions" at reply input line 1.
9>
[download]

1. Attributes can be safely accessed inside classes without the overhead of a function call
2. As a consequence of 1., attributes need not be exposed via methods (unless there is a good reason to do so).
3. No need to clean up animal droppings
4. Private subroutines via the mechanism shown in Re: OO - best way to have protected methods (packages)
5. Class methods are "class only", and object methods are "object only"
6. No compatibility with Moose

You've got your typical company started by ex-software salesmen, where everything is Sales Sales Sales and we all exist to drive more sales.

On the other extreme you have typical software companies built by ex-programmers. These companies are harder to find because in most circumstances they keep quietly to themselves, polishing code in a garret somewhere, which nobody ever finds, and so they fade quietly into oblivion right after the Great Ruby Rewrite, their earth-changing refactoring-code code somehow unappreciated by The People.

-- The Developer Abstraction Layer by Joel Spolsky

Though my natural inclination is to be a bit OCD about keeping code clean, I concede that spending too much time and money on refactoring, writing programmer tools, and endlessly polishing code will likely lead to commercial failure. As will the converse, namely neglecting your developers and their code and architectures in favour of sales and marketing. Successful software companies tend to have a healthy balance.

Refactoring

Booking.com, perhaps the most commercially successful Perl-based company, has caused a bit of controversy over the years with their attitude towards refactoring. To give you a flavour, I present a couple of comments below:

Booking is destroying my career because I am not allowed to do anything new. I am not allowed to use new technologies. I'm not allowed to "design" anything big. I am not allowed to write tests. I am allowed to copy that 500 line subroutine into another module. If people have done that several times before, maybe it should be refactored instead of duplicated? If you do that, you get in trouble. As one boss says, "we do not pay you to write nice code. We pay you to get job done."

Management, and the term is quite lose when applied to Booking.com, sees no gain in refactoring code. By refactoring I'm talking about taking a few weeks to rewrite an existing piece of software. By definition refactoring doesn't bring new functionality so this is why management is reluctant to go down that road. We're quite lenient about code that gets added to the repo, as long as there's a business reason behind it. If a quick hack can be deployed live and increase conversion then it will be accepted. But rest assured that crappy code doesn't last long, specially if other devs have to use it or maintain it.

-- from Truth about Booking.com (Blog)

One of the posts specifically deals with the culture of "get it done and fast" and how they do not encourage refactoring or basic testing. I actually work in a Perl shop where management has the same kind of mentality, and it is slowly killing our efficiency.

Regarding testing, it's true that we're not very unit testing focused. This is mainly because we've decided to spend most of the time/money/infrastructure that you might usually spend on unit testing on monitoring instead. If you have unit tests you still need monitoring, but in practice if your monitoring is good enough and you have an infrastructure to quickly rollout & rollback systems you can replace much of unit testing with monitoring.

We're not adverse to refactoring when appropriate. But if you're going to propose rewriting some code here you'll actually have to make a compelling case for it which isn't just "the old code is hairy". Do you actually understand what it does? Maybe it's hairy and complex because it's solving a hairy and complex problem. Are you not aware of where this system fits into the big picture? We've also had code that's looks fantastic, had tests, used lots of best practices that we've had to throw away completely because it was implementing some idea that turned out to be plain stupid.

-- from What exactly is up with Booking.com? (reddit)

Opportunistic Refactoring and The Boy Scout Rule

Some people object to such refactoring as taking time away from working on a valuable feature. But the whole point of refactoring is that it makes the code base easier to work with, thus allowing the team to add value more quickly. If you don't spend time on taking your opportunities to refactor, then the code base gradually degrades and you're faced with slower progress and difficult conversations with sponsors about refactoring iterations.

There is a genuine danger of going down a rabbit hole here, as you fix one thing you spot another, and another, and before long you're deep in yak hair. Skillful opportunistic refactoring requires good judgement, where you decide when to call it a day. You want to leave the code better than you found it, but it can also wait for another visit to make it the way you'd really like to see it. If you always make things a little better, then repeated applications will make a big impact that's focused on the areas that are frequently visited - which are exactly the areas where clean code is most valuable.

-- Opportunistic Refactoring (Martin Fowler)

The Boy Scouts have a rule: "Always leave the campground cleaner than you found it"

What if we followed a similar rule in our code: "Always check a module in cleaner than when you checked it out"

-- The Boy Scout Rule (O'Reilly)

At work, we perform opportunistic refactoring following the Boy Scout rule, trusting the judgement of developers. How do you do it at your workplace?

Code Reviews

By way of background, my company went agile about five years ago, at first with great zealotry, nowadays with more maturity and less dogma.

Before check-in, all code must be reviewed, either continuously via pair programming, or via a lightweight code review (typically over-the-shoulder). We also have a coding standard, though it is not strongly enforced.

To give a concrete example, during a code review the other day, I persuaded the author to eliminate unnecessary repetition by changing this snippet:

my $config = <<'GROK';
ADD UDP_LISTENER ( 515 )
ADD UDP_LISTENER ( 616, 657 )
ADD UDP_LISTENER ( 987 )
GROK

my @test_cases = (
   {  desc => "# Test 1",
      conf => $config,
      find => [ 'port = 515', 'port = 616', 'port = 657', 'port = 987'
+ ],
   },
);
[download]

my $liststr = 'ADD UDP_LISTENER';
my @ports   = ( 515, 616, 657, 987 );

my $config = <<"GROK";
$liststr ( $ports[0] )
$liststr ( $ports[1], $ports[2] )
$liststr ( $ports[3] )
GROK

my @test_cases = (
   {  desc => "# Test 1",
      conf => $config,
      find => [ map { "port = $_" } @ports ],
   },
);
[download]

What would you have done?

I'm sure some other programmers at my company wouldn't have bothered suggesting any changes at all: after all, the code worked as is, it's pretty clear, plus "it's only a test script", so why bother?

Though I felt the code was more maintainable with duplication eliminated, I had another motivation in this specific case: training. You see, the programmer in question was very new to Perl and, as I found out during the review, had never used map before! Training (and improved teamwork) are important benefits of code reviews.

Eliminating unnecessary duplication and repetition is a common discussion topic during code review in my experience. (Note: I did not include this example to argue further about what DRY means exactly in Room 12A :). Other common discussion points during code review are:

• Commenting.
• Naming.
• Clarity vs Cleverness.
• Encapsulation.
• Interfaces.
• Error handling.
• Testability. Is the code testable in isolation?
• Supportability.
• Portability.
• Security.
• Performance.

I'm interested to learn about your workplace experiences. In particular:

• Do you have a coding standard? How is it enforced?
• Do you do pair programming?
• Do you do code reviews? Are they heavyweight (e.g. Fagan Inspection) or lightweight (e.g. over-the-shoulder)? Mandatory or optional?
• What are common discussion points during your code reviews?

Cleverness

To finish, here's another one, derived from Clever vs. Readable.

Would this statement pass your code review?

my $value = [ $x => $y ] -> [ $y <= $x ];
[download]

my $value = $x < $y ? $x : $y;
[download]

use List::Util qw(min);
my $value = min( $x, $y );
[download]

References

• On Coding Standards and Code Reviews
• Why Create Coding Standards and Perform Code Reviews?
• Nobody Expects the Agile Imposition (Part VI): Architecture
• The Developer Abstraction Layer by Joel Spolsky
• The Boy Scout Rule (O'Reilly)
• Opportunistic Refactoring by Martin Fowler
• stack exchange discussion on refactoring
• What is Clean Code? by Robert C Martin
• Broken Window Theory

It did not, nor did an earlier attempt using than Scalar::Util's dualvar. Still I liked my little Empty but True module enough to post it here. (Seems too useless/dangerous/not worth the bother to be on CPAN.)

package MyEmptyTrueVar;
our $singleton;

use overload fallback => 'TRUE',
  '""' => sub { "" }, # Return empty string on stringification
  bool => sub { 1 },  # Return true in boolean context
  '0+' => sub { 1 },  # Return true in numeric context
  cmp  => sub { !ref $_[1] }; # unequal to empty string, or any other 
+string

bless $singleton=\$singleton;
[download]

say "Single str='$MyEmptyTrueVar::singleton', Num=", (0+$MyEmptyTrueVa
+r::singleton)
   if $MyEmptyTrueVar::singleton &&
   $MyEmptyTrueVar::singleton ne '' &&
   '' ne $MyEmptyTrueVar::singleton;
[download]

Computers can do so many operations per second, so fast, that sometimes are considered omnipotent. But, sometimes an approach a little too direct or dumb can transform our fastest machine in a cart ...

This is not the discovery of the century, I know, and every good programmer pays attention to the issue, but sometimes I stop to think how is easy to turn a good-performing program in a disaster.

Just yesterday, I had to check about 10,000 files in a directory from a database of about 20,000 records: the iterating solution a-query-by-file it take an hour; by loading the entire table in a hash and then looking information it take few seconds.
So far, not the sense of life, but a thing on which I like to meditate.

Hello fellow monks and monkesses.

I am very new to the monastary as far as being a member goes, however have been a frequent part of the 'flock' if you like, for some time.

My meditation today comes from my recent personal experience of applying for a proffessional role as a PERL programmer - which leads to my pondering, 'what do i have to know/should know in order to be qualified for such a position'? I have read many articles before such as Professional perl which are obviously inciteful in this area.

I am really looking for specifics such as a minimum amount of required knowledge in certain areas such as OO Perl, or Database management with Perl, Optimisation, Threads or Web programming in Perl etc. I have only self tought knowledge of PERL, and of course with helpful textbooks, tutorials and articles in the monastary have grown this knowledge over time. I don't however, have any professional experience of PERL per say, other than small scripts I have written for my current job as a C language software Engineer.

SO - I do have a relatively small amount of knowledge in some areas in comparison to, lets say, what I would class as the contrastingly very experienced monks around here, whom provide a vast array of knowledge in most if not all 'Areas' of PERL. I have yet to receive any feedback from my recent application, however 'I' feel like I may be underqualified. It would be very helpful if my fellow peers may be able to offer some incite into 'what the minimum requirements' may be for such a junior PERL programmer role.

Apologies in advance if my question is unhelpful,in the wrong place, poorly titled or worded.

Your ever faithfull functioning perloHolic.

In the earliest days of digital computing, memory was the most-scarce resource.   (After all, magnetic doughnuts could be made only so small, and they had to be strung upon three tiny wires by hand.)   Thus, when external storage devices – disk and tape and drum – were developed, they were referred to as “mass” storage.   Most of us remember the bad old days, of MS-DOS and even earlier, when programs had to be separated into “overlays” in order to be made to fit into the one-and-only place from which instructions could be executed.

Well, fast-forward a few decades and “chips are cheap” now.   Gigabytes if not terabytes of semiconductor storage can fit into a space the size of a pencil eraser.   CPU architectures are especially designed to allow gigabytes of storage to be directly addressed.   Developers are eagerly taking advantage of this, because the days of “overlays” and then “virtual memory” (i.e. “paging”) appear to be long-ago and far-away.   After all, RAM has the unique advantage of being instantaneous.   If the data is available in RAM, disk-I/O does not occur.   (And, if the “disk” storage device is made of semiconductors, at least the “seek latency” and “rotational latency” does not happen, even though most semiconductor devices do have a certain form of “latency” of their own.)

There is a fly in that ointment, however.   RAM capacity is not yet so enormous that concerns about virtual memory can be dismissed outright, especially in production situations where many possibly memory-hungry applications are running on the same box at the same time.   Virtual memory is still with us, and therefore we must be mindful of how to work “with” it and not “against” it, just as we were very-obliged to do in the 1970’s.

When virtual memory is being used, “in-memory” data structures might involve disk I/O.   As you know, physical RAM is divided into equal-sized chunks called “pages,” and each page might be “resident” in memory or it might be “swapped out.”   When any virtual address is touched, a “page fault” might occur, and if so the process will be delayed until the necessary disk-I/O has been completed.   (And, in order to make room for the page, another page might have to be “stolen” from someone and written out to disk … thus, two or more disk-I/O’s must take place before the faulting process is allowed to proceed.)

Virtual memory’s success relies upon the assumption that, while page-faults will occur, they will not occur so frequently that the disk-I/O delays add up too much in practice.   The term is “locality of reference,” and it means that programs typically make memory-references in very concentrated groups.   Once a page-fault is satisfied, things should settle-down for a while as the processes continue to refer, again and again, to the page(s) that have most recently been faulted-in.   “Least Recently Used (LRU)” pages, by comparison, are presumed to be good candidates for page-stealing.   The total set of pages that any process requires in order to run without delay, at any instant in time, is referred to as its current “working set” at that instant.

There is, unfortunately, one data-structure mechanism in particular that flies in the face of “locality of reference,” and therefore of “small and tidy and predictable working-sets.”   That mechanism is:   the hash table.   Perl’s “hashref.”

Hash tables work by permuting a key across some smaller key-space in order to arrive at a single “bucket” that is searched for the target value.   Hash functions are designed to spread the key values more or less uniformly, but randomly, across the key space.   Thus, the hash structure itself can represent a large working-set (although hash algorithm designers, including Perl’s, do seek to constrain this).   But in any case, the hash buckets also refer, by reference, to outside blocks of memory that are obtained using memory allocation functions e.g. “malloc().”   The memory addresses pointed-to by the (already, large) hash table will, over time, become quite-widely distributed.   And so we have a “very random-access” data structure:   a large hash-table referencing an even larger set of memory blocks whose individual addresses are not predictable.   (A highly volatile very-active data structure becomes less and less predictable as the hours and days go by.   Working-set sizes increase quickly.)

(Perl’s designers know their stuff.   They know about these issues and carefully designed an industrial-strength system for all of us to enjoy.   We are well cared-for ... but the issues are still there, and, by definition, always will be.)

Working-set sizes become very large, then.   So, what actually happens when such an application enters service in a production machine that’s using virtual memory?   Unfortunately, it becomes a million-pound elephant … using, shall we say, far more than its fair share of RAM.   A disproportionately large amount relative to the others.   And therefore, both a source of virtual-memory pressure and(!) an especially vulnerable victim of it.   If such a program is to run efficiently (as it was specifically designed to do), it must have “all that RAM.”   But, if it gets what it wants (and must have), the other processes can’t get (and keep) theirs.   Paging activity begins to increase, as does the number of processes that are stuck in page-wait and the frequency that each process is stuck in page-wait.   At a certain point, the processing grinds to a halt.   It “hits the wall.”   It is “thrashing.”   The offending application is especially taking it in the shorts ... being especially big and especially vulnerable, it is “punch-drunk.”   But it’s not the only one.   (If there were any subtle timing-related bugs in this or any other application, this is the time when those kinds of problems will really start to show up.)

Given that, in a virtual memory setting, any “memory” reference can result in “disk” I/O, “memory” must in fact be treated as a “disk.”   Each memory-access, especially any access that might be widely-dispersed from other recent ones, must be considered as possibly taking several milliseconds to occur; not the microseconds or nanoseconds that are usually bantered-about by programmers who like to use the “time” command and discuss the third or fourth digit to the right of the decimal point.

Software developers usually don’t experience these things personally when designing their software:   their systems are the biggest, fastest, and especially the fattest of all.   They’ve got two or three large monitors.   Multiple processors.   Gigs of RAM.   As much as the motherboard will support.   In short, a situation altogether unlike the rack mounted boxes where their brainchildren will labor out their appointed business duties.

To run well, and to do so round-the-clock for days and weeks on end, all applications must be good virtual-memory citizens.   Whether their virtual memory allocations be large or small, their working-set sizes must be small … by design.   There are many possible ways to do that:   storing multiple entries in a single large structure rather than in many small ones; “batching” requests for even in-memory data stores; and, using disk-I/O directly instead of implicitly (as virtual-memory actually does).   All operating systems buffer disk-I/O operations, filling all available RAM with buffers but managing those buffers differently than they do VM page-frames.

Probably the very worst thing that can happen to your program’s design is for it to be very splendid on your machine, but unworkable in production … or even, “a pain in the asterisk in production.”   This requires thinking of RAM as being “a thing that is not without-delay,” from the very earliest stages of your designs.

HTH …

Presenting two ways to skim tar format files: via direct parsing and using the specific module.

The file date of an archive is useful to keep around for chronological listings, or determining its age at a glance. It is however often times lost as the files get downloaded, copied or moved. An obvious fix is to reset the date to that of the most recent member contained within. And a script to this end is what I implemented, years ago. If there is or was a proper tool for that already, I wouldn't know.

But old TODOs came to my attention again recently. What better time to clean up some old code, perl-based and all? In particular, there was this bit to decompress the files with an external utility:

... ? ...
: $file =~ /bz2$/i ? open($fh, '-|', 'bzcat', '--', $file)
: open($fh, '<', $file);
[download]

Minutes later, there it is — the version II — shorter and neater by a fair bit.

Hacking on them tar headers is entertaining for sure, but let's try Archive::Tar now — a module purpose-built for tasks like that. And behold: the version III.

#! /usr/bin/perl

# Usage: $0 [-z] [-n] files ...
#       -z  also check gzip archive time
#       -n  don't actually touch, show what would be done

use Getopt::Std;
getopts('zn', \my %opt);

use List::Util q(max);
use IO::Uncompress::AnyUncompress;

use Archive::Tar;
$Archive::Tar::WARN = 0;

foreach (grep -f, @ARGV) {
        my $fh = new IO::Uncompress::AnyUncompress($_) or next;
        my $zt = $opt{z} && ($fh->getHeaderInfo||{})->{Time} || 0;
        my $tt = max map $_->{mtime}, Archive::Tar->list_archive($fh, 
+0, [q(mtime)]);
        my $t = max $zt, $tt;

        $t && ($t != (stat)[9]) && ($opt{n} || utime($t, $t, $_)) &&
        printf "%-60s %s (%s)\n", $_, scalar localtime($t),
                ($t == $tt) ? q(tar time) : q(zip time);
}
[download]

• Lots of memory is consumed reading big archives. Apparently list_archive method reads the uncompressed data in full. Is there no "metadata-only" flag one could peruse?
• Another thing, list_archive has special cased the [q(name)] to return a flat list instead of hashes. Why not support both [qw(...)] and q(item) requests? Then one might simply write:
  my $t = max Archive::Tar->list_archive($file, 1, "mtime");
• The lzma/xz modules need to be installed separately for those to work. Release 5.20.1 does not (yet?) include them.

Giving it a second glance, the original script seems to do fine as it was. Some TODOs may stay a while longer, I think.

t/middleware/connect.t ....... 
Dubious, test returned 5 (wstat 1280, 0x500)
All 1 subtests passed 
Free to wrong pool 2e8aa90 not 6c4040 at C:/strawberry-perl-5.12.2.0/p
+erl/site/lib/AnyEvent/HTTP.pm line 1083, <> line 6.
t/middleware/loadbalancer.t .. 
Dubious, test returned 5 (wstat 1280, 0x500)
No subtests run 
Free to wrong pool 2e145e0 not 34f90 at C:/strawberry-perl-5.12.2.0/pe
+rl/site/lib/AnyEvent/HTTP.pm line 1083, <> line 6.
t/middleware/rewrite.t ....... 
Dubious, test returned 5 (wstat 1280, 0x500)
No subtests run 
[download]

#define EIO             5
[download]

C:\perl521\bin>perl -E"$! = 5; say $!"
Input/output error
[download]

C:\perl521\srcnewb4opt>perl -Ilib crashtest.pl
disable_interrupts $? 9600 CHILD_ERROR_NATIVE 9600
illegal_instruction $? 1d00 CHILD_ERROR_NATIVE 1d00
deref_null $? 500 CHILD_ERROR_NATIVE 500
deref_neg1 $? 500 CHILD_ERROR_NATIVE 500
write_to_ro_mem $? 500 CHILD_ERROR_NATIVE 500
div_by_0 $? 9400 CHILD_ERROR_NATIVE 9400
call_c_debugger $? 300 CHILD_ERROR_NATIVE 300

C:\perl521\srcnewb4opt>
[download]

Following on from The Top Ten Perl Obfus, let's count down the top ten highest rated Perl Monks poems of all time.

Since I cannot super-search by node reputation, please note that this list is based only on some non-exhaustive searching and my fallible memory. So it's quite likely I've overlooked a deserving poem. If so, please let us know, and I'll correct the root node. Note that, to make the top ten, a poem needs a reputation of at least 120.

That said, please feel free to mention any poem you consider deserving of a wider audience, even if it does not meet the formal reputation criteria. For example, I'd like to recognize and congratulate liverpole for pulling off a brilliant stunt of posting a poem entitled 600000 nodes as the 600000th PerlMonks node!

Unlike obfus, I discovered the top ten qualification criteria for poetry is not so clear-cut. For example, what many folks believe to be the finest Perl Monk node of all time, namely 1st Monasterians by Erudil, was posted not as a poem, but a meditation. Though somewhat poetic, I judged that this node did not qualify because it was not a Perl poem and was not posted in the Perl Poetry section. Curiously, a response to this node, namely Re: 1st Monasterians by japhy, did qualify because, though it too was not posted in the Poetry section, it was definitely a Perl poem. Conversely, though posted in the Perl Poetry section, I chose to disqualify Aaah, spring (A Very Special Perlmonks Contest) by boo_radley because it was a poetry competition, rather than a poem. Admittedly, these decisions were somewhat arbitrary, and someone else may have decided differently.

Now to work.

No 10: Stayin' Alive (with CPAN) by joecamel Feb 05 2004 rep:120+

Well, you can tell by the way I use File::Lock
I'm a Perl Monk: no time to talk
Got DBI and Test::More,
been reusin' code since version four

You know it's all right. It's okay.
With GD::Graph and Class::Flyweight.
We don't have time to reinvent
so we get it from CPAN.

Whether you're a hacker or whether you're a slacker
You're stayin' alive, stayin' alive.
net communicatin' and input validatin',
And we're stayin' alive, stayin' alive.
Ah, ha, ha, ha, stayin' alive, stayin' alive.
Ah, ha, ha, ha, stayin' alive.
...

To the tune of Stayin' Alive by the Bee Gees.

I just uploaded the first Alpha version of Crypt::U2F, which allows you to handle the server side cryptography of the FIDO alliance's Universal 2nd factor authentication method. See also here.

This is the same one used by Google services and fully supported in Google Chrome.

Internally, Crypt::U2F requires Yubico's libu2f-server library installed on your system. I implemented this in two Perl modules: Crypt::U2F is the low level module (sand subject to change), that let's you play around with the underlying library. Crypt::U2F::Simple is the one you should use in most cases.

Let's have a look into the two examples provided with the tarball. For this to work, you need to install libu2f-server and also install libu2f-host, because we need the u2f-host binary to talk to the actual USB dongle. (I'm currently in the process of making a Perl module for libu2f-host as well, but this will only finish after the hollidays.)

The whole thing is a two part process: First you have register a new key once, then you can authenticate as often as you like. Each part (registering, authentication) itself is a two-part process as well, first you generate a challenge and send it to the client, then you have to validate the response.

Ok, let's start with registering a key. For this example, we pass around files to and from u2f-host and also save the registered keyHandle and public key into files as well. In a real world scenario, you will probably use HTTP and Javascript to communicate with the key and save keyHandle and the public key into a database. Here's the code:

The reason we use Base64 is simple, yet annoying: Everything except the public key is either some sort of text or even ASCII JSON. The public key on the other hand is a binary blob. It's just a matter of convenience to turn it into Base64, because that we it works in textfiles and text columns in databases as well. It don't convert directly in the library, because that might make it problematic to cooperate with other implementations of U2F authentications that also use the original C library (which delivers a binary blob), including the u2f-server example binary that comes with it.

All of the calls to Crypt::U2F::Simple may fail for one reason or another (including new() and DESTROY()), so make sure you check all the return values!

Let's tackle the authentication part. We'll use the keyHandle.dat and publicKey.dat generated in the previous step:

As you can see, the process is quite similar: We load the keyHandle.dat and publicKey.dat (the second one we decode_base64()) and initialize Crypt::U2F::Simple with it. Then we generate a challenge and verify it.

If you want to make sure the verification step actually works, you can comment out the call can try to fuss the result of u2fhost in authReply.dat. Or just comment out the call to u2fhost after you you did one successfull authentication, this one should give you a u2fs_authentication_verify (-6): Challenge error.

Limitations and Bugs: Currently (Version 0.10), each Challenge/Verify combo has to run in the same instance of the module. I'm still working on finding out how to fix that. Also, sometimes the USB keyfob seems to be in a strange state after plugging in, returning wrongly calculated authentication replies (at least mine does). Unplugging and replugging solves that problem.

Following on from The First Ten Perl Obfus, I thought it would be fun to count down the top ten highest rated Perl Monks obfus of all time.

Since I cannot super-search by node reputation, please note that this list is based solely on my memory of spectacular obfus I've seen over the years. So, if I have overlooked an obfu gem, please let us know, and I will correct the root node. Note that, to make the top ten, a node needs a reputation of at least 240.

No 10: Fun With Reserved Keywords by blokhead Sep 11 2003 rep:200+

#!/usr/bin/perl

not exp log srand xor s qq qx xor
s x x length uc ord and print chr
ord for qw q join use sub tied qx
xor eval xor print qq q q xor int
eval lc q m cos and print chr ord
for qw y abs ne open tied hex exp
ref y m xor scalar srand print qq
q q xor int eval lc qq y sqrt cos
and print chr ord for qw x printf
each return local x y or print qq
s s and eval q s undef or oct xor
time xor ref print chr int ord lc
foreach qw y hex alarm chdir kill
exec return y s gt sin sort split
[download]

Constraints are the heart of obfu.

Here blokhead constrains himself to using lowercase alphabetic characters only, no punctuation at all. Combining with an exact right hand margin produces a visually stunning and surprising block-shaped obfu.

Following on from The First Ten Perl Monks, I thought it would be fun to explore the origins of PerlMonks Obfuscated code.

What was the first PerlMonks Obfuscated code?

As far as I can tell, it was written on Oct 13 1999 by the early PM developers at the very end of this ancient command line examples faq. Of course, this was well before the PerlMonks official opening on 23 Dec 1999. So, if we disqualify this pre-historic (accidental) obfuscation, the first deliberately obfuscated PerlMonks node was probably OBFUSCATE!!! by the thirteenth Perl Monk jdube on Dec 30 1999 at 04:51.

PerlMonks Obfuscation Founding Father: jdube

As was typical of early Perl Monks, jdube also had an everything2 account. Curiously, jdube further authored a companion POETRY!!! node, presumably a crude attempt to pressure vroom into creating PerlMonks Obfuscated code and Perl Poetry sections. Well, by using all caps and three exclamation marks in the node titles he was certainly shouting at vroom.

Anyway, it seems jdube's clever ploy worked because vroom did indeed create our much loved Obfuscated code and Perl Poetry sections later that same fateful day, Dec 30 1999. By the way, I was shocked to see that the historic Poetry and Obfuscated Sections by vroom has received just one up-vote (mine). If you feel this historic node is worth more than that, you know what you need to do.

An interesting piece of trivia is that jdube's medieval OBFUSCATE!!!/POETRY!!! barrage took place in the (now obsolete) perlcraft arena. It seems this ancient "perlcraft" section has since been re-branded as Cool Uses For Perl.

Tragically, our PerlMonks poetry and obfu founding father jdube does not appear in the first ten official Obfuscated nodes, listed below. Sadly, he never did write an official PerlMonks obfuscated node. Sadder still, Perl Monk number 13 jdube was last seen wandering around the monastery in May 2000, and his account now sits abandoned and disabled. Perhaps being the thirteenth user proved unlucky. I wonder what jdube is doing nowadays. Will he return one day to write us another obfu or poem?

Obu No 1: #!/usr/bin/perl by BBQ (last here Apr 07 2009)

Created: Dec 30 1999, Rep: 14, 3 replies.

      %A=('r'=>"\n","\t"=>'#','/'=>
     's','f'=>'p',"b"=>'n');@C=qw (e
    ! r/ / e );foreach $k (sort keys
   %A){ @B=(" ",'u','i',"\b",'l');$s
  .=qq{$k$B[$x++]$A{$k}$C[$x] \b};if
 ($x==1){$t=$k.$B[$x-1].$A{$k};}#ops 
} print$A{r}.$s;#i h8 left over \s's
[download]

Code speaks louder than words.
Here I go:

Yep, python is a nice lib... Long live Inline::Python !

while (<>) {
    if (/^#ERRORS/ .. /^CELLS/) {
        push(@errors, $_);
    } elsif (/^CELLS/ .. /^\s*$/) {
        push(@errors, $_);
    } else {
        ...; # other processing
    }
}
[download]

Which doesn't work.

Occurs to me that it might be a useful enhancement to allow .. conditionX to be an additional stage to the preceding condition1 .. condition2, creating a linked cascade of flip-flops:

if    (/match1/ .. /match2/) { doStage1(); }
elsif (.. /match3/)          { doStage2(); }
elsif (.. /match4/)          { doStage3(); }
else                         { doOtherProcessing(); }
[download]

The semantics would be an extension of .. In the second example, match1 would trigger stage 1 (doStage1() will be called). Then match2 will end stage 1 and trigger stage 2 (doStage2() will be called). Then match3 ends stage 2 and triggers stage 3 (doStage3() will be called). Finally, match4 resets the cascade.

Thoughts?

Updated to mention the first example doesn't work.

Updated second example and the description of the semmantics.