Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Meditations

( #480=superdoc: print w/ replies, xml ) Need Help??

If you've discovered something amazing about Perl that you just need to share with everyone, this is the right place.

This section is also used for non-question discussions about Perl, and for any discussions that are not specifically programming related. For example, if you want to share or discuss opinions on hacker culture, the job market, or Perl 6 development, this is the place. (Note, however, that discussions about the PerlMonks web site belong in PerlMonks Discussion.)

Meditations is sometimes used as a sounding-board — a place to post initial drafts of perl tutorials, code modules, book reviews, articles, quizzes, etc. — so that the author can benefit from the collective insight of the monks before publishing the finished item to its proper place (be it Tutorials, Cool Uses for Perl, Reviews, or whatever). If you do this, it is generally considered appropriate to prefix your node title with "RFC:" (for "request for comments").

User Meditations
Time for an application portfolio
8 direct replies — Read more / Contribute
by talexb
on Jul 27, 2015 at 12:04

    I have been tinkering with a few tools lately, and now want to put up a portfolio of some web applications that I am working on. I have an account on pair Networks (they also host this site), so I set up local::lib and went ahead and tried to install Mojolicious::Lite, since that's the platform I'm working on these days.

    No dice -- Mojo::Lite requires 5.10, and pair only has 5.8.9. I checked with the other provider I use, and they have 5.8.8.

    So the two options I can see are a) install an up-to-date Perl on one of those accounts, or b) have these web applications run on my home machine (perhaps using http://www.easydns.com to provide consistent name resolution -- not sure is this is still available).

    I could go find another web provider, but that's additional expense, and not really my best option right now. Feedback welcome!

    Alex / talexb / Toronto

    Thanks PJ. We owe you so much. Groklaw -- RIP -- 2003 to 2013.

RFC: newscript.pl , my very first script!
6 direct replies — Read more / Contribute
by Darfoune
on Jul 24, 2015 at 15:40

    Hello monks,

    This is my very first post on this site! I wrote my first program, newscript.pl , a few times ago, in order to save some repetitive typing and I really want your opinions, critics, suggestions etc..

    It's job is very simple, create an empty script. At first written in bash, with only bash as supported language, now written in Perl it includes Perl, Bash and I started slowly working to include C as well. It's not very portable and surely not very efficient, but I'm using it everyday, when reading thru intermediate Perl, advanced Bash scripting and learning C the hard way.

    The script first ask user for the name of the new script, then it asks for the language you're going to use. It will then print the shebang line as well as wanted modules (if you asked for a Perl script) and save it to a file withyourname.test. It then fires up emacs -nw with your newscript so you are ready to input code right away.

    #!/usr/bin/perl use strict; use warnings; use File::Basename; ############################ # # Name : newscript # Usage: Makes ready to use script templates for # Bash and Perl only at the moment. It includes # the shebang line for both. Perl templates # includes some useful pragmas and the option to # include the required modules on the command line. # ############################ # declare some required vars my ($name, $language); my @modules = (); my $fullname = $0; my $progname = basename($fullname); ## main program: print "Name of the new script : "; chomp ($name = <STDIN>); print "Language of $name script: "; chomp ($language = <STDIN>); # If laguage is Bash, make a Bash script if ($language =~ /bash/i) { $name = "$name.test"; print "\nMaking a Bash script: $name\n"; _makebash(); # If language is Perl, make a Perl script } elsif ($language =~ /perl/i) { $name = "$name.pl.test"; print "\nMaking a Perl script: $name\n"; print "\nAdd modules? ex: File::Basename;\n(use strict and use war +ning are turned on by default).\n"; print "[yes/no]: "; # check if user wants modules chomp (my $addmodule = <STDIN>); if ($addmodule =~ /yes/i) { print "\nThis script does NOT add a ';' for you!\nSay 'done' w +hen you done..\nModules: "; while (<STDIN>) { last if ($_ =~ /done(;)?/i); push @modules, $_; } _makeperl(); } elsif ($addmodule =~ /no/i) { _makeperl();} else { print "I assume no.\n"; _makeperl(); } # If language is C, make a C program } elsif ($language =~ /c/i) { $name = "$name.test.c"; print "\nMaking a C program: $name\n"; print "\nThis is the first version with C included, no more option +s yet.\n"; print "Only '#include <stdio.h>' added at this time.\n\n"; _makec(); } else { print "This might help you:\n"; _usage(); } # Make a bash script sub _makebash { if ($language eq 'bash') { open (NEWSCRIPT, '>', $name); print NEWSCRIPT "#!/bin/bash\n\n"; close NEWSCRIPT; chmod 0700, "$name"; exec (`emacs -nw +3 $name`); } } # Make a perl script sub _makeperl { open (NEWSCRIPT, '>', $name); print NEWSCRIPT "#!/usr/bin/perl\n\nuse warnings;\nuse strict;\n"; if (defined($modules[0])){ # if module is defined +, include them to the template for my $mods (@modules) { print NEWSCRIPT "use $mods"; } } print NEWSCRIPT "\n"; close NEWSCRIPT; print "\n"; chmod 0700, "$name"; exec (`emacs -nw +50 $name`); } # Make a C program sub _makec { open (NEWPROG, '>', $name); print NEWPROG "#include <stdio.h>\n\n"; close NEWPROG; print "\n"; chmod 0700, "$name"; exec (`emacs -nw +10 $name`); } # Sets the usage message. sub _usage { print<<EOF; Usage: $progname [no options yet] Creates ready to use script templates. The script will first ask you for the name of your program, then the language in which you want it written. If your chosen language is supported, it will make an empty script, with your name and 'test' appended to it. The script then makes an exec call to emacs -nw with your new file. (not very portable yet ..) note: If your chosen language is Perl, The script will ask you if you wish to import more modules. If you do want more input them then followed by a ';' and input 'done' when finish. bash: #/bin/bash perl: #/usr/bin/perl use warnings; use strict; use [yourmods]; C: #include <stdio.h> EOF }
IBM Cloud Challenge.
3 direct replies — Read more / Contribute
by BrowserUk
on Jul 21, 2015 at 18:07

    I just read about an IBM programming challenge to try and entice developers to IBMs Bluemix Cloud development environment.

    (Don't bother if you're outside the UK; or if you want to use Perl (it ain't supported :(); or if stupid sign-up processes that don't work annoy you; or ... )

    What struck me was that the three programming tasks are, at least notionally, so trivial. It took me less than 5 minutes to write (working, but probably not best) solutions to all three.

    (Whether they would pass their test criteria I guess we'll probably never know)

    I was also struck by this part of the description:

    that you can put together a programme that can run within a time limit or on limited resources rather than just lashing together a hideous brute-force monstrosity. And that you can actually read the questions properly in the first place (a useful start, but one that's often forgotten).

    I think it would be interesting to see how the best Perlish solutions we can come up with compare with those other languages that get entered to the competition; when and if they are actually made public.

    So have at them. (Don't forget to add <spoiler></spoiler> tags around your attempts.)

    I'd post the questions here but I'm not sure it wouldn't be a problem copyright wise?

    I'll post my solutions here in a few days.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
    I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!
Beyond Agile: Subsidiarity as a Team and Software Design Principle
3 direct replies — Read more / Contribute
by einhverfr
on Jul 20, 2015 at 21:24

    This is a collection of thoughts I have been slowly putting together based on experience, watching the development (and often bad implementation of) agile coding practices. I am sure it will be a little controversial in the sense that some people may not see agile as something to move beyond and some may see my proposals as being agile.

    What's wrong with the Waterfall?

    I think any discussion of agile programming methodologies has to start with an understanding of what problems agile was intended to solve and this has to start with the waterfall model of development, where software has a slow, deliberate life cycle, where all design decisions are supposed to be nailed down before the code is started. Basically the waterfall approach is intended to apply civil engineering practices to software and while it can work with very experienced teams in some limited areas it runs into a few specific problems.

    The first is that while civil engineering projects tend to have well understood and articulated technical requirements, software projects often don't. And while cost of failure in dollars and lives for a civil engineering disaster can be high, with software it is usually only money (this does however imply that for some things, a waterfall approach is the correct one, a principle I have rarely seen argued against by experienced agile developers).

    The second is that business software requirements often shift over time in ways that bridges, skyscrapers, etc don't. You can't start building a 30 floor skyscraper and then have the requirements change so that it must be at least 100 floors high. Yet we routinely see this sort of thing done in the software world.

    Agile programming methodologies arose to address these problems. They are bounded concerns, not applicable to many kinds of software (for example software regulating dosage of radiotherapy would be more like a civil engineering project than like a business process tool), but the concerns do apply to a large portion of the software industry.

    How Agile is Misapplied

    Many times when companies try to implement agile programming, they run into a specific set of problems. These include unstructured code and unstructured teams. This is because too many people see agile methodologies as devaluing design and responsibility. Tests are expected to be documentation, documentation is often devalued or unmaintained, and so forth.

    Many experienced agile developers I have met in fact suggest that design is king, that it needs to be done right, in place, and so forth, but agile methodologies can be taken by management as devaluing documentation in favor of tests, and devaluing design in favor of functionality.

    There are many areas of any piece of software where stability is good, where pace of development should be slow, and where requirements are well understood and really shouldn't be subject to change. These areas cut against the traditional agile concerns and I think require a way of thinking about the problems from outside either the waterfall or agile methodologies.

    Subsidiarity as a Team Design Principle

    Subsidiarity is a political principle articulated a bit over a hundred years in a Papal encyclical. The idea is fairly well steeped in history and well applicable beyond the confines of Catholicism. The basic idea is that people have a right to accomplish things and that for a larger group to do what a smaller group can therefore constitutes a sort of moral theft. Micromanagement is therefore evil if subsidiarity is good but also it means that teams should be as small as possible, but no smaller.

    In terms of team design, small teams are preferable to big teams and the key question is what a given small team can reasonably accomplish. Larger groupings of small teams can then coordinate on interfaces, etc, and sound, stable technological design can come out of this interoperation. The small team is otherwise tasked with doing everything -- design, testing, documentation. Design and testing are as integrated with software development as they are in agile, but as important as they are in the waterfall approach.

    This sort of organization is as far from the old waterfall as agile is but it shares a number of characteristics with both. Design is emphasized, as is documentation (because documentation is what coordinates the teams). Stability in areas that need it is valued, but in other areas it is not. Stable contracts develop where these are important and both together and individually, accomplishments are attained.

    Subsidiarity as a Technical Design Principle

    Subsidiarity in team design has to follow what folks can accomplish but this also is going to mean that these follow technological lines as well. Team responsibility has to be well aligned with technological responsibility. I.e. a team is responsible for components and components are responsible for functionality.

    The teams can be thought of as providing distinct pieces of software for internal clients, taking on responsibility to do it right, and to provide something maintainable down the road. Once a piece of software is good and stable, they can continue to maintain it while moving on to another piece.

    Teams that manage well defined technical and stable components can then largely end up maintaining a much larger number of such components than teams which manage pieces that must evolve with the business. Those latter teams can move faster because they have stability in important areas of their prerequisites.

    But the goal is largely autonomous small teams with encapsulated responsibilities, producing software that follows that process.

Advanced techniques with regex quantifiers
5 direct replies — Read more / Contribute
by smls
on Jul 19, 2015 at 05:29
RFC: Net::SNTP::Server v1
2 direct replies — Read more / Contribute
by thanos1983
on Jul 17, 2015 at 10:45

    Dear Monks,

    Once again I need your expertise opinion as I am about to upload my second module related with Net::SNTP::Client.

    The module serves purpose of a simple SNTP server able to reply back to client requests based RFC4330 message format.

    In order to test the code I will provide both the client and server scripts so you can easily test and observe bugs or possible improvements based on your experience.

    Update: changing value(s) from "0" to "1" -RFC4330 => "1" and -clearScreen => "1" on the client code.

    Update 2: Modifying POD based on new findings, also the function $verify_port and last adding a new error message/restriction in checking $moduleInput{-port}.

    Update 3: Modifying server module based on comments from Monk::Thomas.

    Update 4: Modifying server module based on comments from Anonymous Monk.

    Client script:

    Server script:

    Server module:

    Thank you all in advance for your time and effort reviewing my work.

    Seeking for Perl wisdom...on the process of learning...not there...yet!
Recamán's sequence and memory usage
3 direct replies — Read more / Contribute
by Athanasius
on Jul 13, 2015 at 04:41

    Esteemed Monks,

    I was looking at The On-Line Encyclopedia of Integer Sequences (OEIS) (you know, as one does), and on the Welcome page I found a list of Some Famous Sequences, of which the first is Recamán’s sequence, defined as follows:

    R(0) = 0; for n > 0, R(n) = R(n-1) - n if positive and not already in the sequence, oth +erwise R(n) = R(n-1) + n.

    What makes this sequence interesting is N. J. A. Sloane’s conjecture that every number eventually appears.

    Coding the sequence is simplicity itself; the challenge is to test Sloane’s conjecture by keeping track of the numbers that have not yet appeared in the series. My initial, naïve approach was to use a sieve, à la Eratosthenes:

    But this turned out to be far too memory-hungry: for values of MAX of the order of twenty million, RAM usage on my 3GB system approaches 100%, thrashing sets in, and the script (along with the rest of Windows) grinds to a shuddering halt.

    Surely, I thought, there must be a memory-efficient way to represent a sieve? And of course there is, and of course it was already implemented on CPAN. A little searching led to the Set::IntSpan module which stores runs of consecutive integers as spans, allowing large (even infinite) collections of integers to be represented very economically.

    Calculation of successive terms in the Recamán sequence is noticeably slower using Set::IntSpan for lookup than it is using a hash. But, as the adage says, it’s better to be late than be dead on time. (This was the slogan of an Australian safe driving ad campaign some years ago.) For the record: I also looked at Set::IntSpan::Fast and Set::IntSpan::Fast::XS. The latter failed to install on my system, and the former actually ran slower than Set::IntSpan for this use-case.

    Turns out that Set::IntSpan not only solves the memory problem, it also makes it possible to dispense with an upper bound for the sieve. How, then, to display progressive results? Well, the OEIS has a couple of additional series related to Recamán’s:

    • A064228: values of R(n) that take a record number of steps to appear: 1, 2, 4, 19, ...
    • A064227: the values of n corresponding to the values in A064228: 1, 4, 131, 99734, ...

    So I recast the script to output successive values of these two series:

    14:20 >perl recaman.pl 1 <-- 1 2 <-- 4 4 <-- 131 19 <-- 99734 ...

    Here is the new script:

    use strict; use warnings; use sigtrap handler => \&int_handler, 'INT', handler => \&break_handler, 'BREAK'; use Set::IntSpan; use Time::HiRes qw(gettimeofday tv_interval); $| = 1; my $t0 = [gettimeofday]; my $min0 = 1; my $n = 0; my $r0 = 0; my $missing = Set::IntSpan->new( '1-)' ); print "$min0 <-- "; while (++$n) { my $r = $r0 - $n; $r = $r0 + $n if $r < 0 || !$missing->member($r); $missing->remove($r); if ((my $min1 = $missing->min) > $min0) { print "$n\n$min1 <-- "; $min0 = $min1; } $r0 = $r; } sub int_handler { printf "\nn = %d, elapsed time: %.1fs\n", $n, tv_interval($t0); } sub break_handler { int_handler(); exit 0; }

    This script was developed under Windows 8.1, 64-bit, using Strawberry Perl:

    14:20 >perl -v This is perl 5, version 22, subversion 0 (v5.22.0) built for MSWin32-x +64-multi-thread

    The two signal handlers allow the script to be interrupted as follows:

    • Control-C causes the script to display the current value of $n and the total running time of the script so far.
    • Control-Break causes the script to display the same information and then exit.

    My takeaways from this meditation?

    First, we all know that micro-optimisation is pointless until you have first selected the best algorithm(s). But optimising an algorithm may actually consist in optimising its underlying data structures. Obvious? Yes, but still worth a reminder now and then.

    Second, CPAN is awesome! But you knew that already. :-)

    Cheers,

    Athanasius <°(((><contra mundum Iustus alius egestas vitae, eros Piratica,

tied hash for data munging
3 direct replies — Read more / Contribute
by shmem
on Jul 11, 2015 at 08:21

    This meditation is about a tied hash package, and how it came into existence. I am still meditating whether this is too obscure, or whether its goal is better achieved using some other technique; is it worth being uploaded to CPAN as yet another strange perl delirium? is the name ok? Any suggestions, review, critics are welcome. Thanks for your time.

    Itch

    Over 100 poorly performing scripts written in some BASIC dialect for exactly the same purpose (read source records, transform them, write target records), sporting hardcoded parameters and different output assembling code, proliferating with each new customer (copy over, twiddle, tweak).

    Scratch

    Perl to the rescue to do the data gathering and munging, and write a unified import CSV to be fed into that dratted basic script - one for all. Parameters and data transforming procedures should be kept separate, in a format editable by non-perlers. I choose INI file style, which fitted both that BASIC dialect and perl:

    SCALAR=Some value LIST=Foo,Bar,Baz HASH=Foo:Bar,Baz:Quux

    Straight forward. What about the data transforming rules? Since these are concatenations of values from the input record - conveniently present as a hash - and the output of some functions munging those values, these are something that could easily be transformed into subroutines:

    DOMAIN=@example.com USER=sAMAccountName.DOMAIN PASS=md5sum(PRE.sAMAccountName.POST,15) PRE=a874f4u POST=ea748tyoal MAIL=join(DOT,givenName,sn).DOMAIN

    md5sum is a function using Digest::MD5::md5_hex.

    With a subroutine generating subroutine using a regex, the above values are converted into the following hash:

    %h = ( DOMAIN => '@example.com', USER => sub { $r->{sAMAcountName} . $c->{DOMAIN} }, PASS => sub { md5sum($c->{PRE} . $r->{sAMAcountName} . $c->{POST +}, 15) }, MAIL => sub { join('.', $r->{givenName}, $r->{sn}).$c->{DOMAIN} +}, );

    where $r is the current record, and $c is the hash representing the INI files content.

    So far, so good. Retrieving a hash value consisting in a subroutine is done with

    $out{$key} = $h{$key}->();

    but that dies if the value slot holds a scalar. Yes, I could iterate over the keys using ref or such, but I would rather want to say

    %out = %h;

    and have a magic hash %h which encapsulates all that logic and "knows" what to deliver.

    Relief

    A tied hash. update: - which is a subroutine factory.

    <update>
    ...due to BrowserUk's immediate reaction below: this is overkill to just execute CODE in the value slots of a hash, of course. Well, the itch was the starting point. The code generation and closure bits (see EXAMPLE in the pod) tell what it might be useful for: currying, building dispatch tables, ... - I have to play more with this, yet.
    </update>

    perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'
Tutorial RFC: Guide to Perl references, Part 1
5 direct replies — Read more / Contribute
by stevieb
on Jul 07, 2015 at 16:45

    NOTE: I've been considering translating this five part series I wrote a few years ago to PerlMonks, and I thought I'd go for it to see what others thought. Parts 2-5 are still linked to the blog to give context to the entire series. If people think this should be made into a Tutorial, I'll clean up and translate the rest of the docs and link them properly.

    Thoughts?

    BEGIN TUTORIAL...

    Understanding references and their subtleties in Perl is one of the more difficult concepts to fully wrap one's head around. However, once they are fully understood by the blossoming developer, they find a whole new level of capability and power to exploit and explore.

    I often see newer programmers struggle with the concept of references on the Perl help sites I frequent. Some still have a ways to go, but many are at the stage where perhaps one more tutorial may push them over the edge and give them that 'Ahhhh' moment of clarity. My moment of clarity came when I read Randal Schwartz's "Learning Perl Objects, References & Modules" book for the something like the 8th time. Although once the concept of references is understood, the syntax and use cases can still be confusing for quite some time, especially in Perl, because There Is More Than One Way To Do It.

    This tutorial is the first in a five part series. This part will focus on the basics, preparing you for more complex uses in the following four parts. I've created a cheat sheet that summarizes what you'll learn in this document.

    • Part 1 - The basics (this document)
    • Part 2 - References as subroutine parameters
    • Part 3 - Nested data structures
    • Part 4 - Code references
    • Part 5 - Concepts put to use

    I will stick with a single consistent syntax throughout the series and will refrain from using one-line shortcuts and other simplification techniques in loops and other structures in hopes to keep any confusion to a minimum. Part one assumes that you have a very good understanding of the Perl variable types, when they are needed, and how they are used. Some exposure to references may also prove helpful, but shouldn't be required.

    THE BASICS

    References in Perl are nothing more than a scalar variable that instead of containing a usable value, they 'point' to a different variable. When you perform an action on a reference, you are actually performing the action on the variable that the reference points to. A Perl reference is similar to a shortcut to a file or program on your computer. When you double click the shortcut, the shortcut doesn't open, it's the file that the shortcut points to that does.

    We'll start with arrays, and I'll get right into the code.

    We'll define an array as normal, and then print out its contents.

    my @array = ( 1, 2, 3 ); for my $elem ( @array ){ say $elem; }

    Prepending the array with a backslash is how we take a reference to the array and assign the reference to a scalar. The scalar $aref now is a reference that points to @array.

    my $aref = \@array;

    At this point, if you tried to print out the contents of $aref, you would get the location of the array being pointed to. You know you have a reference if you ever try to print a scalar and you get output like the following:

    ARRAY(0x9bfa8c8)

    Before we can use the array the reference points to, we must dereference the reference. To gain access to the array and use it as normal, we use the array dereference operator @{}. Put the array reference inside of the dereference braces and we can use the reference just as if it was the array itself:

    for my $elem ( @{ $aref } ){ say $elem; }

    The standard way of assigning an individual array element to a scalar:

    my $x = $array[0];

    To access individual elements of the array through the reference, we use a different dereference operator:

    my $y = $aref->[1];

    Assign a string to the second element of the array in traditional fashion:

    $array[1] = "assigning to array element 2";

    To do the same thing through an array reference, we dereference it the same way we did when we were taking an element from the array through the reference:

    $aref->[1] = "assigning to array element 2";

    You just learnt how take a reference to an array (by prepending the array with a backslash), how to dereference the entire array reference by inserting the reference within the dereference block @{}, and how to dereference individual elements of the array through the reference with the -> dereference operator. That is all there is to it. Hashes are extremely similar. Let's look at them now.

    Create and initialize a normal hash, and iterate over its contents:

    my %hash = ( a => 1, b => 2, c => 3 ); while ( my ( $key, $value ) = each %hash ){ say "key: $key, value: $value"; }

    Take a reference to the hash, and assign it to a scalar variable:

    my $href = \%hash;

    Now we'll iterate over the hash through the reference. To access the hash, we must dereference it just like we did the array reference above. The dereference operator for a hash reference is %{}. Again, just wrap the reference within its dereferencing block:

    while ( my ( $key, $value ) = each %{ $href } ){ say "key: $key, value: $value"; }

    Access an individual hash value:

    my $x = $hash{ a };

    Access an individual hash value through the reference. The dereference operator for accessing individual elements of a hash through a reference is the same one we used for an array (->).

    my $y = $href->{ a };

    Assign a value to hash key 'a':

    $hash{ a } = "assigning to hash key a";

    Assign a value to hash key 'a' through the reference:

    $href->{ a } = "assigning to hash key a";

    That's essentially the basics of taking a reference to something, and then dereferencing the reference to access the data it points to.

    When we operate on a reference, we are essentially operating on the item being pointed to directly. Here is an example that shows, in action, how operating directly on the item has the same effect as operating on the item through the reference.

    my @b = ( 1, 2, 3 ); my $aref = \@b; # assign a new value to $b[0] through the reference $aref->[0] = 99; # print the array for my $elem ( @b ){ say $elem; }

    Output:

    99 2 3

    As you can see, the following two lines are equivalent:

    $b[0] = 99; $aref->[0] = 99;

    CHEAT SHEET

    Here's a little cheat sheet for review before we move on to the next part in the series.

    my @a = ( 1, 2, 3 ); my %h = ( a => 1, b => 2, c => 3 ); # take a reference to the array my $aref = \@a; # take a reference to the hash my $href = \%h; # access the entire array through its reference my $elem_count = scalar @{ $aref }; # access the entire hash through its reference my $keys_count = keys %{ $href }; # get a single element through the array reference my $element = $a->[0]; # get a single value through the hash reference my $value = $h->{ a }; # assign to a single array element through its reference $a->[0] = 1; # assign a value to a single hash key through its ref $h->{ a } = 1;

    This concludes Part 1 of our Guide to Perl references. My goal was not to compete with all the other reference guides available, but instead to complement them, with the hope that perhaps I may have said something in such a way that it helps further even one person's understanding. Next episode, we'll learn about using references as subroutine parameters.

The problem of documenting complex modules.
11 direct replies — Read more / Contribute
by BrowserUk
on Jul 05, 2015 at 04:41

    This is meditation; but I also hope that it might start a discussion that will come up with (an) answers to what I see as an ongoing and prevalent problem.

    This has been triggered at this time by my experience of trying to wrap my brain around a particular complex module; but I don't want to get into discussion particular to that module, so I won't be naming it.

    Suffice to say that CPAN is replete with modules that are technically brilliant and very powerful solutions to the problems they address; and that deserve far wider usage than they get.

    In many cases the problem is not that they lack documentation -- often quite the opposite -- but more that they don't have a simple in; a clearly defined and obvious starting point that gives a universal starting point on which the new user can build.

    And example of (IMO) good documentation is Parallel::ForkManager. It's synopsis (I've tweaked it slightly to remove a piece of unnecessary fluff):

    use Parallel::ForkManager; my $pm = Parallel::ForkManager->new($MAX_PROCESSES); foreach my $data (@all_data) { # Forks and returns the pid for the child: my $pid = $pm->start and next; ... do some work with $data in the child process ... $pm->finish; # Terminates the child process }

    is sufficient to allow almost anyone needing to use it, for almost any purpose, to put together a reasonable working prototype in a dozen lines of code without reading further into the documentation.

    It allows the programmer to get started and move forward almost immediately on solving his problem -- which isn't "How to use P::FM" -- and only refer back to and utilise the more sophisticated elements of P::FM, as and when he encounters the limitations of that simple starting point.

    As such, the module is successful in hiding the nitty-gritty details of using fork correctly; whilst imposing the minimum of either up-front learning curve or infrastructural boiler-plate upon the programmer; who has other more important (to him) things on his mind.

    Contrast that with something like POE which requires a month of reading through the synopsis of the 800+ modules in that namespace POE::*, and then another month of planning, before the new user could put together his first line of code. As powerful as that module, suite of modules; dynasty of modules is, unless you have the author's help, and lots of time, getting started is an extremely daunting process. In that respect (alone perhaps), POE fails to enable a 'simple in'.

    And before anyone says that it is unfair to compare those two modules -- which maybe true -- the purpose was to pick extremes to make a point; not to promote or denigrate either.

    Another module that I know I should have made much more use of in the type of code I frequently find myself writing, is PDL. I've tried at least a dozen times to use PDL as a part of one of my programs; and (almost) every time I've abandoned the attempt before ever writing a single line of PDL, because I get frustrated by the total lack of a clear entry point in to the surfeit of documentation.

    There's the FAQ, and the Core; and the Index; and the QuickStart; and the Doc; and the Basic; and the Lite; and the Course; and the Philosophy; and the pdldoc; and the Tips; and ... I'm outta here. I'm trying to write my program, which does a little math on some biggish datasets that would benefit from being vectored, but life's too short...

    Again; the underlying code is brilliant (I am assured), and it isn't a case of a lack of documentation; just a mindset that says: "this is PDL in all its glory, power and nuance. bathe yourself in its wonderfulness and wallow in its depth". Oh, and then when you've immersed yourself in its glory, understood its philosophy, and acclimated its nuance, then you can get back to working out how to use it to solve your problem.

    And that's a real shame; and a waste.

    I'm not sure what the solution is. I do know that the modules I use most List::Util, Data::Dump, threads etc. I have rarely ever had to look at the documentation; their functionality has (for me) become an almost invisible extension of Perl itself, and only the occasional (perhaps you forgot to load "sum"?) reminds me that they aren't.

    Of course, what they do is essentially pretty simple; but that in itself is a perhaps a clue.

    I do know that (for me) the single most important thing in encouraging my use of a module is being able to C&P the synopsis into my existing program, tweak the variable names, and have it do something useful immediately.

    In part, that comes down to a well designed API; in part, to well-chosen defaults; and part having a well-chosen, well-written synopsis that addresses the common case; with variable names and structure that make it obvious how to adapt that synopsis for the common case. Once I have something that compiles and runs -- even if it doesn't do exactly what I need it to do; or even what I thought it would do from first reading -- it gives me a starting point and something to build on. And that encourages me to persist. To read the documentation on a as-I-need-to basis to solve particular problems as I encounter them.

    Over a decade ago, I posted My number 1 tip for developers.; and this is the other side of that same philosophy. Start simple and build.

    And that I think has to be the correct approach to documenting complex modules. They need to:

    1. Offer a single, obvious, starting point. The in.
    2. That needs to be very light on history, philosophy, jargon, technical and social commentary and background. And choice.
    3. It needs to offer a single, simple, well-chosen, starting point, that requires minimal reading to adapt to the users code, for the common case.
    4. It then needs to offer them a quick, clear, simple path to solving their problem.

    What it must not do:

    • It mustn't present them with 'a bloody great big list of entrypoints/methods'.
    • It mustn't offer them a myriad of choices and configuration options.
    • It mustn't take them on a deep immersion in the details of either algorithms or implementation.
    • It mustn't present them with either "Ain't this amazing" nor "Ain't I clever" advert.
    • It mustn't waste their time with details of your personal preferences, prejudices, philosophies and theologies.

    If you want programmers to use your modules, you need to tell them what (the minimum) they *NEED TO KNOW* to get started. And then give a clear index to the variations, configurations and extensions to that basic starting point.

    Achieve that, give them their 'in', with the minimum of words, fuss or choice, and they'll come back for all the rest as they need it.


    This is ill-thought through and incomplete, so what (beyond risking offending half the authors on CPAN) am I trying to achieve with this meditation?

    I'd like to hear if you agree with me? Or how you differ. What you look for in module documentation. Examples that you find particularly good; or bad.

    It'd be nice to be able to derive from the thread, a set of consensus guidelines to documenting moderate to complex modules -- that almost certainly won't happen -- but if we managed to get a good cross section of opinions on what makes for good and bad documentation; and a variety of opinions of the right way to go about it; it might provide a starting point for people needing to do this in the future.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
    I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!
Writing multiple Excel::Writer::XLSX worksheets in parallel (3rd and final attempts)
4 direct replies — Read more / Contribute
by marioroy
on Jul 04, 2015 at 23:16

    July 22, 2015. The example was updated to work with MCE in trunk.

    My 1st and 2nd attempts got me warmed up and thought faster is possible. The following demo is my 3rd attempt and writes 1 million cells combined in less than 6 seconds from start to finish and 57 seconds for 10 million cells. Running serially takes 15 and 141 seconds for 1 and 10 million cells respectively. Processors have turbo boost for some time. Thus, serial code is likely to run at a faster GHz.

    for ( 1 .. 111_111 ) { ... } # 3 * 3, 1 million for ( 1 .. 1_111_111 ) { ... } # 3 * 3, 10 million

    Writing text data will slow this down a little due to obtaining the next unique id from the shared strTable object. The internal str_table is shared between worksheets in Excel::Writer::XLSX. Thus, synchronization is necessary as well.

    Note: This requires MCE from trunk r957 or later which includes MCE::Shared as MCE 1.700 is not yet released. The logic consumes only the memory necessary. There is never duplicate data from running multiple workers.

    #!/usr/bin/env perl use strict; use warnings; # --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- package StrTable; sub new { my ($class, $self) = ( shift, { table => {}, unique => 0 } ); bless $self, $class; } sub table { $_[0]->{table } } sub unique { $_[0]->{unique} } sub value { if (exists $_[0]->{table}->{ $_[1] }) { $_[0]->{table}->{ $_[1] }; } else { $_[0]->{table}->{ $_[1] } = $_[0]->{unique}++; } } # --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- package main; use Archive::Zip (); use File::Copy qw(move); use File::Find (); use File::Temp (); $File::Temp::KEEP_ALL = 1; use Excel::Writer::XLSX; use MCE::Signal qw($tmp_dir); use MCE::Loop 1.699; use MCE::Shared; my $nodeList = [ [ 'AMS' , 'a' ], [ 'APJ' , 'ap' ], [ 'EMEA', 'e' ] ]; my $strTable = mce_share( new StrTable ); my ($center, $format); { # Override _get_shared_string_index to synchronize str_table update +s no warnings 'redefine'; sub Excel::Writer::XLSX::Worksheet::_get_shared_string_index { my ($self, $str) = (shift, shift); if ( not exists ${ $self->{_str_cache} }->{$str} ) { ${ $self->{_str_cache} }->{$str} = $strTable->value($str); } else { ${ $self->{_str_cache} }->{$str}; } } } sub init_wb { my ($wn, $file) = (shift, shift); # Increment $wn by 1 since worksheet xml files begin at 1 $wn++; mkdir "$tmp_dir/$wn"; my $wb = Excel::Writer::XLSX->new($file || "$tmp_dir/$wn/tmp.xlsx") +; $wb->set_tempdir("$tmp_dir/$wn"); # Set workbook properties $wb->set_properties( title => 'Node List', author => 'L_WC demo', comments => 'Node List', ); # Define/add formats to the workbook $center = $wb->add_format(align => 'center'); $format = $wb->add_format(align => 'center', bg_color => 44); # Add worksheets, specify formats for columns/rows for (0 .. @{ $nodeList } - 1) { $wb->add_worksheet($nodeList->[$_][0]); $wb->sheets($_)->set_column(0, 4, 15, $center); } return $wb; } sub close_wb { my $wb = shift; MCE->sync(); # Wait for others to complete, important $wb->{_str_table } = $strTable->table(); # Replace str_table $wb->{_str_total } = 0+$strTable->unique(); # Update str_total $wb->{_str_unique} = 0+$strTable->unique(); # Update str_unique $wb->close(); # Close workbook } sub merge_wb_data { my $wb_file = shift; my ($zip, @pths, @xlsx_files) = (Archive::Zip->new()); local ($@, $!, $^E, $?); # Other files, e.g. table data likely need the same and not done # for this demonstration. Just worksheet files are merged. # I received help by reading _store_workbook inside # Excel::Writer::XLSX::Workbook.pm. # Find worksheet files 2,3,... for my $_num (1 .. @{ $nodeList }) { my $wanted = sub { push @pths, $1 if $File::Find::name =~ /(.*)\/sheet$_num\.xml +/; }; File::Find::find({ wanted => $wanted, untaint => 1, untaint_pattern => qr|^(.+)$ +| }, "$tmp_dir/$_num"); } # Move worksheet files 2,3,... to where worksheet 1 data resides for (0 .. @pths - 1) { unlink $pths[$_]."/../../../tmp.xlsx"; if ($_ > 0) { my $_num = $_ + 1; unlink $pths[0]."/sheet$_num.xml"; move $pths[$_]."/sheet$_num.xml", $pths[0]."/sheet$_num.xml"; } } # Re-zip xlsx files my $wanted = sub { push @xlsx_files, $File::Find::name if -f }; my $temp_dir = $pths[0]."/../../"; my $short_name; File::Find::find({ wanted => $wanted, untaint => 1, untaint_pattern => qr|^(.+)$| }, $temp_dir); for my $file_name (@xlsx_files) { $short_name = $file_name; $short_name =~ s{^\Q$temp_dir\E/?}{}; $zip->addFile($file_name, $short_name); } open my $fh, '>', $wb_file or die "Error opening xlsx file: $!\n"; binmode $fh; $zip->writeToFileHandle($fh); close $fh; } # --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- MCE::Loop::init( max_workers => scalar(@{ $nodeList }), chunk_size => 1, posix_exit => 1, use_threads => 0, ); mce_loop { my ($region, $sql) = ($_->[0], $_->[1]); my ($wb, $ws); # Acquire data from the DB. Each worker must obtain a handle. # The DB logic is similar to running serially. Just the where # clause is likely unique for each region. # Fill worksheet rows/cells if ($region eq 'AMS') { $wb = init_wb(0); $ws = $wb->sheets(0); $ws->write(0, 2, 'foo', $format); for ( 1 .. 111_111 ) { $ws->write(0 + $_, 0, 1000 + $_); $ws->write(1 + $_, 2, 2000 + $_); $ws->write(2 + $_, 4, 3000 + $_); } print "AMS ---- DONE.\n"; } elsif ($region eq 'APJ') { $wb = init_wb(1); $ws = $wb->sheets(1); $ws->write(0, 2, 'bar', $format); for ( 1 .. 111_111 ) { $ws->write(0 + $_, 0, 4000 + $_); $ws->write(1 + $_, 2, 5000 + $_); $ws->write(2 + $_, 4, 6000 + $_); } print "APJ ---- DONE.\n"; } elsif ($region eq 'EMEA') { $wb = init_wb(2); $ws = $wb->sheets(2); $ws->write(0, 2, 'baz', $format); for ( 1 .. 111_111 ) { $ws->write(0 + $_, 0, 7000 + $_); $ws->write(1 + $_, 2, 8000 + $_); $ws->write(2 + $_, 4, 9000 + $_); } print "EMEA ---- DONE.\n"; } close_wb($wb) if $wb; } $nodeList; # Shutdown MCE MCE::Loop::finish(); # Merge data into one workbook merge_wb_data('Node_List.xlsx'); print "Node List is Done.\n";

    Kind regards, Mario

RFC: Net::SNTP::Client v1
5 direct replies — Read more / Contribute
by thanos1983
on Jun 30, 2015 at 12:45

    Hello Everyone,

    About a year ago I started with the idea of creating a Perl module based on Net::NTP. The module that I am thinking to create would be named (Net::SNTP::Client). The difference between those two is the precision, from my point of view the Net::NTP module does not get correct millisecond/nanosecond precision. The module is based on RFC4330, where according to the RFC different precision will achieved on LinuxOS and WindowsOS.

    In theory the module should be compatible with all OS (WindowsOS, LinuxOS and MacOS) please verify that with me since I only have LinuxOS.

    I am planning to create also another module Net::SNTP::Server which is approximately an SNTP server and when I say approximately is because I can not figure it out how to replicate the server side. But any way first thing first.

    Is it possible to take a look and assist me in possible improvements and comments. Since this is my first module I have no experience so maybe the module is not well written.

    The execution of the script is very simple, create a script e.g. client.pl and put the code bellow.

    client.pl

    I have inserted four options:

    -hostname => NTP Hostname or NTP IP -port => 123 Default or Users choice e.g. 5000 -RFC4330 => 1 -clearScreen => 1

    The first option is to get an RFC4330 printout way, and the second option is to clear the screen before the printout. I think both options will be useful on the printout of the script.

    I have chosen to paste the module in the folder path "/home/username/Desktop/SNTP_Module/Net/SNTP/Client.pl". Remember for testing purposes to change the path on client.pl accordingly on the location that you will place the module.

    Update 1: Removing (EXPORT_OK, EXPORT_TAGS, shebang line) based on toolic comments.

    Update 2: Removing $frac2bin unused sub.

    Update 3: Adding some checks on the input of getSNTPTime sub

    Update 4: Adding Plain Old Documentation format and updating code based on Monk::Thomas comments.

    Update 5: Updating code based on Monk::Thomas new comments.

    Update 6: Updating code, with new updated Plain Old Documentation.

    Net::SNTP::Client.pm

    Thank you for your time and effort reading and replying to my question/review.

    Seeking for Perl wisdom...on the process of learning...not there...yet!
Software Projects In Real Life: "I See Dead People"
8 direct replies — Read more / Contribute
by sundialsvc4
on Jun 23, 2015 at 08:44

    The title of this Meditation comes, of course, from the punch-line of a really bad movie with a classic O’Henry Ending:

    But in many ways, it also sums up my career.   (Polite pause as the twitter of laughter dies down.)   For most of the past 25 years or so, I’ve been involved in projects.   Generally, not ones that I had started.   Generally, not healthy ones.   Dead ones, or very nearly so.   My task was to try to “turn them around,” and I generally did.   Whether or not my attempts at resuscitation were actually long-term successful, this experience did teach me a lot of the reasons ... and they are human reasons ... why software projects so often go so badly wrong.   I’m not going to do any preaching here, although it may seem so.   I’m just relating some of my personal experiences in a mortuary project triage.   (FYI:   Teams-in-place were anywhere from one to fifteen people, most of whom had “split the coop.”)

    First of all, these projects typically started out with “a great deal of enthusiasm, but no real plan.”   The usual justification was that the project needed to “hurry up to market,” or that the stakeholders in the project “would know it when they saw it” and the managers of the project (if there were any ...) simply gave-up trying to ask them to make up their minds.

    And, of course, in several cases, those stakeholders were assured that they didn’t have to make up their minds.   “Self-directed teams,” the programmers purred self-confidently, “would produce a ‘potentially viable product(!)’ every two weeks!

    “SOP = SOTP.™”   Standard Operating Procedure = Seat Of The Pants.

    And yet, what happened ... what inevitably happened ... is that everything in the “software mechamism” turned out to be inextricably coupled to everything else.   As layers of code were piled on, and as changes pinged-and-ponged throughout all those layers, the whole thing fell down in a heap as the programmers sailed on to the next green pasture.

    Many software projects are actually the work of one Guy.   (Sorry, ladies ...)   That “one guy” might be surrounded by several other people, but this is simply an attempt to scale-up the only modus operandi that this One Guy actually knows:   himself.   The project “feels its way along” because that’s how he’s used to doing it.   (And, because he is a crackerjack programmer, is used to eventually succeeding producing something.)   There simply isn’t any experience in being part of a successfully managed project:   most programmers, I candidly suspect, haven’t actually seen one.   (And there were no Angelic Choirs that started singing when I showed up either, I’m afraid ... no self-sunshine here.)

    The underlying reason for these problems, I think, is:   a very natural human reaction to what is a virtually-unmanageable technical situation.   The objective of the project is to build a self-directing machine ... and to do it perfectly, because nothing less than perfection will do.   Viewed as a mechanism, software would be said to have “unlimited degrees-of-freedom.”   i.e. “Anything is connected to everything else.”   Although the instant-to-instant flow of control within the software is of course described by if/then/else and looping constructs, the actual mechanism is also determined by its internal and external state.   This concern for “state” is what causes the coupling.   (And it’s also one of the reasons why “Functional Programming” is such a hot research topic.)

    My biggest criticism of Scrum, and Agile, and XP, and, well, most “methodologies,” is that they ignore this aspect.   They focus, instead, upon the organization and the daily work-activities of the team.   They discuss things like “user stories,” which are simply one possible way of trying to express one’s ideas and plans to a customer, but then omit from consideration exactly how that “story” is to become if/then/else, and how that new web of decision-logic is to be tested, and how it both affects and is affected by (“is infinitely coupled to ...”) everything else.   As a paradigm, useful in one sense though it may be, it does not and probably cannot (IMHO) go far enough.

    “We are building a self-directing machine.”   That, quite frankly, is the light-bulb moment that I got from the Managing the Mechanism e-book.   It’s something that we can say to business stakeholders, except that it is extremely likely to scare them off.   It certainly does, I think, cast some useful insights on what we might be missing in our present-day methodologies.   We certainly do need better processes for our work, better ways to describe them, and better ways to inform stakeholders of exactly what we need from them and why.

    In closing, one of the most prevalent things that I have seen, in every project that I have tried to turn-around, is disillusionment.   On both sides of the aisle.   Long before the software had broken down, communication had also broken down, and so had business process (if it ever truly existed).   No one builds houses and bridges that way.   (For very obvious, flammable and heavy reasons, no one is allowed to ...)   I suspect that the seeds of project failure are sown almost as soon as the first plow-blade cuts the soil.   This is our problem, as a profession, and we need a better solution to it.   Perhaps a different viewpoint is a start.

    That’s my Meditation.   Borne, as I said, from a most-interesting career path that has not always been a happy one.   What do you think?   What have your experiences been?   For instance, have you worked-through a spectacular success story from one of these other strategies?   I’d love to hear it . . .   The water in the cooler is ice-cold and there’s beer in the fridge that’s even colder.   May the discussions begin?

Nobody Expects the Agile Imposition (Part IX): Culture
3 direct replies — Read more / Contribute
by eyepopslikeamosquito
on Jun 20, 2015 at 03:40

    In 2008, we were pretty much a Scrum company ... however, a few years later, we had grown into a bunch of teams and we found that some of the standard Scrum practices were actually getting in the way. Rules are a good start, then break them when needed. We decided that Agile matters more than Scrum.

    Autonomy means the squad decides what to build, how to build it, and how to work together while doing it. One consequence of autonomy is that we have very little standardization. When people ask things like which code editor do you use or how do you plan, the answer is mostly depends on which squad. Some do Scrum sprints, others do Kanban ... it's really up to each squad. Instead of formal standards we have a strong culture of cross-pollination; when enough squads use a specific practice or tool, such as git, that becomes the path of least resistance, and other squads tend to pick the same tool.

    Why is autonomy so important? Well, because it's motivating and motivated people build better stuff.

    -- from Spotify Engineering Culture (Part 1) by Henrik Kniberg (0:30-4:40)

    Instead of blindly following Scrum dogma, I advise you to analyse the problems you and your company face daily. Reason about them. Consider applying Agile and Lean principles to them. Experiment to see what works for you and what doesn't.

    After feeling isolated and alone in resisting the Scrum imposition, reading Spotify's story has cheered me up. I've become more hopeful that things will change, that over time more folks will come to see the benefits of greater team autonomy.

    Autonomy and Alignment

    It's kind of like a jazz band, although each musician is autonomous and plays his own instrument, they listen to each other and focus on the whole song together. That's how great music is created. So our goal is loosely coupled but tightly aligned squads.

    Down here is low alignment and low autonomy, a micromanagement culture, no high level purpose, just shut up and follow orders. High alignment and high autonomy means leaders focus on what problem to solve but let the teams figure out how to solve it. Alignment enables autonomy.

    -- from Spotify Engineering Culture (Part 1) by Henrik Kniberg (3:20-3:50)

    For high autonomy to work, you need high alignment. With low alignment, teams simply do whatever they want, with each team going off in a different direction.

    Specialists vs Generalists

    Each system is owned by one squad. But we have an internal open source model and our culture is more about sharing than owning. Suppose squad one here needs something done in system B and squad two knows that code best, they'll typically ask squad two to do it. However, if squad two doesn't have time, then squad one doesn't necessarily need to wait. Instead they're welcome to go ahead and edit the code themselves and then ask squad two to review the changes. So anyone can edit code but we have a culture of peer code review. This improves quality and spreads knowledge. Over time we've evolved design guidelines, code standards, and other things to reduce engineering friction, but only when badly needed.

    -- from Spotify Engineering Culture (Part 1) by Henrik Kniberg (5:10-6:00)

    The basic unit of development is the squad. Because each squad sticks with one mission and one part of the product for a long time, they can really become experts in that area. A tribe is a collection of squads that work in related areas. The chapter is your small family of people having similar skills and working within the same general competency area (e.g. QA or Web Development), within the same tribe. As a squad member, my chapter lead is my formal line manager, a servant leader, focusing on coaching and mentoring me as an engineer, so I can switch squads without getting a new manager. A guild is a more organic and wide-reaching "community of interest", a group of people that want to share knowledge, tools, code, and practices. Chapters are always local to a tribe, while a guild usually cuts across the whole organization. Some examples are: the web technology guild, the tester guild, the agile coach guild. Anyone can join or leave a guild at any time. Most organizational charts are an illusion, so our main focus is community rather than hierarchical structure.

    -- from Scaling Agile @ Spotify (with Tribes, Squads, Chapters and Guilds) (pdf) by Henrik Kniberg & Anders Ivarsson and Spotify Engineering Culture (Part 1) by Henrik Kniberg (7:30-8:50)

    Alistair Cockburn (one of the founding fathers of agile software development) visited Spotify and said "Nice - I've been looking for someone to implement this matrix format since 1992 :) so it is really welcome to see"

    -- from Scaling Agile @ Spotify (with Tribes, Squads, Chapters and Guilds) (pdf) by Henrik Kniberg & Anders Ivarsson

    In learning more about Spotify, I was also glad to learn how they deal with specialization. You see, this tricky topic has been a chronic nuisance for us, for a number of reasons.

    First, code quality. New code written by a generalist, and reviewed by a generalist, is a long term code quality nightmare. Can you imagine what Perl code written by a ten-year Java veteran with a few days of Perl experience looks like? I can because I've seen it. There have been times when I've opened up a Perl file and rolled on the floor laughing because it was clear that whoever wrote it had no understanding of Perl at all.

    Second, employee engagement. Though many programmers are happy to become generalists, a significant minority (including me) are deeply dissatisfied with that role. I derive much more job satisfaction from doing an expert job in something I deeply understand than from "cargo-culting" some code that seems to kinda work even though I lack a deep understanding of why. I find that dissatisfying. I do not want to write Java code that causes my Java expert colleague to roll on the floor laughing.

    Another problem is that system architecture can be compromised if nobody focuses on the integrity of the system as a whole. To mitigate this risk, Spotify have a "System Owner" role. All systems have a system owner, or a pair of system owners (one with a developer perspective and one with an operations perspective is common). They further have a chief architect role, someone who coordinates work on high-level architectural issues that cut across multiple systems.

    Healthy Culture Heals Broken Process

    Trust is more important than control. Agile at scale requires trust at scale. And that means no politics. It also means no fear. Fear doesn't just kill trust, it kills innovation because if failure gets punished people won't dare try new things.

    -- from Spotify Engineering Culture (Part 1) by Henrik Kniberg (12:40-13:00)

    We focus on motivation, community and trust rather than structure and control. Healthy Culture Heals Broken Process.

    -- from Spotify Engineering Culture (Part 2) by Henrik Kniberg (0:40-0:50, 12:00-12:30)

    I've seen first hand how a healthy organisational culture of good communication and continuous improvement can effectively solve process problems.

    I've also seen first hand how a culture of fear and an over-emphasis on control and structure can harm innovation.

    Lean Startup

    We aim to make mistakes faster than anyone else. Continuous improvement, driven from below and supported from above. Failure must be non-lethal and with a "limited blast radius". Lean startup principles: think it, build it, ship it, tweak it. The biggest risk is building the wrong thing. Release first to a small percentage of users, then use A/B testing, then gradually roll out to the rest of the world. Impact is more important than velocity. Innovation more important than predictability.

    -- from Spotify Engineering Culture (Part 2) by Henrik Kniberg (2:00-)

    These are some of the ideas used by Spotify to build and release product. Since there is enough in this installment already, I'll postpone a discussion of Lean startup and related ideas to the next episode.

    Other Articles in This Series

    External References

Nobody Expects the Agile Imposition (Part VIII): Software Craftsmanship
4 direct replies — Read more / Contribute
by eyepopslikeamosquito
on Jun 11, 2015 at 05:22

    Long long ago, at an Agile summit in Utah, a group of software industry veterans, frustrated by what they saw as overly heavyweight software processes, concocted the Manifesto for Agile Software Development:

    • Individuals and interactions over processes and tools
    • Working software over comprehensive documentation
    • Customer collaboration over contract negotiation
    • Responding to change over following a plan
    ... and so the "Agile Transformation era" began:

    The Agile Alliance formed, conferences were held, companies went mad wanting a bit of this, everyone was having standup meetings, burndown charts, product backlogs, Agile coaches all over the place, more Agile coaches than developers ... and Post-its, Post-its, Post-its everywhere. The more Post-its you have the more agile you are.

    We spent ten years talking about people, interactions, team building, eliminating waste ... and at some point agile took a detour ... process became more important than technical practices ... everyone went crazy at the Post-it party, three years later they woke up and realised that every two weeks we see the pile of s#?! getting bigger.

    -- from Software Craftsmanship talk by Sandro Mancuso (2:00-5:00)

    Uncle Bob proposed adding the assertion that we value "Craftsmanship over crap" to the manifesto

    -- Uncle Bob, craftsmen and the Agile Manifesto

    Around 2008, some of the original Agile Manifesto-istas, led by Robert C Martin (aka "Uncle Bob"), felt that Agile had gone off the rails, with too much emphasis on process rather than technical practices and code quality. This group felt that Agile projects, every two weeks, relentlessly, steadily, iteratively, were producing more and more crap code. Adding to technical debt. Slowing us down. Though Agile gave good feedback on many things, code quality was not one of them; it is not visible on burndown charts or the Scrum board.

    How does this happen? Well, there seems to be an unwritten law of the Daily Standup that every morning you have to move at least one Post-it note. To comply with this "law", I sometimes pick up the Post-it note I was working on yesterday and move it a few millimeters to the right on the Scrum board while explaining why it is not really done yet. I always feel bad when I do this though; I would much rather proudly proclaim that it is done, so as to publicly show-off how productive I am.

    Sadly, I've occasionally noticed folks succumb to this psychological pressure to proclaim something done when it is not ... resulting in an every increasing pile of poo, as illustrated by Sandro's next (pair programming) anecdote:

    "but what about these names, they don't make sense ... and there's lots of duplication ... and ..." yeah, well as it's almost done now let's just check it in and add that to the technical debt backlog. This was a brand new feature in an agile team! The guy (Sandro was pairing with) was writing brand new code already thinking to add stuff to the technical debt backlog!

    -- from Software Craftsmanship talk by Sandro Mancuso (5:30-6:20)

    How to fix this lamentable state of affairs? The new Software Craftsmanship group chose to add four new Software Craftsmanship principles to the original Agile manifesto, creating a Manifesto for Software Craftsmanship:

    • Not only working software but also well-crafted software. Code we can refactor confidently and without fear.
    • Not only responding to change but also steadily adding value. Not just bug-fixing, but improving code structure, relentlessly keeping it clean.
    • Not only individuals and interactions, but also a community of professionals. Mentoring, sharing, user groups, passion, professionalism.
    • Not only customer collaboration, but also productive partnerships. Not a "manager giving orders to a developer" relationship, rather a partnership of equals built on trust.

    Well-crafted Software

    God forbid that we have one day someone who is a Software Craftsmanship coach ... it is about lead by example, being a mentor ... not about beautiful code, just trying to provide value, not writing crap code for your customers. The lack of craftsmanship can be one of the main causes of failing projects.

    -- from Software Craftsmanship talk by Sandro Mancuso (30:00-31:00)

    Like Sandro, I disagree the view that "Software craftsmen just care about beautiful code". Today, for example, it took me quite a while to figure out why some code that "should" fail was in fact reporting "success". I almost fell off my chair when I finally realized that it was silently catching and throwing away all exceptions! No comment explaining why it would do such a bizarre thing. To me, this sort of sloppiness shows a basic lack of respect for your colleagues; lack of care and comments wastes their time. It is unprofessional. Nothing to do with beautiful code.

    Steadily Adding Value

    After five years, the software is so s#?! that in the beginning it took two days to add a feature a feature of the same size now takes two months ... so they write a brand new one that is as s#?! as the previous one that will also be decommissioned in five years time.

    -- from Software Craftsmanship talk by Sandro Mancuso (17:00-18:00)

    Now the two teams are in a race. The tiger team must build a new system that does everything that the old system does. Not only that, they have to keep up with the changes that are continuously being made to the old system. Management will not replace the old system until the new system can do everything that the old system does. This race can go on for a very long time. I've seen it take 10 years. And by the time it's done, the original members of the tiger team are long gone, and the current members are demanding that the new system be redesigned because it's such a mess.

    -- Robert C Martin in Clean Code (p.5)

    The only way I can see to avoid this sort of fiasco is to to have the discipline to always keep the code clean in the first place, relentlessly refactoring as required every time you add a new feature.

    Productive Partnerships

    As soon as the button was pressed to mute NASA from our meeting, the managers said "we have to make a management decision", said Boisjoly.

    The general manager of Thiokol turned to his three senior managers and asked what they wanted to do. Two agreed to go to a launch decision, one refused. So he (the general manager) turns to him and said "take off your engineering hat and put on your management hat" -- and that’s exactly what happened, said Boisjoly. He changed his hat and changed his vote, just thirty minutes after he was the one to give the recommendation not to launch. I didn’t agree with one single statement made on the recommendations given by the managers.

    The teleconference resumed and NASA heard that Thiokol had changed their mind and gave a recommendation to launch. NASA did not ask why.

    I went home, opened the door and didn’t say a word to my wife, added Boisjoly. She asked me what was wrong and I told her "oh nothing hunny, it was a great day, we just had a meeting to go launch tomorrow and kill the astronauts, but outside of that it was a great day".

    -- from Remembering the mistakes of Challenger

    As indicated by the appalling "management decision" to launch the Challenger Space Shuttle, the lack of a true partnership, built on trust, between technical folks and management can have truly tragic consequences.

    Under pressure we cut corners ... No one wakes up in the morning and says "today I'm going to screw up, today I am going to write the worst code I can possibly write, I'm going to f#?! up with this company" ... I met some people who did that, but normal people don't do that ... everyone is trying to do a good job.

    The business asks how long is it going to take? ... The pressure we keep talking about, the pressure we put on ourselves ... we think we don't have time, even when we have the power to go to the business and tell them how long something is going to take!

    -- from Software Craftsmanship talk by Sandro Mancuso (7:00-9:00)

    A Community of Professionals

    Researchers (Bloom (1985), Bryan & Harter (1899), Hayes (1989), Simmon & Chase (1973)) have shown it takes about ten years to develop expertise in any of a wide variety of areas, including chess playing, music composition, telegraph operation, painting, piano playing, swimming, tennis, and research in neuropsychology and topology. The key is deliberative practice: not just doing it again and again, but challenging yourself with a task that is just beyond your current ability, trying it, analyzing your performance while and after doing it, and correcting any mistakes. Then repeat. And repeat again. There appear to be no real shortcuts: even Mozart, who was a musical prodigy at age 4, took 13 more years before he began to produce world-class music.

    -- from Peter Norvig's Teach Yourself Programming in Ten Years

    As indicated by Norvig above, it takes time and effort to become an expert, a true software craftsman. There appear to be no real shortcuts. With the current trend towards generalists in agile teams, finding the time to become a craftsman can be problematic as you find yourself constantly switching from one domain to another, from one language to another. This also affects code quality in that most code is being written by generalists, i.e. non-experts. This will be the topic of the next installment in this series.

    Other Articles in This Series

    External References

    Perl Monks References


Add your Meditation
Title:
Meditation:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others taking refuge in the Monastery: (7)
    As of 2015-08-04 00:04 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      The oldest computer book still on my shelves (or on my digital media) is ...













      Results (59 votes), past polls