Perl White Magic - Special Variables and Command Line Switches

Context

This tutorial on Special Variables and Command Line Switches was originally published in the YAPC::EU::2004 proceedings (where it was awarded the prize for best paper).

Prologue

Everything was silent, but for the two of them breathing.

- 'All my life I've felt there was more to system administration than this... And now... now that you've arrived... now I know I was right!'

- 'You really think I'll be up to the job, Command Line?'

- 'Yes, you will. You will, $_.'

Super-powers come to those who wait

$_ woke up with a sudden and loud noise. It would have sounded like a fan, should he know what a fan was.
He knew there were things to be done, and that sooner or later he would be called into aid. He did not know, however, what those particular things were and who would be in charge of choosing how and what to do exactly.
As soon as these thoughts crossed his mind, gravity suddenly disappeared. Some kind of magnet picked him up and promptly took him to the top of a script, right after a shebang.
Beneath him, comments could be seen.

Commented code can mean an awfull lot of different things, and $_ knew this better than any other variable. It can be meant to identify the Creator or the purpose of life, but can also exist just to state that something is yet unfinished. One would hope it was not the case...

Slowly at first, he started going down the script. Slowly enough to read the comments:

# counts words and lines in a file

He passed that line and saw the next following ones:

        while (<>) {
                @words = split /\W+/;
                $words += @words;
        }
        print "lines -> $.\nwords -> $words\n";
[download]

Not knowing what to do, he started feeling anxious as gravity pulled him nearer to his destiny. At this point, Command Line shouted at him:

- 'Use the monitor on your wrist!!'

He looked at his wrist and saw something that resembled a smaller version of something someone wanting to hitchhike through the galaxy would take as a guide, but it was actually something someone wanting to hitchhiking through a Perl script would take as a guide. Plus, it had no "Don't Panic" stuff; instead, it had Perldoc written on it.
He quickly turned it on and searched for split (he already knew what a while was). He realized that being called with a single argument, split would need him to complete his task.
Knowing this, and reaching that line (several times in a row), $_ took his place in front of the split.
ZAP!!! ZAP!!!
His contents were instantely splited up and the resulting array was counted, having that result summed up to $words.
After exiting the while, $_ met $.:

  - 'Hi', said $_.
  - 'Hi.'
  - 'Who are you?'
  - 'I'm $..'
  - 'That I can see, but... exactly what are you doing here?'
  - 'I am counting lines.'
  - 'What lines have you been counting if I've been through all of them and never saw you?'
  - 'I see you're not familiar with special variables... I would advise you to make use of that little device on your wrist and search for perlvar.'

So he did, and promptly understood what a special variable was: they were there, they just couldn't be seen... $., in particular, had been counting the input lines right from the beginning of the script, and was ready to pop into action when called... and so it did.
Not a second after realizing all this, $_ was pulled back to where he was before. Command Line walked near him for a while:

  - 'How do you feel?'
  - '... Strange...'
  - 'Don't worry, you'll get over it. Happens to all of us. Soon you'll be ready for more.'

Different perspectives

As before, $_ was taken again by the strange force. He was ready for it before it started, but then came the surprise...
Strangely enough, this time he wasn't moving vertically, but rather horizontally... Everything resembled an assembly line.
The first thing he noticed was a command line switch, but he did not know what one was yet. He reached the monitor on his wrist and started hitting it desperately. He eventually strolled across perlrun and searched for it. He found it:

        -e commandline
            may be used to enter one line of program.
[download]

That explained the horizontal movement.
Looking back and forward, he saw the complete line of code:

perl -e 'while (<>) {$w += split /\W+/} print "$. $w\n"'

He could now understand what was going on. This was practically the same script as before, except now it wasn't a script anymore, it was an "One-Liner", one of the most beautiful creations by the hand and mind of the Original Creator (The one and only: Larry Wall).
The whole line resumed as expected, and he was on his way back home when he was suddenly pulled back again into action.
He saw a weird switch, this time. It was -ne. He looked it up but couldn't find it. He was staring at the beginning of chapter "Command Switches", wondering, when he read something that washed away any doubt:

        As with all standard commands, a single-character switch
       may be clustered with the following switch, if any.

           #!/usr/bin/perl -spi.orig   # same as -s -p -i.orig
[download]

-ne was surely the same as -n -e, and he already knew one of them. He looked for the other and promptly found it:

        -n   causes Perl to assume the following loop around
            your program [...]

              LINE:
                while (<>) {
                    ...             # your program goes here
                }
[download]

He looked at the rest of the line to get a better feel of what was going on:

perl -ne '$w+=split /\W+/; END{print"$w $.\n"}'

Everything seemed OK now. He already knew that any END block was to be executed only at the end of the script, after everything else. The program would go like this:

        LINE:
          while (<>) {
                  $w += split /\W+/;
                  END { print"$w $.\n" }
          }
[download]

That was rather close to his first task, but with less coding going around.
He finished his task and rushed for another one; this time it was a script inside a file again:

        #!/usr/bin/perl -n

        # counts words and lines in a file

        @words = split /\W+/;
        $words = @words;
        END { print "lines -> $.\nwords -> $words\n"; }
[download]

Someone was obviously doing experiments, but that was good for him, as he was surely learning from it.
By knowing the -n switch, someone had actually shrinked the code.
He would have asked -n what he though about it, but everyone knows command line switches don't speak. That would be just stupid.
$_ went home and rested for a while.

A whole bag of switches

Again, the same startup routine. This time, the user was noticeably more experienced:

        #!/usr/bin/perl -w
        use strict;
[download]

At least he seemed to know what he was doing. Every good script should start with something like that. The Perldoc screen told him:

-w prints warnings about dubious constructs

One should always use it.
Below, another comment:

#Adding a commit each 100 lines

He had heard something about this... DBA's sometimes have trouble inserting too many rows in a table, as that very same information is kept as roolback data between commit instructions (so everything can be undone, if needed).
Adding a commit each 100 lines would solve this problem, as the maximum of information kept at a time would consist of 100 records.
Following, the code:

        my $line;

        while (<>) {
          $line++;
          unless ( $line % 100 ) {
            print "commit;\n";
          }
          print;
        }
[download]

It wasn't as complicated as it would seem at first. He noticed the $line part could be replaced with $., and the rest of the code could be shortened with the -n, but he really wasn't expecting what happened in his next task.
Just after finishing this, he was called again into action. He was getting used to this and wouldn't get dizzy anymore.

        #!/usr/bin/perl -wp
        use strict;

        #Adding a commit each 100 lines

        unless ( $. % 100 ) { print "commit;\n"; }
[download]

He looked up for -p:

        -p   causes Perl to assume the following loop around
            your program [...]

              LINE:
                while (<>) {
                    ...             # your program goes here
                } continue {
                    print or die "-p destination: $!\n";
                }
[download]

- 'So that's why the print statement has been removed.'

It simply wasn't needed anymore, as the -p switch would do the same as -n and also print each line.
He was pondering about this, when yet another surprise called for his attention. He was, once again, in the Command Line; the user was making use of the -e switch once again:

perl -i -pe 'print "commit;\n" unless ($.%100)'

He knew what to do:

        -i[extension]
            specifies that files processed by the "<>"
            construct are to be edited in-place.  It does
            this by renaming the input file, opening [...]
[download]

- 'Wow.' He couldn't help the remark.

That meant the input was to be read, processed, and then rewritten over the original file. This would probably save the user the trouble of copying a new file over an older one.
He was taken back to his home.
How many more switches would he meet?

They come in pairs

The same routine. Once again, $_ was ready for surprises and had his Perldoc prepared.

        #!/usr/bin/perl -wlp
        use strict;

        # trims lines to 80 columns

        substr($_, 80) = "";
[download]

He looked up -l and, to his surprise, he found the exact code he was running on. Hum... the user probably had a Perldoc of his own... (Perhaps he was a special variable too?) The text on -l would go:

        -l[octnum]
            enables automatic line-ending processing.
[download]

After reading a little bit more, he understood what was going on: the trailing "\n" on each line would be discarded, each line would be trimmed up to 80 chars and printed with the "\n" back on. He noticed the text mentioned two other variables: $/ and $\... and their names sounded fierce.

Usually, one goes from more code to less code, and not the other way around, but this was exactly what happened next. His next script, instead of using the -l switch, seemed to do the same by using those very same two variables. The user was obviously up to something (probably just testing and learning new features).

        #!/usr/bin/perl -wp
        use strict;

        # trims lines to 80 columns

        chomp;
        $\ = $/;

        substr($_, 80) = ""
[download]

Dazzled in astonishment, he didn't even think about reaching for his Perldoc. He instead asked the variables directly what they were up to.

  - 'Hi.
  - 'Can't you see we're busy?' said $/.
  - 'I'm sorry, but I'm new at this, and so many special variables and command switches are starting to mess with my mind...'
  - 'Oh, a beginner...' said $\.
  - 'Indeed, a beginner.' $/ agreed.
  - 'Yes. I was wondering who you guys are, and what you are doing.'
  - 'I am the input record separator. Usually, I contain a "\n"', explained $/.
  - 'And I am the output record separator... usually containing nothing...' sighed $\, with a slight note of disappointment.
  - 'And what are you doing?' asked $_.
  - 'We're swapping contents.' said $\.
  - 'Yes, it's a rather common practice.' Continued $/.
  - 'Actually, it's so common that someone actually invented a switch for this.'
  - 'So I noticed. -l, isn't it?'
  - 'So you've seen it too, uh?'
  - 'Yes.'

As good as the conversation was getting, $_ noticed the end of the script arriving.

  - 'What's the purpose of what you're doing?'
  - 'Well, you see, most of the times you print something, you're actually printing a line, but that means you have to print the "\n" too, as the common print statement does not assume that. However, by making use of $\ here, you have no need of doing that...'
  - 'Yes, I'll do it for you!' interrupted $\ 'Just give me whatever you want printed on every line, and I'll take care of it for you! We're close friends, print and I.'
  - 'Wow... that's nice...'
  - 'Yes, and that's not all. You should see what *I* can do!' said $/.
  - 'Really, what can you do?'
  - 'Later, my friend, here comes EOF to take us home.'

So that was the name of this magnet: EOF.

The mighty variable

$_ fell asleep with $/ on his head, and woke up still thinking about it.
He was called into action and, to his contentment, there was $/:

        #!/usr/bin/perl -wpi
        use strict;

        $/ = "";

        while(<>) {
                chomp;
                print "$_\n";
        }
[download]

  - 'Hi.'
  - 'Hi.'
  - 'I can't see any comments on this script. What's going on?'
  - 'We're joining paragraphs.'
  - 'What do you mean?'
  - 'I mean we're joining paragraphs. Those having more than two blank lines between them are being pushed closer to each other. Basically, we're normalizing text.'
  - 'I see... And how exactly are we doing that? And what's your part in all this?'
  - 'Well, as I told you before, I am the input record separator, which means I'm the one who decides the input that gets in at a time. Regularly, I have a string (usually "\n"), which is what I use to make that decision, spliting the input on that string, but I do have a couple of more tricks up my sleeve...'
  - 'I can imagine that. Are you using one of those right now?'
  - 'Yes. I am using the paragraph mode. By assigning me to an empty string, the user gets me to separate input at blank lines. Basically, at "\n\n+".'
  - 'I see. But why not assigning you to "\n\n+" directly?'
  - 'Hey, kiddo! See me flying?'
  - 'No...'
  - 'Know why?'
  - 'Why?'
  - 'That's because I'm a special variable, not a super variable. I can't deal with regular expressions.'
  - 'Hum... And I assume a simple "\n\n" wouldn't suffice, as that wouldn't get all the cases...'
  - 'You assume correctly. I would take "\n\n\n\n", for instance, and find another record in the middle of it. Hence, paragraph mode.'
  - 'Wow... Nice trick!'
  - 'Thanks.'
  - 'Too bad you don't do regular expressions, though...'
  - 'It's not that bad, really. Anyway, awk has to be better for something.'
  - 'I suppose you're right. What about your other tricks?'
  - 'Well, I can be assigned to a reference to an integer, a scalar containing an integer or a scalar that's convertible to an integer...'
  - 'And what will that do?'
  - 'Given that integer, I will read that many bytes at a time.'
  - 'Wow.'
  - 'Gotta go. Bye.'
  - 'See you around.'

That night, $_ had trouble falling asleep, dreaming of the possibilities those $/'s tricks could mean.

Separations

Yet another day.

perl -ane 'print shift(@F), "\n"'

Everything was so easy now that he was used to Perldoc. He was even able to open two windows at a time, one with perlrun and another with perlvar. He used the first one to search for -a:

        -a   turns on autosplit mode when used with a -n or
            -p.  An implicit split command to the @F array
            is done as the first thing inside the implicit
            while loop produced by the -n or -p.

                perl -ane 'print pop(@F), "\n";'

            is equivalent to

                while (<>) {
                    @F = split(' ');
                    print pop(@F), "\n";
                }
[download]

and the second for @F:

        @F      The array @F contains the fields of each
               line read in when autosplit mode is turned
               on. [...]
[download]

To tell the truth, the entry for -a already explained what @F was doing there. Anyway, the code in the documentation was very similar to the one he was running on. He was clearly on a script for printing the first word of every input line. What for? He would soon find out.
Next, he passed the following code:

        #!/usr/bin/perl -wane
        use strict

        # study commands (input is .bash_history)

        $commands{shift(@F)}++;

        END {
                for (keys %commands) {
                        print "$_ -> $commands{$_}\n"
                }
        }
[download]

This was a very simple way of studying how often commands are ran on the system.
There was still another line of documentation, regarding -a:

An alternate delimiter may be specified using -F.

Another switch. And soon he would see and example of its usage:

perl -F: -ane 'print shift(@F),"\n"'

Hum... this would split the input lines on ':' and print every first element of those lines. (Maybe it would run on /etc/passwd, for a list of users)

TIMTOWTDI

Command Line approached $_ softly and woke him gently:

  - 'Wake up, $_!'
  - 'What is it?'
  - 'Come, you're going to learn an important lesson today.'
  - 'OK.'

They walked along for a while.

- 'So what's this lesson I'm going to learn?'
- 'Be patient, my friend. Look, here's your first script for today!'

        #!/usr/bin/perl -w
        use strict;

        my @a = (1, 2, 3, 4, 5);

        print "@a";
[download]

$_ went through it as if he hadn't been doing anything else all his life.

- 'There.'

- 'Very good. Did you notice the output?'

1 2 3 4 5

- 'Cool...', said $_, 'Perl already knows he has to put a space between those elements, right?'
- 'Wrong. The truth is Perl knows he has to put something between those elements, allright. You're about to meet that something.'

        #!/usr/bin/perl -w
        use strict;

        my @a = (1, 2, 3, 4, 5);

        $" = "|";
        print "@a";
[download]

- 'Done!'
- 'Well done. Now, did you notice the difference?'

1|2|3|4|5

- 'Yes. I assume Perl takes the contents of $" and prints that everytime an array is printed. Right?'
- 'Wrong again... Perl only uses $" if the array is inside quotes. Take a look at our next example!'

        #!/usr/bin/perl -w
        use strict;

        my @a = (1, 2, 3, 4, 5);

        print @a;
[download]

$_ went through it.

- 'Now look at the output produced.'

12345

- 'Odd... shouldn't a space be there too, separating elements?'
- 'No. When not quoted, an array to be printed is joined with the contents of $,, and that variable is empty by default. Next example!'

        #!/usr/bin/perl -w
        use strict;

        my @a = (1, 2, 3, 4, 5);

        $, = "-";
        print @a;
[download]

- 'And now the output!'

1-2-3-4-5

  - 'See?'
  - 'Yes... Interesting...'
  - 'Indeed. Let's go.'
  - 'OK...'
  - 'See how you can do the same with two different variables?'
  - 'Yes.'
  - 'That's your lesson for today. There's more than one way to do it!'
  - 'I see...'

$_ was clever enough to remain silent for a while and think about what Command Line had just said... "There's more than one way to do it!"

Thoughts

'I like it here.' exclaimed $_. 'All these switches and special variables, being famous and loved by all.'

  - 'That's not entirely true... Some variables are not very well known, and others are not very appreciated.' replied Command Line.
  - 'For instance...?'
  - '$..'
  - 'Really? Is that true?'
  - 'Yes, it is.'
  - 'Who doesn't appreciate it?'
  - 'YAPC speakers who have to write articles with sentences ending in $..'
  - 'Why? I don't see any problem with a sentence ending in $..'
  - 'Oh, but there are problems with sentences ending in $..'
  - 'I really can't see what's wrong with $....'
  - 'Then you probably didn't pay attention to the four previous lines.'
  - 'Oh! That! I didn't realize that... Boy, it must really suck, being the only variable some people don't like.'
  - 'Oh, not the only one...'
  - 'Who else, then?'
  - 'Guess.'
  - '$,, $??'
  - 'Those and $!!'

Modules and one-liners

$_ was on the move again.

        perl -MExtUtils::Installed -le '
                print for ExtUtils::Installed->new()->modules'
[download]

He looked -M on Perldoc and was preparing to look for -E too when he noticed there was no need for that...
-M, according to Perldoc, would do the same as a use clause inside the script. Basically, that line was the same thing as

perl -        le 'use ExtUtils::Installed;
                print for ExtUtils::Installed->new()->modules'
[download]

The E was actually the beginning of a parameter for the -M switch.
He also looked for documentation on that module (with Perldoc, too). He found it and came to the conclusion that that line of code would print the name of every Perl module installed. That was a nice way to know what was on the system.

The sky is the limit

'I think I'm getting the hang of this.'
  - 'There is still much to learn, $_.'
  - 'Are there more special variables?'
  - 'Quite a few. And some more command line switches, too.'
  - 'Wow... This is just amazing...'
  - 'I'm glad you like it. However, we won't have the time to go through all of them now.'
  - 'Oh... That's too bad...'
  - 'Don't worry. There are lots of places where you can learn more, and Perldoc is one of them.'

$_ stared at his Perldoc.

- 'And besides that, you have lots of mailing lists, perlmonks, etc. You'll never run out of information. The trick is to keep practising.'

$_ smiled.

Revelations

$_ was browsing his Perldoc and talking to Command Line:

- 'Perl really seems to be a wonderful language, and all the special variables and command line switches take an important part in it.'

- 'Yes, but there is much more than that to Perl.'

- 'Really?'

- 'Yep. You should wait to read some of the other articles the author of this one is thinking of writing.'

- 'Wow... I can see why the user chose Perl.'

- 'Yes, but it wasn't like that in the beginning.'

- 'Really?'

- 'Really. Things were a lot different, until the moment he had an epiphany, and heard a voice saying: "Don't try to do everything with the Bash. Instead, only try to realize the truth."'

- 'What truth?'

- 'There is no Bash.'

Comment on Perl White Magic - Special Variables and Command Line Switches Select or Download Code


laziness, impatience, and hubris
	PerlMonks