Shell style line-editing without benefit of CPAN modules

[SirBones]

This is kind of a forced exercise in minimalism; good for the soul, I guess, if not for the digestion.

What I've got: A bunch (potentially dozens) of Linux-based control stations, all with special Linux builds with a pre-packaged Perl (5.8.3) with a set of some very few CPAN modules, selected as the Bare Necessities by The Great and Powerful Code Image Builder Who Cannot Be Argued With. For various practical, logistical, and possibly legal reasons (uninteresting in this discourse, I assure you) I will not have the option to install any additional Perl modules (and would have a hard time doing so anyway, since nether CPAN.pm or a make utility is included on these boxes.) So I am absolutely, positively stuck with the Perl capability I've got on these, and I have to make do.

I've written a Perl utility that provides a simple shell to process various user commands that interface with some external hardware; a standard while(<>) { }loop. A user will tend to be sitting there for a long time issuing commands and wanting to repeat them in various ways; in other words, it would be really nice if they had a standard left/right arrow non-destructive spacing, and up/down arrow command history/retrieval. Well, more than really nice; it's going to be a real burden on them if every time they make a typo or want to repeat the command with a slight change they have to retype the whole line. As it is right now, all they get when they hit the arrows is the code sequences '^[[A' or '^[[B' stuff. (esc character plus whatever.)

So perhaps you've guessed by now: There is no functional Term::Readline or Term:Readkey on these systems, and I cannot install one. I don't know a lot about terminal I/O or how shells operate on command lines under the covers, but I am guessing that these (or some of their brethren) would get me out of this mess. But I digress. It's obvious I need to come up with something home-grown, or do without. So, in that light...

Taking the example code for using the standard POSIX module from Camel pp. 905-906, I've hacked a possibility for detecting up and down arrow presses and gathering the current command string. This is far from elegant right now; it's just meant to show how far I've gotten, and the direction I'm going:

#!/usr/bin/perl
use warnings;
use strict;

my $str;
my $buff;

$| = 1;
for (1..100) {
    my $got;
    $got = getone();
    my $hx = unpack("H*",$got);
    # Look for "esc" character
    if ($hx eq '1b') {
        $str = $hx;
    }
    # Look for the 3 byte arrow sequence
    elsif ($str) {
        $str .= $hx;
        if ( length($str) == 6 ) {
            if ( lc($str) eq '1b5b41' ) {
                print "up arrow!\n";
            }
            elsif ( lc($str) eq '1b5b42' ) {
                print "down arrow!\n";
            }
            $str = undef;
        }
    }
    # If not enter key, add to buffer
    elsif (lc($hx) ne '0a') {
        print "$got";
        $buff .= $hx;
    }
    else {
        print "\nline is: ". pack("H*", $buff)."\n";
        $buff = undef;
    }
}
exit;

# Camel 3rd ed, pp. 905-906

BEGIN {
    use POSIX qw(:termios_h);
    my ($term, $oterm, $echo, $noecho, $fd_stdin);
    $fd_stdin = fileno(STDIN);
    $term     = POSIX::Termios->new();
    $term->getattr($fd_stdin);
    $oterm    = $term->getlflag();
    $echo     = ECHO | ECHOK | ICANON;
    $noecho   = $oterm & ~$echo;
    sub cbreak {
        $term->setlflag($noecho);
        $term->setcc(VTIME, 1);
        $term->setattr($fd_stdin, TCSANOW);
    }
    sub cooked {
        $term->setlflag($oterm);
        $term->setcc(VTIME, 0);
        $term->setattr($fd_stdin, TCSANOW);
    }
    sub getone {
        my $key = "";
        cbreak;
        sysread(STDIN, $key, 1);
        cooked();
        return $key;
    }
}
END { cooked() }

It would obviously be easy enough to extend this to detecting left and right arrows as well. But that's not the problem; I don't want to just detect them. I want them to function as if I'm in a regular shell and let me move around and edit the line.

A sidebar is that when I run my tool under a Win32 system (ActiveState Perl), it seems to have all this command line editing stuff built in without my having to do anything; I don't know if that's a function of ActiveState Perl, or the Win32 DOS Command shell. In any case, that's not an option here; the "shell" will need to run under Linux.

All this hot air (and I apologize for its length) is simply to prefix the question: Is there a way to get a line editing function using the arrow keys just using POSIX (or some other standard approach) without access to any of the extended functionality provided by CPAN modules?

Thanks in advance, lads & lassies; you are the best. But you knew that already.

-Ken

[chromatic]

Dear Boss,

Hello, how are you? I am doing well. I want to give you an update on my progress.

As you know, part of our product includes a command-line processor, where users can edit their commands, review previous commands, and edit and resubmit previous commands. This is familiar to anyone who's used a capable command-line on a Unix-like operating system, for example.

This is quite possible in Perl. However, I've come to a place of decision and would appreciate your guidance; I have two incompatible alternatives that affect the schedule. Can you advise me on which requirement is more important?

As you know, The Great and Powerful Code Image Builder Who Cannot Be Argued With has selected a small number of Perl modules, beyond which he will not install any more. There are several very capable Perl moduels which provide the command-line history and editing capability our product requires. They are small, well-tested, and well-documented. I estimate that we could integrate one into our product (from me selecting it, understanding its documentation, testing it, and having it installed into our image) in one afternoon.

If keeping the image as it is is more important, I estimate that it will take at least two weeks for me to reimplement this feature by myself. I have some confidence that I can get it working on our major platform, but with each subsequent platform, my confidence wavers. The difficulty of this task relates to historical standards for character sets, user shells, command characters, and terminal settings. For a good time, read through /etc/termcap on a Unix-like OS sometime.

A third option, of course, is to remove this feature.

As you can see, each of these options has positives and negatives which affect the end product. I feel that this is a decision which can best benefit from your input.

Best regards,
Ken

[SirBones]

Yes, yes. ++chromatic

To be a little more clear about the nature of my utility, it's not going to packaged as part of the final product. The machines themselves are controllers for much larger systems, and will go out to customers in this "plain vanilla" configuration for logistical, legal, and whatever reasons. My Perl code is only being used for engineering debug and bringup, and will be long gone before the machines go out. It's (usually) not the inclination of my boss to make my work more difficult; this is an unusual circumstance.

So in fairness, there are a lot of complex issues involved, and I didn't mean to over-simplify the Dilbert-nature of my workplace. Not in this case, anyway. ;-)

-Ken

[chromatic]

That's different, but I still think my point applies. The policy of not installing modules has a sane reason (we both assume), but in this case it has a cost tradeoff greater than a couple of hours. I don't see it as your job to decide how to proceed in such a case. I don't know what the right answer is for your business, but I do think it's worth bringing this up to your boss for advice.

This is one of those cases where something you didn't forsee in advance has the potential to change the scope of the work or the schedule. That happens. The best thing you can do is to identify it, be honest about the costs of all of the options, and let the person whose job it is to adjust the schedule or the scope do so.

[roboticus]

SirBones:

In that case, perhaps you could get someone to relent and let you install it in a *local* directory (like ~/perlib), and add that directory to your @INC for your debugging scripts. Then you can remove it all before delivery. That way, it needn't contaminate the normal /usr, /bin, /etc, etc. directories.

--roboticus

[samtregar]

Is there a way to get a line editing function using the arrow keys just using POSIX (or some other standard approach) without access to any of the extended functionality provided by CPAN modules?

Of course. How do you think the author of Term::ReadLine and friends did it? Of particular interest to you might be Term::ReadLine::Zoid, a pure-perl Term::ReadLine implementation.

So perhaps you've guessed by now: There is no functional Term::Readline or Term:Readkey on these systems, and I cannot install one.

Bullshit. You just haven't figured out how to install one yet! Hell, if you succeed in your attempt to reinvent Term::ReadLine you'll have done it via the hardest possible route - recode it by hand! Here's a few easier possibilities:

• See if you can get PAR to work in your environment. PAR lets you bundle scripts with the modules they need. Might work, depending on how crippled your Perl is.
• Copy code from useful modules directly into your scripts. Some modules will need massaging to work like this (if they use __DATA__ or AutoLoader, for example) but many will work just fine. If the module you need has XS pieces you'll need to cross-compile them and ship the built shared-libs with your script.
• Do a detailed estimate of how much time you're expecting to waste working around the lack of a reasonable Perl development environment. Present this estimate to The-Great-and-Powerful-Code-Image-Builder-Who-Cannot-Be-Argued-With's boss. Explain in excruciating detail exactly how much money is being lost due to his policy on Perl module installs. Who knows, maybe he's not so powerful!
• Find a new job. Coding in Perl without CPAN is a foolish waste of time with very little to recommend it.

-sam

[SirBones]

Find a new job. Coding in Perl without CPAN is a foolish waste of time with very little to recommend it.

True enough. Although I'd still like Perl even without CPAN. ;-)

I should make it clear however that this is a very unusual situation; this restriction is definitely not business as usual, and everyone understands that.

Thanks for the pointers, I think that will get me going.

-Ken

[samtregar]

True enough. Although I'd still like Perl even without CPAN. ;-)

To be honest, I don't. If CPAN weren't a factor I'm sure I'd have switched long ago - Scheme, Python and TCL all have syntax that makes more sense to me. CPAN, both as a place to find code and a place to share it, is what makes Perl my language of choice.

Re-reading, I realized I forgot to throw in a link to Krang. Krang is a big hairy Apache/mod_perl app that can be distributed and installed without requiring any external modules. It packages the CPAN modules it needs and has a build system that allows you to build on one machine and install on another. Overkill for a little script, perhaps, but another take on the problem.

-sam

[Anonymous Monk]

does your linux install have the bash shell with readline support??

#!/usr/bin/perl

open $shell, q(/bin/bash -c 'while read -e LINE; do history -s $LINE; 
+echo $LINE; done' |)
    or die $!;
while (<$shell>) {
    print;
}

muahahaha....

[SirBones]

(As my darling 16 YO daughter would say): O M G

Thanks so much; that seems to get me most of the way there. If yer weren't so Anonymous, I'd give ya a big ol hug. :-)

Not to be greedy (always spoken before one gets greedy): I note that this only returns a "history" of the last command entered, at least on the two systems where I tried it. Is there a way to get it to save/scroll a full history list? (As I diligently go back and read the Bash Reference Manual...)

Most appreciative,

-Ken

[zengargoyle]

really? on my bash: BASH_VERSION='3.1.16(1)-release', the above code does keep the the full history of the commands, not just the last command. (that was the hardest part of writing it :) ).

and i believe that there is probably a way to set a .history file so that it would keep commands between sessions.

anyway, good luck digging through the bash manpages. i hope you can get it to work "well enough" for your needs. it's probably just one of the bash 'set' options or an environment variable that you need set correctly to make the history work.

[graff]

Rather than trying to re-invent the interactive shell-command-editor wheel, it might make more sense for the users to adapt to a slightly different procedure...

You already have a handy tool to read and execute commands via  while (<>), so think about storing one or more commands in a script file, and simply running your "CLI" tool with the script file as input (instead of the keyboard).

What you need to optimize is just the "edit/re-execute" loop. Do you have any sort of multi-window-display ability (or at least, multi-tasking job control within a single window)? Edit the file in one window, and each time you save it, re-run the command line (the same string) again to see what the output is.

(If you don't have multiple windows for multiple tasks, maybe you can run vim or dumb-term-mode emacs at the shell to edit/save the script, ^Z to background that and return to the shell so you can run the script, then "fg" to foreground the editor again as needed.)