Despite my limited knowledge of both subjects, I have been asked to
give a 3-day course "Unix and Perl for Beginners" for some of my
cow-orkers. I was thinking of looking at problems such as Untangling
Log Files, seeing what sort of simple, ad hoc approaches a Unix shell
such as bash might offer, and then develoing a nice Perl solution.
What I need are a few more ideas, besides the one mentioned above,
for suitable problems. Other than dealing with huge, incomprehensible
log files on a daily basis, my cow-orkers also have to grub about in
rather large databases and tweak data. In addition they use a complex
build process which is controlled by setting dozens of environment
variables.
I had a look at the 99 Problems in Perl6, but they are rather too
rarified for my purposes. I'm looking for something more along the
lines of "Plenty of Practical Perl Problems" but maybe it is expecting a
little too much to find many problems that are both somehow generic
and at the same time specific enough for the solution to be a
stand-alone utility.
Any ideas?
Update: I should have said that my cow-orkers are developing and/or testing software (not written in Perl) and need Unix and Perl to paper over the cracks in the various processes.
loris
"It took Loris ten minutes to eat a satsuma . . . twenty minutes to get from one end of his branch to the other . . . and an hour to scratch his bottom. But Slow Loris didn't care. He had a secret . . ." (from "Slow Loris" by Alexis Deacon)
|
Re: Plenty of Practical Perl Problems?
by Corion (Patriarch) on Jul 25, 2007 at 09:03 UTC
|
I would try to cover the following topics as they seem to be appropriate to your cow-orkers, together with the problems. I would try to structure the path to these problems by asking the cow-orkers the "right" questions leading them to the problem domain ("What task do you do when loading the new customer data" could lead to DBI or to SQL::Statement or to Template::Toolkit, depending on whether you (want to) load your data through DBI or the datbase loader tool for example).
Topics to know would be:
- Munging data:
s///, m//, Text::CSV, SpreadSheet::ParseExcel, SpreadSheet::WriteExcel and possibly automating Microsoft Office if your platform has OLE)the "Unix shell to Perl traveler's handbook", UNIX 'command' equivalents in Perl
- Launching subprocesses (or, "how to use Perl as if it were a shell script")
perlipc, backticks, sed, awk. A good example might be how to transparently read from a possibly compressed file, by reading from open my $fh, "gzcat $file |"; or somesuch.
- Something for the build process could be a reprise of perlipc, possibly together with Expect for automating some user inputs or Net::Telnet or Net::SSH for kicking off processes remotely, or even parallelizing stuff via Parallel::ForkManager. Or maybe transferring files via Net::FTP.
- Things where Perl "shines" over "The Shell" are file/terminal IO, arrays and dictionaries, while Perl is not necessarily as convenient for oneliners and process control as some shells. I'm sure your shell-loving cow-orkers will find plenty of things where Perl is uglier than shell :)
| [reply]
[d/l]
[select] |
Re: Plenty of Practical Perl Problems?
by roboticus (Chancellor) on Jul 25, 2007 at 11:37 UTC
|
loris:
When people at work ask me why I do stuff in perl, I often show 'em a couple of the more convincing examples that I've encountered:
- Query a database (DBI, DBD::ODBC), format the results into a spreadsheet (Spreadsheet::WriteExcel), and then EMail (I forgot which package I use here) the resulting file to a distribution list.
- Query several different databases (DBI, DBD::ODBC), correlate the data (hash), and dump a simple report.
- Rip through a text report and find anomalous records. Yeah this one is an easy awk script too, but the more I use perl the less often I reach for awk...
...roboticus
| [reply] |
Re: Plenty of Practical Perl Problems?
by hawtin (Prior) on Jul 25, 2007 at 09:17 UTC
|
I use Perl to manipulate my music collection. I have scripts
that validate MP3 tags, convert between formats, re-tag mp3 files as they copy them to diferent players etc.
My reasons for using Perl in this case are that there are lots of things I want to do, my processing time is close to infinite but my development time is very limited.
I have long suspected that these type of issues would be
a good way to discuss many of the key Perl skills (data
manipulation, controlling other programs etc) in a context
that is both interesting to most people and slightly away from "work".
Validate MP3 Tags
When I get a new CD I rip it in FLAC and save the data. Once I have more than 4Gb I create an index file (with a Perl script) and write it to a DVD. In the meantime I create a 128kB MP3 and tag it using public tools. The issue is that the artist names are not consistent and the year field is almost never set, so I have a Perl script that checks the tags for consistency and works out the year.
I then have a Perl script that places the new tracks in my "on-line" set (on the house DAAP server), it takes care of transferring to the dedicated machine and making sure the server can see it.
These tasks are all implemented by combinations of CPAN
modules and system() calls to existing programs (for example I use lame to encode mp3s).
Different Portable Players
The family have a number of different MP3 players, each of
which has its own strange behaviour. I have a Perl script
that, once you have selected which player you have, may
convert the format (for example to 64kB MP3 for the older
players with limited capacity) and retag.
I have one player in particular that either plays the tracks
in order of creation, or in alphabetical order of track name. If the album needs to be played in the right order the transfer program prepends the track number to the
trackname tag (so it becomes, for example "a01 -
Grace Kelly") and playing the tracks in alphabetical order works.
Finally
I also have a collection of music charts which are consolidated in Perl to create the web site http://www.tsort.info/music/ (that is how I can assign the years to mp3 tracks).
| [reply] |
Re: Plenty of Practical Perl Problems?
by scorpio17 (Canon) on Jul 25, 2007 at 13:38 UTC
|
Take a look at "The Perl Cookbook" - almost every recipe in there would make a good class topic.
Or - browse topics on this site for a few days - especially "Seekers of Perl Wisdom" - people here come up with some real doozies!
In general, any problem that involves lots of file manipulation and/or text processing lends itself to a good perl solution.
If you're presenting to an audience that wants to know "what's it good for?" or "why should I care?", consider this: Unix has a large toolbox of text processing utilities: sed, grep, awk, etc - just to name a few. Perl is like all of these rolled up into one nice package. The syntax is similar to C/C++, so most programmers can learn it pretty fast. It's not called the "swiss army knife of programming languages" for nothing!
Another big selling point: CPAN. They say 90% of any perl project has already been written by sombody, somewhere - and it's all there for free.
| [reply] |
Re: Plenty of Practical Perl Problems?
by planetscape (Chancellor) on Jul 25, 2007 at 18:02 UTC
|
You may wish to have a look at Re: Real Life Perl Exercises, where I have compiled a fairly comprehensive list of threads like this one. It hasn't been updated in a while, but it should give you plenty to chew on.
| [reply] |
Re: Plenty of Practical Perl Problems?
by blazar (Canon) on Jul 25, 2007 at 10:22 UTC
|
What I need are a few more ideas, besides the one mentioned above, for suitable problems. Other than dealing with huge, incomprehensible log files on a daily basis, my cow-orkers also have to grub about in rather large databases and tweak data. In addition they use a complex build process which is controlled by setting dozens of environment variables.
Not exactly the kind of thing your cow-orkers seem to be deaing with on a daily basis, but I may suggest you two tasks from my early Perl days, that I had formerly addressed with rather complex and clumsy shell scripts:
- Pick up files from a complex directory hyerarchy and move them into a simpler one, with new filenames corresponding to the whole old path and somewhat massaged for compatibility with M$ filesystems and with iso9660 ones (yes I know 'bout RR before someone mentions that). Since the number of files is very huge, move to numbered subdirectories of a given target directory with each subdirectory holding at most 1000 files, all from corresponding original subdirectories or -in the remote event that one of them contained more than 1000- split them intelligently.
- generate some off line html docs and/or reports by first rolling your own mini-templating-language first in shell, then in Perl and then add features till it starts to become a mess and then move to some real templating system.
| [reply] |
Re: Plenty of Practical Perl Problems?
by philcrow (Priest) on Jul 25, 2007 at 13:29 UTC
|
This is only an example, but I have just written a script to check the crontab to ensure that backups of databases are scheduled. This is a run of the mill task for an admin oriented perl programmer. For those unfamiliar with perl, it would show off many useful concepts, like hashes to keep track of what needs to be backed up and what is backed up; foreach loops to build and compare the hashes; string manipulation to pull names out of crontab lines; etc.
All sorts of monitoring scripts can help to prevent disasters, or help you deal with their consequences.
Phil
| [reply] |
Re: Plenty of Practical Perl Problems?
by eyepopslikeamosquito (Archbishop) on Jul 26, 2007 at 03:55 UTC
|
| [reply] |
|
|
