Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Cool Uses for Perl

( #1044=superdoc: print w/ replies, xml ) Need Help??

This section is the place to post your general code offerings.

CUFP's
Archive by month and extension
No replies — Read more | Post response
by GotToBTru
on Nov 24, 2014 at 16:01

    Created the following to archive data from our applications. We archive by month, and by file extension, so those are built in assumptions in this program.

    Source Code:

    POD:

    1 Peter 4:10
img - display a small graphic file at the command line
1 direct reply — Read more / Contribute
by sflitman
on Nov 22, 2014 at 21:21

    I do a lot of work in Putty and need to look at icon files sometimes. I thought it would be cool to get Putty to display them in bash directly, rather than using X11 forwarding. This is not meant to be any kind of substitute for real graphics, but is a quick way to see whether a particular image file (like an icon or a web button) is what I think it is. Note that it requires 256 color to be turned on in Putty, that your Terminal setting is putty-256color, and it only handles image formats handled by GD (png, jpg, gif).

    #!/usr/bin/perl # Steve Flitman - released to Public Domain - display a small image on + the console using 256-color mode Putty/Screen # Color output to terminal derived from Todd Larason <jtl@molehill.org +> use strict; use warnings; use GD; unless (@ARGV) { die "img file ...\nDisplay files at command line using ANSI 256 col +or mode\n"; } # set colors 16-231 to a 6x6x6 color cube for (my $red=0; $red<6; $red++) { for (my $green=0; $green<6; $green++) { for (my $blue=0; $blue<6; $blue++) { printf("\x1b]4;%d;rgb:%2.2x/%2.2x/%2.2x\x1b\\", 16 + ($red * 36) + ($green * 6) + $blue, ($red ? ($red * 40 + 55) : 0), ($green ? ($green * 40 + 55) : 0), ($blue ? ($blue * 40 + 55) : 0)); } } } # colors 232-255 are a grayscale ramp, intentionally leaving out black + and white for (my $gray=0; $gray<24; $gray++) { my $level=($gray * 10) + 8; printf("\x1b]4;%d;rgb:%2.2x/%2.2x/%2.2x\x1b\\", 232 + $gray, $level, $level, $level); } my ($file,$x,$y,$r,$g,$b,$color,$index,$image,$width,$height); for $file (@ARGV) { die "Cannot read $file: $!\n" unless -r $file; my $image=GD::Image->new($file); die "Not a recognized image format: $file\n" unless defined $image; my ($width,$height)=$image->getBounds(); for (my $y=0; $y<$height; $y++) { for (my $x=0; $x<$width; $x++) { my $index=$image->getPixel($x,$y); my ($r,$g,$b)=$image->rgb($index); if ($r+$g+$b==0) { # black $color=0; } elsif ($r==255 && $g==255 && $b==255) { # white $color=15; } elsif ($r==$g && $g==$b) { # grayscale $color=232+($r>>3); } else { $color=16+(int($r/42.6)*36)+(int($g/42.6)*6)+int($b/42.6); + # smush 256 color range to 6 levels } print "\x1b[48;5;${color}m "; } print "\x1b[0m\n"; # reset } print "\x1b[0m\n"; # reset } exit;

    Dedicated to the memory of John Todd Larason, http://molehill.org

    Enjoy!

    SSF

NSA's FoxAcid
2 direct replies — Read more / Contribute
by morgon
on Nov 20, 2014 at 09:49
    According to various top-secret documents provided by Snowden, FoxAcid is the NSA codename for what the NSA calls an "exploit orchestrator," an internet-enabled system capable of attacking target computers in a variety of different ways. It is a Windows 2003 computer configured with custom software and a series of Perl scripts.

    (Source: http://www.theguardian.com/world/2013/oct/04/tor-attacks-nsa-users-online-anonymity)

    It may stretch the definition of "cool" and may be old news but maybe a few monks will find this amusing...

read file a paragraph at a time and return paragraph OR get specific data from paragraph
3 direct replies — Read more / Contribute
by james28909
on Nov 02, 2014 at 16:10
    i have figured out how to read a file a paragraph at a time :D lol. in this example i will be reading a file a paragraph at a time and then getting only certain data back.

    this will return everything AFTER the pattern up until a newline char.works very quickly for me but prob could be done faster by a seasoned coder. you can modify it to suit your needs hopefully.
    tested with active perl and win7 ultimate x64
    $/ = ""; #set to paragraph mode while(<$file>){ if ($_ =~ /$first_match/ && $_ =~ /$second_match/ && $_ =~ /$third +_match/){ print "$_\n"; my ($needed_data_0) = (/^data_here(.+)/, $_); my ($needed_data_1) = (/^more_data(.+)/, $_); my ($needed_data_2) = (/^other_data(.+)/, $_); print "data0 is: $needed_data_0\n"; print "data1 is: $needed_data_1\n"; print "data2 is: $needed_data_2"; } }
    any input or pointers are welcome.
    :)
Random video player
2 direct replies — Read more / Contribute
by james28909
on Oct 29, 2014 at 14:02
    this little script will use file random to get a video file and will use your player of choice to launch it. mediainfo.exe was used to get sleep count.

    if you see anyway i can improve it please make a comment. i am trying to figure out how to push the filename to an array and do a check if it has played it in the last 100 videos/loops(that would help keep it from repeating if it does i think). i would also like to specify how many times to loop before exiting, that way it doesnt just loop forever and ever and the user can specify how many shows they want to watch.

    i wasnt please with the way i had to get duration, but it worked and i was astounded, so if you have of a better was to get duration time, please let me know. the problem i was dealing with while getting duration with any tool was spaces in the path/filenames. Here is the code:
    use strict; use warnings; use diagnostics; use File::Random qw(random_file); my $dir = $ARGV[0]; if ( not defined $dir ) { print "\nUsage: random.pl [folder]; exit(0); }else{ random($dir); } sub random{ my ($dir) = @_; while (1){ my $mpc = "C:/Program Files (x86)/K-Lite Codec Pack/Media +Player Classic/mpc-hc.exe"; my $rndm_file = random_file( -dir => $dir, #-check => qr/./, -recursive => 1 ); if ($rndm_file =~ /\.(ini|nfo|db)$/i){ print "$rndm_file\n"; random($dir); } print $rndm_file; #get duration my $t = ("MediaInfo.exe --Output=Video;%Duration% \"F:/TV/ +$rndm_file\""); system(1, $mpc, "F:/TV/$rndm_file"); my $time = qx($t); my $sleep_time = $time/1000; #in seconds because m +ediainfo.exe outputs milliseconds i think. print "\nDuration in seconds: $sleep_time\n"; sleep($sleep_time); random($dir); } }
    this works great if you have a ton of media in a folder and want to randomly watch any given inside in that directory, and you do not have to worry about spaces in path/filenames. and this is also for windows but i think it could be used on any other platform as well with small changes
Using Data::Compare recursively to better identity differences between hashes
2 direct replies — Read more / Contribute
by Lady_Aleena
on Oct 18, 2014 at 01:49

    Yesterday I wanted to compare two hashes to see if they were the same. I looked around a little bit and found Data::Compare. It was good at telling me the two hashes were different, however, it did not tell me where. So, I wrote a small little subroutine to recursively check my hash of hashes (of hashes). It was able to identity where I had to look to make corrections, almost to the exact spot. (I am unsure how to compare arrays of hashes just yet which is why the following little subroutine will almost take you to the right spot.)

    There are still holes in the script, but it worked for me today.

    #!/usr/bin/perl use strict; use warnings FATAL => qw( all ); use Data::Compare; use Data::Dumper; # You can take out all instances of the subroutine 'line' to print wha +t you want in those places. sub deep_data_compare { my ($tab, $old_data, $new_data, $data) = @_; my $old = $old_data; my $new = $new_data; my $compare = new Data::Compare($old, $new); if ($compare->Cmp == 0) { line($tab,$data) if $data; if (ref($old) eq 'HASH' && ref($new) eq 'HASH') { line($tab+1,'old to new'); for (keys %$old) { deep_data_compare($tab+2,$_,$$old{$_},$$new{$_}); } } # I have not figured out this part yet. # elsif (ref($old) eq 'ARRAY' && ref($new) eq 'ARRAY') { # } else { print Dumper($new); print Dumper($old); } } } sub rline { my ($tab,$line) = @_; return qq(\t) x $tab.qq($line\n); } sub line { print rline(@_); } deep_data_compare(0, \%old_hash, \%new_hash, 'widgets');
    No matter how hysterical I get, my problems are not time sensitive. So, relax, have a cookie, and a very nice day!
    Lady Aleena
Identifying scripts (writing systems)
2 direct replies — Read more / Contribute
by AppleFritter
on Sep 16, 2014 at 17:32

    Dear monks and nuns, priests and scribes, popes and antipopes, saints and stowaways lurking in the monastery, lend me your ears. (I promise I'll return them.) I'm still hardly an experienced Perl (user|programmer|hacker), but allow me to regale you with a story of how Perl has been helping me Get Things Done™; a Cool Use for Perl, or so I think.

    I was recently faced with the problem of producing, given a number of lines each written in a specific script (i.e. writing system; Latin, Katakana, Cyrillic etc.), a breakdown of scripts used and how often they appeared. Exactly the sort of problem Perl was made for - and thanks to regular expressions and Unicode character classes, a breeze, right?

    I started by hardcoding a number of scripts to match my snippets of text against:

    my %scripts; foreach (@lines) { my $script = m/^\p{Script=Latin}*$/ ? "Latin" : m/^\p{Script=Cyrillic}*$/ ? "Cyrillic" : m/^\p{Script=Han}*$/ ? "Han" : # ... "(unknown)"; $scripts{$script}++; }

    Obviously there's a lot of repetition going on there, and though I had a list of scripts for my sample data, I wasn't sure new and uncontemplated scripts wouldn't show up in the future. So why not make a list of all possible scripts, and replace the hard-coded list with a loop?

    my %scripts; LINE: foreach my $line (@lines) { foreach my $script (@known_scripts) { next unless $line =~ m/^\p{Script=$script}*$/; $scripts{$script}++; next LINE; } $scripts{'(unknown)'}++; }

    So far, so good, but now I needed a list of the scripts that Perl knew about. Not a problem, I thought, I'll just check perluniprops; the list of properties Perl knows about was staggering, but I eventually decided that any property of the form "\p{Script: ...}" would qualify, so long as it had short forms listed (which I took as an indication that that particular property was the "canonical" form for the script in question). After some reading and typing and double-checking, I ended up with a fairly long list:

    my @known_scripts = ( "Arabic", "Armenian", "Avestan", "Balinese", "Bamum", "Batak", "Bengali", "Bopomofo", "Brahmi", "Br +aille", "Buginese", "Buhid", "Canadian_Aboriginal", "Carian", "Chakma", "Cham", "Cherokee", "Coptic", "Cuneiform", "Cypriot", "Cyrillic", # ... );

    Unfortunately, when I ran the resulting script, Perl complained:

    Can't find Unicode property definition "Script=Chakma" at (...) line ( +...)

    What had gone wrong? Versions, that's what: I'd looked at the perluniprops page on perl.org, documenting Perl 5.20.0, but this particular Perl was 5.14.2 and didn't know all the scripts that the newer version did, thanks to being built against an older Unicode version. Now, I could've just looked at the locally-installed version of the same perldoc page, but - wouldn't it be nice if the script automatically adapted itself to the Perl version it ran on? I sure reckoned it'd be.

    What scripts DID the various Perl versions recognize, anyway? What I ended up doing (perhaps there's an easier way) was to look at lib/unicore/Scripts.txt for versions 5.8, 5.10, ..., 5.20 in the Perl git repo (I skipped 5.6 and earlier, because a) the relevant file didn't exist in the tree yet back then, and b) those versions are ancient, anyway). And by "look at", I mean download (as scripts-58.txt etc.), and then process:

    $ for i in 8 10 12 14 16 18 20; do perl scripts.pl scripts-5$i.txt >5$ +i.lst; done $ for i in 8 10 12 14 16 18; do diff --unchanged-line-format= --new-li +ne-format=%L 5$i.lst 5$((i+2)).lst >5$((i+2)).new; done $

    scripts.pl was a little helper script to extract script information (apologies for the confusing terminology, BTW):

    #!/usr/bin/perl use strict; use warnings; use feature qw/say/; my %scripts; while(<>) { next unless m/; ([A-Za-z_]*) #/; $scripts{$1}++; } $, = "\n"; say sort { $a cmp $b } map { $_ = ucfirst lc; $_ =~ s/(?<=_)(.)/uc $1/ +ge; qq/"$_"/ } keys %scripts;

    I admit, I got lazy at this point and manually combined those files (58.lst, as well as 510.new, 512.new etc.) into a hash holding all the information, instead of having a script output it. Nonetheless, once this was done, I could easily load all the right scripts for a given Perl version:

    # New Unicode scripts added in Perl 5.xx my %uniscripts = ( '8' => [ "Arabic", "Armenian", "Bengali", "Bopomofo", "Buhid", "Canadian_Aboriginal", "Cherokee", "Cyrillic", "Deseret", "Devanagari", "Ethiopic", "Georgian", "Gothic", "Greek", "Guja +rati", "Gurmukhi", "Han", "Hangul", "Hanunoo", "Hebrew", "Hiragana", "Inherited", "Kannada", "Katakana", "Khmer", "Lao", "Latin", "Malayalam", "Mongolian", "Myanmar", "Ogham", "Old_Italic", "O +riya", "Runic", "Sinhala", "Syriac", "Tagalog", "Tagbanwa", "Tamil", "Telugu", "Thaana", "Thai", "Tibetan", "Yi" ], '10' => [ "Balinese", "Braille", "Buginese", "Common", "Coptic", "Cuneif +orm", "Cypriot", "Glagolitic", "Kharoshthi", "Limbu", "Linear_B", "New_Tai_Lue", "Nko", "Old_Persian", "Osmanya", "Phags_Pa", "Phoenician", "Shavian", "Syloti_Nagri", "Tai_Le", "Tifinagh", "Ugaritic" ], '12' => [ "Avestan", "Bamum", "Carian", "Cham", "Egyptian_Hieroglyphs", "Imperial_Aramaic", "Inscriptional_Pahlavi", "Inscriptional_Parthian", "Javanese", "Kaithi", "Kayah_Li", "Lepcha", "Lisu", "Lycian", "Lydian", "Meetei_Mayek", "Ol_Chik +i", "Old_South_Arabian", "Old_Turkic", "Rejang", "Samaritan", "Saurashtra", "Sundanese", "Tai_Tham", "Tai_Viet", "Vai" ], '14' => [ "Batak", "Brahmi", "Mandaic" ], '16' => [ "Chakma", "Meroitic_Cursive", "Meroitic_Hieroglyphs", "Miao", "Sharada", "Sora_Sompeng", "Takri" ], '18' => [ ], '20' => [ ], ); (my $ver = $^V) =~ s/^v5\.(\d+)\.\d+$/$1/; my @known_scripts; foreach (keys %uniscripts) { next if $ver < $_; push @known_scripts, @{ $uniscripts{$_} }; } print STDERR "Running on Perl $^V, ", scalar @known_scripts, " scripts + known.\n";

    The number of scripts Perl supports this way WILL increase again soon, BTW. Perl 5.21.1 bumped the supported Unicode version to 7.0.0, adding another bunch of new scripts as a result:

    # tentative! '22' => [ "Bassa_Vah", "Caucasian_Albanian", "Duployan", "Elbasan", "Gra +ntha", "Khojki", "Khudawadi", "Linear_A", "Mahajani", "Manichaean", "Mende_Kikakui", "Modi", "Mro", "Nabataean", "Old_North_Arabia +n", "Old_Permic", "Pahawh_Hmong", "Palmyrene", "Pau_Cin_Hau", "Psalter_Pahlavi", "Siddham", "Tirhuta", "Warang_Citi" ],

    But that's still in the future. For now I just tested this on 5.14.2 and 5.20.0 (the two Perls I regularly use); it worked like a charm. All that was left to do was outputting those statistics:

    print "Found " . scalar keys(%scripts) . " scripts:\n"; print "\t$_: " , $scripts{$_}, " line(s)\n" foreach(sort { $a cmp $b } + keys %scripts);

    (You'll note that in the above two snippets, I'm using print rather than say, BTW. That's intentional: say is only available from Perl 5.10 on, and this script is supposed to be able to run on 5.8 and above.)

    Fed some sample data that I'm sure Perlmonks would mangle badly if I tried to post it, this produced the following output:

    Running on Perl v5.14.2, 95 scripts known. Found 18 scripts: Arabic: 21 line(s) Bengali: 2 line(s) Cyrillic: 12 line(s) Devanagari: 3 line(s) Georgian: 1 line(s) Greek: 1 line(s) Gujarati: 1 line(s) Gurmukhi: 1 line(s) Han: 29 line(s) Hangul: 3 line(s) Hebrew: 1 line(s) Hiragana: 1 line(s) Katakana: 1 line(s) Latin: 647 line(s) Sinhala: 1 line(s) Tamil: 4 line(s) Telugu: 1 line(s) Thai: 1 line(s)

    Problem solved! And not only that, it's futureproof now as well, adapting to additional scripts in my input data, and easily extended when new Perl versions support more scripts, while maintaining backward compatibility.

    What could still be done? Several things. First, I should perhaps find out if there's an easy way to get this information from Perl, without actually doing all the above.

    Second, while Perl 5.6 and earlier aren't supported right now, they could be. Conveniently, the 3rd edition of Programming Perl documents Perl 5.6; the \p{Script=...} syntax for character classes doesn't exist yet, I think, but one could write \p{In...} instead, e.g. \p{InArabic}, \p{InTamil} and so on. Would this be worth it? Not for me, but the possibility is there if someone else ever had the need to run this on an ancient Perl. (Even more ancient Perls may not have the required level of Unicode support for this, though I wouldn't know for sure.)

    Lastly, since the point of this whole exercise was to identify writing systems used for snippets of text, there's room for optimization. Perhaps it would be faster to precompile a regular expression for each script, especially if @lines is very large. Most of the text I'm dealing with is in the Latin script; as such, I should perhaps test for that before anything else, and generally try to prioritize so that lesser-used scripts are pushed further down the list. Since I'm already keeping a running total of how often each script has been seen, this could even be done adaptively, though whether doing so would be worth the overhead in practice is another question, one that could only be answered by measuring.

    But neither speed nor support for ancient Perls is crucial to me, so I'm done. This was a fun little problem to work on, and I hope you enjoyed reading about it.

Mojolicious starting template
1 direct reply — Read more / Contribute
by neilwatson
on Sep 05, 2014 at 14:03

    I like Mojolicious, but it was hard to learn. More than six months later I still feel I'm just scratching the surface. So, what I'm about to offer may not be great, but it is as far as I've come. You'll still need to study the Mojolicious documentation, but you can start with this rather than nothing.

    Template here.

    And a tip of the hat to Sebastian and his team mates, who have answered my novice questions and have been quick to improve the documentation to help newbies like me.

    Neil Watson
    watson-wilson.ca

Commodore disk image processor thingy
5 direct replies — Read more / Contribute
by rje
on Sep 01, 2014 at 19:57
    Dear Perlmonks,

    I wrote a Perl library, and I think it's pretty cool, but I'm also asking your opinions about it - is it worth putting on CPAN, for instance.

    It is a pure-Perl library for handing Commodore disk images. For those needing a refresher, these are digital images of diskettes and hard disks used by Commodore computers in the late 1970s thru the 1980s.

    It's hosted inside my network, behind my modem's firewall, by my Raspberry Pi running a cheapo web server I wrote (also in Perl) specifically for the purpose of serving and manipulating Commodore disk images.

    My library handles D64, D71, D81, D67, D80, D82, and X64 image types. Each format is a little package (about 8k) with data specific to that image. I made them packages, although I could have just used parametric data. These packages are essentially parametric data anyhow, and provide context to a generic engine that knows how Commodore disk images work.

    The library is 140k (includes good POD documentation, which is rare for me) split among about 20 files.

    First, is it worth posting to CPAN. It's awfully specialized. Maybe it would be better just to post it as a tarball on a website (or github?).

    Second, it's been nearly 10 years since I've uploaded to CPAN, and I am intimidated by the process. Yes, I've read the rules, but I'm concerned about uploading 20 related files in one batch. Anyone have any advice beyond what PAUSE has to say?

    Thanks for listening.

Duct taping spam-bot protection to a web forum
1 direct reply — Read more / Contribute
by aitap
on Aug 31, 2014 at 10:24

    There is a free web hosting which offers a third-level domain name and an installation of their proprietary CMS. It is somewhat widely known in the ex-USSR countries. They have "Web 2.0" AJAX interface, a lot of modules for nearly everything, from a simple forum to a web shop, and a primitive read-only API. I happen to be moderating one of such forums. Despite not being popular in terms of human population it has recently gained a lot of popularity among spam-sending robots.

    At first they all were making the same mistake of posting messages with titles equal to their nicknames, and so the first version of bothunter.pl was born. It employed link parsing routines of WWW::Mechanize and reproduced a sniffed AJAX request by some black magic of parsing JavaScript source for variables. Needless to say, soon it broke, both because JavaScript source slightly changed and because bots became slightly smarter, so the moderators went back to deleting bots manually.

    Yesterday I thought: with PhantomJS, I could mimic the browser and click all these AJAX buttons required to ban a user. As for the spam, maybe it's possible to count unique words in a message and warn if some of them is repeated a lot, and a list of stop-words could help, too... Before I started thinking of ways to automatically build a list of stop words from spam messages I realised that I was reiventing the wheel and searched for spam detection engines.

    My first try was Mail::SpamAssassin, because it's written in Perl and I heard a lot of stories about plugging it into other programs. It turned out to be not so easy to make it work with plain text (non-mail) messages, so I searched for alternatives. It is Mail::SpamAssassin, after all. Bogofilter is not written in Perl, but still was easy to plug in my program, thanks to its -T option, and it happily works with plain text without complaining.

    Interfacing with the site was not so easy. Banning a spam robot (click-click-tab-tab-"spam robot"-tab-space-tab-space) exploded into a mess of ->clicking xpath-found elements; at one time the site refused to register my click no matter how I tried, so I had to call the corresponding JS function manually; in the other place of program I find myself logged out, and the only way to get back in is to load the page I'm going to delete, load the login page, log in, then load the to-be-deleted page again. Kludges. Ew.

    So, here it is: the second version of bot hunter Perl program. I hope it won't break as fast as the first one. Or, at least, will be easier to fix.

LED blinking Morse code from Raspberry Pi
4 direct replies — Read more / Contribute
by rje
on Aug 30, 2014 at 11:23
    With the handy Device::BCM2835 package, I tossed together this little package so I could send "morse code" out on an LED I connected to GPIO Pins #5 and #7.

    (FYI: the Raspberry Pi can run a number of Linux systems, all with Perl. I wrote a tiny HTTP server on it which lets me create and serve Commodore disk images, also allowing me to extract and inject files, in Perl of course. It runs behind my firewall...)

'bld' project - signature(SHA1) based replacement for 'make'
4 direct replies — Read more / Contribute
by rahogaboom
on Aug 22, 2014 at 15:14
    NAME bld VERSION bld version 1.0.4 USAGE usage: bld [-h] -h - this message.(exit) ARGUMENTS None OPTIONS bld [-h] -h help message(exit) ENVIRONMENT VARIABLES None RC CONFIGURATION FILES None DESCRIPTION bld(1.0.4) is a simple flexible non-hierarchical program that builds +a single C/C++/Objective C /Objective C++/Assembler target(executable or library(static or share +d)) and, unlike 'make', uses SHA1 signatures(no dates) for building software and GNU cpp for autom +atic header file dependency checking. The operation of bld depends entirely on the construction +of the Bld(bld specification) and Bld.gv(bld global values) files. See the bld.README file. There + are no cmd line arguments or options(except for -h) or $HOME/.bldrc or ./.bldrc files and no envir +onment variables are used. A bld.rm program is provided to clean up the main bld directory +. bld is based upon taking the SHA1 signature of anything that, when ch +anged, would require a re-build of the executable/library. It is not, like 'make', based in + any way on dates. This means that source or header files may be moved about, and if the file +s do not change then nothing needs to, or will, be re-built. bld is not hierarchical; all + of the information to re-build the executable is contained in the Bld(and Bld.gv) file. Bl +d can descend recursively to pick up and build source, however, the specification for this is s +till in the Bld file at the top of the source tree. Complex multi-target projects are built with the use of a Bld.<projec +t> (Bld files and target bld output files) directory, bld.<project>(project source) directory, + bld.<project>(target construction) script, bld.<project>.rm(target and bld.<info|warn|fata +l>.<target> file removal) script, Bld.<project>.gv(project global values) file, bld.<project>.i +nstall(target and file install) script and bld.<project>.README(project specific documentati +on) file. Current example projects: git - the git project http://git-scm.com/ svn - the subversion project http://subversion.apache.org/ systemd - the systemd project http://www.freedesktop.org/wiki/Sof +tware/systemd/ example - misc examples intended to show how to create Bld and Bl +d.gv files To understand the bld'ing of a single target the following should be +understood: ~/bld directory files: bld - the bld perl script bld.rm - script to clean the bld directory bld.README - for first point of contact quick start Bld - the bld file which controls bld and the construction + of a target Bld.gv - the file of global values imported into the Bld file (unusually used only for multi-target builds) Bld.sig - the signature(SHA1) file created from the Bld file bld.info - information about the bld bld.warn - warnings from the bld bld.fatal - the fatal msg that ended the bld The Bld file(and Bld.gv) controls the entire target bld. The Bld fil +e is divided into three sections - Comments, EVAL and DIRS: ################################################## Comments - Add comments before the EVAL line EVAL # add perl variables to be used in the DIRS section 'dir' or {cmds} p +arts. six are mandatory. $PROJECT = "git"; $CC = "clang"; $INCDIR = "test/include"; # optional variables e.g. $CCOPT = "-O"; # mandatory defined variables # the target to build e.g. executable, libx.a, libx.so $bld="exec-c"; # cmd used in perl system() call to build $bld target - requires +'$bld'(target) and '$O'(object files) internally $bldcmd = "$CC $INCDIR -lm -o \$bld \$O"; # space separated list of directories to search for libraries $lib_dirs = "example/lib /usr/lib /lib /usr/local/lib"; # use system header files in dependency checking("system" or "nos +ystem") $opt_s = "system"; # inform about any files that will require re-building, but do no +t rebuild("rebuild" or "norebuild") $opt_r = "rebuild"; # do dependency checking on libraries("nolibcheck", "libcheck", " +warnlibcheck" or "fatallibcheck") $opt_lib = "fatallibcheck"; DIRS # '{cmds}' cmd blocks e.g. {ls;pwd;date;} # or # '[R] dir:regex:{cmds}' specifications e.g. bld.$PROJECT/git-2.3.0/: +^zlib\.c$:{gcc $INCDIR -o zlib.o $s} # see examples below in 'Bld FILE FORMAT AND EXAMPLES'. both 'dir' a +nd '{cmds}' fields below may have EVAL # defined variables interpolated into them. {cmds} [R] dir:regex:{cmds} {cmds} [R] dir:regex:{cmds} [R] dir:regex:{cmds} ... ... ################################################## The Bld file has three sections, a starting comments section to docum +ent the Bld, an EVAL section to define variables(and Bld.gv defined variables read in at t +he beginning of the EVAL section) for interpolation into DIRS section 'dir' and '{cmds}' +fields and a DIRS section that defines either {cmd} blocks or '[R] dir:regex:{cmds}' sp +ecifications. The entire EVAL section is eval{}'ed in bld. Any errors will terminate t +he run. There are six mandatory variables(see above). The {cmd} blocks just execute a g +roup of shell cmds. The '[R] dir:regex:{cmds}' specifications are used to build source, l +ocated in directories below the bld directory. These specifications are composed of three f +ields: [R] dir - a directory with possibly variables to be interpolated +and [R] if a recursive search for source is to be made. regex - a perl regular expression to match sources to be built +by the {cmds} field. if [R] is specified then this same regex will be applied t +o every directory recursively searched. the same {cmds} are applied to e +very source found. {cmds} - a group of cmds to construct all the source files selec +ted by the regex field. {cmds} will be executed successively for each source ma +tched. the cmds should have at least one '$s' variable specified. each of the + matched source files will be substituted for '$s'. Rebuilds will happen if: 1. a source file is new or has changed 2. the corresponding object file is missing or has changed 3. the command that is used to compile the source has changed 4. a dependent header file has been added/deleted or changed 5. the command to link the executable or build the library archiv +e has changed 6. the executable or library has changed or is missing The execution of bld will produce four files: Bld.sig The Bld.sig file, holds the source/object/header/executable/l +ibrary file names and corresponding signatures and build cmd signatures used to det +ermine if a source should be re-built. System header files and libraries may optionall +y be included. bld.info Contains detailed information about the the stages of the bui +ld. 1. date/time, OS version, compiler version, cpp version etc. 2. comments section of Bld file 3. EVAL section expansion of $bld, $bldcmd and $lib_dirs mand +atory variables($O is object files) 4. Bld file DIRS section specification lines with irrelevant +white space compressed out 5. R recursively expanded and numbered DIRS section specifica +tion lines 6. a. DIRS section specification lines b. variable interpolated(except for $s) specification +line cmd field c. matching compilation unit source file(s) d. source file header dependencies 7. EVAL section expansion of defined variables used in DIRS s +ection cmd fields 8. List of all build - System headers(full path) User headers(relative path) System libraries(full path) User libraries(relative path) Source files(relative path) Build target(bld directory) bld.warn Warnings about the build e.g. multiple copies of the same head +er files in different directories, multiple '$s' variables specified in DIRS line c +ommand field, same source file specified in more than one directory, EVAL de +fined variables not used in DIRS cmds. Each warning will include the package, + filename and source line etc. bld.fatal Fatals terminate the build e.g. Same source file \'$basename\' + specified in same directory, extra unused variable(s) in DIRS section etc. Thes +e files should be empty if everything built correctly. The six mandatory EVAL defined variables are: 1. $bld # the target to build e.g. executable, libx.a, libx.so $bld="exec-c"; 2. $bldcmd # cmd used in perl system() call to build $bld target - requi +res '$bld'(target) and '$O'(object files) internally $bldcmd = "$CC $INCDIR -lm -o \$bld \$O"; 3. $lib_dirs # space separated list of directories to search for libraries $lib_dirs = "example/lib /usr/lib /lib /usr/local/lib"; 4. $opt_s # use system header files in dependency checking("system" or +"nosystem") $opt_s = "system"; 5. $opt_r # inform about any files that will require re-building, but d +o not rebuild("rebuild" or "norebuild") $opt_r = "rebuild"; 6. $opt_lib # do dependency checking on libraries("nolibcheck", "libcheck +", "warnlibcheck" or "fatallibcheck") $opt_lib = "fatallibcheck"; To understand the building of complex multi-target projects the follo +wing directories and files should be understood: ~/bld directories: Bld.<project>/<version> - has all files controlling <project> <ve +rsion>s blds and bld target output files bld.<project>/<version> - source code for <project> <version>s aux - template scripts directory for <project +> blds ~/bld/aux files: aux/bld.<project> - template copied to Bld.<project>/<versi +on> directories to bld multi-target projects aux/bld.<project>.rm - template copied to Bld.<project>/<versi +on> directories to clean multi-target projects ~/bld/Bld.<project>/<version> files: bld.<project> - for initiating single target, mul +ti-target or all target blds of a <project> bld.<project>.rm - for initiating single target, mul +ti-target or all target clean of a <project> bld.<project>.targets - list of all <project> targets bld.<project>.README - <project> README bld.<project>.install - <project> install script bld.<project>.script.<script> - scripts called by the Bld.<projec +t>.<target> files Bld.<project>.<target> - the Bld file for each <project> < +target> Bld.gv.<project> - global values imported into all B +ld.<project>.<target> files Bld.sig.<project>.<target> - the signature(SHA1) file for each + <project> <target> bld.info.<project>.<target> - the bld.info file for each <proje +ct> <target> bld.warn.<project>.<target> - the bld.warn file for each <proje +ct> <target> bld.fatal.<project>.<target> - the bld.fatal file for each <proj +ect> <target> The previous description material focused on the construction of a si +ngle target. This requires only the use of the bld program and Bld file. To bld complex multi-target + projects requires use of some standard directory structure conventions, several programs for bld'in +g targets and cleaning project directories and directory/file naming conventions. For example, usin +g git-2.3.0 and a <project> name of 'git': Bld.git/git-2.3.0 All of the target Bld files and bld run output files are here +. During development the initiation of target re-blds or full project re-blds are initiated from +here. bld.git/git-2.3.0 The git-2.3.0.tar.gz file is unzipped here and ./configure is + run against the git project source code. aux Template scripts for use in the Bld.git/git-2.3.0 directory. + The bld.<project> script is copied to Bld.git/git-2.3.0 as bld.git. The bld.<project>.rm is cop +ied to Bld.git/git-2.3.0 as bld.git.rm. No changes are required to the scripts code. aux/bld.git The bld.<project> file would be copied to Bld.git/git-2.3.0 a +s bld.git. There is no need to change the file; the file name is picked up internally. This is the + main project bld script. It is executed either with a sequence of target names or with '--al +l' for building all targets. Do 'perldoc bld.git' for the man page. aux/bld.git.rm The bld.<project>.rm file would be copied to Bld.git/git-2.3. +0 as bld.git.rm. There is no need to change the file; the file name is picked up internally. This script + is used to remove build files related to specific targets or clean files related to all targets. It i +s executed either with a sequence of target names or with '--all' for cleaning all targets. For example, + if test-wildmatch is used as a target then the following files will be removed: bld.fatal.git.test-wildmatch bld.info.git.test-wildmatch bld.warn.git.test-wildmatch Bld.sig.git.test-wildmatch test-wildmatch The target Bld file, Bld.git.test-wildmatch, will not be remo +ved. Do 'perldoc bld.git.rm' for the man page. bld.git.targets All the project targets go here, one to a line. The targets +will be built in order from top to bottom. Make sure than libraries that are required by subsequent targ +ets are built first. The file may have any number of blank or comment lines. bld.git.README The bld.git.README file provide getting started and contact i +nformation. bld.git.install For installing the project. This most likely will be a scrip +t to create an installable package. At present none are populated. bld.git.script.<script> These are scripts to call from the Bld files that do various +functions required for target construction. There may be any number of these. The git pro +ject did not require any. Bld.git.<target> The Bld file for a specific target. Bld.gv.git The project global values file. This file is imported at the + beginning of every project target Bld file. It has no structure. It contains defined variable +s that are common to all project Bld files e.g. $CC. Bld.sig.git.<target> bld.info.git.<target> bld.warn.git.<target> bld.fatal.git.<target> See above for a description of these files. See BUILDING MULTI-TARGET PROJECTS below for the detail steps in +building complex multi-target projects. FEATURES AND ADVANTAGES 1. Everything is done with SHA1 signatures. No dates are used anywh +ere. Signatures are a property of the file and not meta data from the system used for the build. Any t +ime issues, whether related to local clocks, networked host clocks or files touched by command activit +ies are eliminated. Modern signature algorithms are strongly randomized even for small file changes - +for the 160 bit SHA1 hash collisions are unlikely in the extreme. The Digest::SHA module is fast. The ex +pense of signature calculation times is small relative to the expense of programmer time. An investig +ation of some other make alternatives e.g. scons, cook - will disclose that they too are using signatur +es - maybe for exactly for the same reasons. 2. bld is REALLY simple to use. There are no arguments, no options( +except -h), no environment variables and no rc files. The entire bld is controlled from the Bld(and Bld.g +v file) file. Only a minimal knowledge of perl is needed - variable definitions and simple regular expre +ssions. 3. Automatic dependency checking - GNU cpp is used to find the heade +r file dependencies. Optionally, header file checking may be done for user header files only or for simul +taneously both system header and user header files. All header file dependency information associated +with each source is saved to the bld.info file. 4. There are no built in dependency rules. The Bld file DIRS sectio +n specifications give what is to be built from what and the Bld file EVAL section gives how to assemb +le all the components for the target. 5. bld is not hierarchical. A single Bld file controls the construc +tion of a single target(a target is an executable or library(static or shared)). Complex multi-target p +rojects use one Bld.gv(global values) file and many Bld files - one to a target. The source directory +structure goes under bld.<project>/<version> and each target Bld file(Bld.<project>.<target>) encapsulates all + the build information for all the source directories under bld.<project>/<version>. All the built +targets and build information files go into the Bld.<project>/<version> directory. See "'make' and it's + difficulties:" below for reasons why recursive make causes problems. 6. Each source file will have three signatures associated with it - +one for the source file, one for the corresponding object file and one for the cmds use to re-build th +e source. A change in any of these will result in a re-build. A change in the target signature will resu +lt in a re-build. Optionally, the signatures of dynamic libraries may be tracked. If a library sig +nature changes the bld may warn or stop the re-build. If dynamic libraries are added or deleted from the + bld this can ignore/warn/fatal. 7. If any files in the bld have the same signature this is warned ab +out e.g. two header or source files of the same or different names. 8. Complex multi-target projects are built with a standard directory + setup and a standard set of scripts - Directories: Bld.<project>/<version> - has all files controlling <pr +oject> <version>s blds and bld target output files bld.<project>/<version> - source code for <project> <ve +rsion>s Files: bld.<project> - for initiating single target, + multi-target or all target blds of a <project> bld.<project>.rm - for initiating single target, + multi-target or all target clean of a <project> bld.<project>.targets - list of all <project> targets bld.<project>.README - <project> README bld.<project>.install - <project> install script bld.<project>.script.<script> - scripts called by the Bld.<pr +oject>.<target> files Bld.<project>.<target> - the Bld file for each <projec +t> <target> Bld.gv.<project> - global values imported into a +ll Bld.<project>.<target> files 9. Security - since the signatures of everything(source, objects, li +braries, executable) are checked it is more difficult to insinuate an exploit into a source, object, lib +rary or executable during the build process. 10. The capture of the full build process in the bld.info, bld.warn a +nd bld.fatal files allows easy access to and saving of this information. For multi-target projects with t +he target names appended to these files it allows quick investigation of the build process of many interr +elated targets at the same time. 11. Perl - since bld is all perl and since all warnings and fatals ha +ve the source line number associated with them, it is very easy to locate in the source code the exact loca +tion of an error and examine the context about which the error occurred and routine that the error was pro +duced in. 12. Time - programmer time; learning about, maintaining/debugging Mak +efiles and Makefile hierarchies, dependency checking integration and formulation of Makefile strategies, auto +matic Makefile generation with Autotools - these all dominate the programmer time and expense of 'make'. bl +d only requires basic perl variables(in the Bld file EVAL section) and '[R] dir:regex:{cmds}' line specif +ications(in the Bld file DIRS section). 13. 'make' and it's difficulties: http://www.scons.org/wiki/FromMakeToScons a detailed critique of make and some alternatives(Adrian +Neagu) http://freecode.com/articles/what-is-wrong-with-make What is Wrong with Make?(Adrian Neagu) http://www.scons.org/architecture/ a description of the scons architecture and in particular + the reasons for the use of signatures instead of dates ftp://ftp.gnu.org/old-gnu/Manuals/autoconf/html_mono/autoconf +.html#SEC3 a brief critique of make and how GNU automake from the GN +U Build System contributes http://aegis.sourceforge.net/auug97.pdf an article "Recursive Make Considered Harmful" by Peter M +iller from the Australian UNIX Users Group http://www.conifersystems.com/whitepapers/gnu-make/ an in depth critique of make http://www.conifersystems.com/whitepapers/gnu-make/ What's Wrong With GNU make? http://www.leosimons.com/2006/bettermake/what-is-a-better-mak +e.html What is a better make? http://www.leosimons.com/2006/a-better-make.html Building a better (make|ant|maven|...) DEPENDENCIES I use Fedora for development on an x86_64 architecture. Yum is th +e package installation tool. Required for execution: experimental.pm(3pm) - for smartmatch and switch features cpp(1) - gnu cpp cmd is required for dependency determination ldd(1) - used for library dependency determination Do: cpan install cpanm cpanm experimental.pm Required for test: gcc(1)/g++(1) (http://gcc.gnu.org/) clang(1) (http://llvm.org/) yacc(1)/flex(1) Some variation might be required on(for git, svn, systemd test): yum install gcc.x86_64 yum install gcc-c++.x86_64 yum install gcc-objc.x86_64 yum install gcc-objc++.x86_64 yum install libobjc.x86_64 yum install gnustep-base.x86_64 yum install gnustep-base-devel.x86_64 yum install clang yum install byacc yum install flex yum install zlib-devel.x86_64 yum install libxdiff.x86_64 yum install apr.x86_64 yum install apr-util.x86_64 yum install apr-util-devel.x86_64 yum install sqlite-devel.x86_64 yum install intltool yum install libcap-devel yum install gtk-doc yum install lzma.x86_64 yum install libmount-devel.x86_64 If ./configure generates the following msg about missing requireme +nts: "configure: error: Package requirements (foo) were not met: No package 'foo' found Consider adjusting the PKG_CONFIG_PATH environment variable if + you installed software in a non-standard prefix." Do: yum install "pkgconfig(foo)" Also check: http://who-t.blogspot.com/2014/05/configure-fails-with +-no-package-foo.html If the downloaded project code does not have a configure script th +en check for a configure.ac file and run 'autoconf' with no arguments or options. This will g +enerate a configure script. PROJECT STATE 1. The git, svn and systemd projects need work. I ran ./configure be +fore each bld. I used no options. How options affect the generated code and thus the Bl +d files is important. Anyone willing to investigate configure options and how these opti +ons affect the Bld files is welcome. 2. The bld.<project>.install scripts all need to be done. These scri +pts need to build the the packages that 'yum install <package>' would install or possibl +y packages tailored to other systems. I'd prefer to partner with someone knowledgeable a +bout the installation of git, svn and systemd. 3. All the Bld.gv.<project> files should be vetted by a <project> kno +wledgeable builder. 4. The git, svn and systemd projects will all be creating new version +s eventually. Anyone that would like to add bld.<project>/<version> and Bld.<project>/< +version> directories with the new versions is welcome. 5. I need someone with substantial experience building the linux kern +el to advise me or partner with me on the construction of 4.0 or later. 6. If you successfully bld a new project and wish to contribute the b +ld, please do so. I'm interested in how others construct/organize/document/debug project +s and their Bld files. QUICK START 1. Bld'ing the systemd project - http://www.freedesktop.org/wiki/Soft +ware/systemd/ a. cd Bld.systemd/systemd-208 # puts you into the systemd(systemd- +208) project directory b. ./bld.systemd --all # bld's all of the systemd targets a +nd bld target output files - the bld.info.systemd.<target>, the bld.warn.systemd.<target>, the bld.fatal.systemd.<target> +, files c. ./bld.systemd.rm --all # cleans up everything 2. Bld'ing the svn project - https://subversion.apache.org/ a. cd Bld.svn/subversion-1.8.11 # puts you into the svn(subversion +-1.8.11) project directory b. ./bld.svn --all # bld's all of the svn targets and + bld target output files - the bld.info.svn.<target>, the bld.warn.svn.<target>, the bld.fatal.svn.<target>, files c. ./bld.svn.rm --all # cleans up everything 3. Bld'ing the git project - http://www.git-scm.com/ a. cd Bld.git/git-2.3.0 # puts you into the git(git-2.3.0) project + directory b. ./bld.git --all # bld's all of the git targets and bld tar +get output files - the bld.info.git.<target>, the bld.warn.git.<target>, the bld.fatal.git.<target>, files c. ./bld.git.rm --all # cleans up everything 4. Bld'ing any single target a. cd bld # the main bld directory - cd here when you unpack + the bld.tar.xz file b. Install the source code in a sub-directory of the bld directory c. Create a Bld file - the Bld file entirely controls the target b +ld - see example below d. ./bld -h # the bld usage msg e. ./bld # do the bld f. ./bld.rm # clean up g. vi Bld.sig # examine the bld signature file h. vi bld.info # detailed info about the stages of the bld i. vi bld.warn # warning msgs from the bld j. vi bld.fatal # fatal msgs that terminated the bld - should be e +mpty if bld is successful Bld FILE FORMAT AND EXAMPLES A more detailed description of the Bld(and Bld.gv) files initially pr +esented in the DESCRIPTION section above is presented here. 1. A comments section 2. An EVAL(starts a line) section - this is perl code that is eval'ed + in bld. Six variables are required. These are: e.g. EVAL # mandatory defined variables # the target to build e.g. executable, libx.a, libx.so $bld="exec-c"; # cmd used in perl system() call to build $bld target - re +quires '$bld'(target) and '$O'(object files) internally $bldcmd = "$CC -lm -o \$bld \$O"; # space separated list of directories to search for librar +ies $lib_dirs = "example/lib /usr/lib /lib /usr/local/lib"; # use system header files in dependency checking("system" +or "nosystem") $opt_s = "system"; # inform about any files that will require re-building, bu +t do not rebuild("rebuild" or "norebuild") $opt_r = "rebuild"; # do dependency checking on libraries("nolibcheck", "libch +eck", "warnlibcheck" or "fatallibcheck") $opt_lib = "fatallibcheck"; Any other simple perl variables can be defined in the EVAL sec +tion and used in the DIRS section. Environment variables may be set. 3. A DIRS(starts a line) section - this section will have either {cmd +s} cmd blocks or '[R] dir:regex:{cmds}' specifications. The {cmds} blocks are just a group of shell cmds, always executed. + A dir specification is a source directory relative to the bld directory. The regex specification is a perl regular e +xpression that will pick up one or more of the source files in dir. The {cmds} specification describes how to bu +ild the selected source files. Any number of cmds, ';' separated, may be specified within the {} brackets. Example Bld Files: Simplest(Bld.example/example/Bld.example.helloworld-c): The 'Hello World!' program with only the minimal required defi +nitions. Comments EVAL $CC = "gcc"; # mandatory defined variables # the target to build e.g. executable, libx.a, libx.so $bld="helloworld-c"; # cmd used in perl system() call to build $bld target - re +quires '$bld'(target) and '$O'(object files) internally $bldcmd = "$CC -o \$bld \$O"; # space separated list of directories to search for librar +ies $lib_dirs = "/usr/lib /lib /usr/local/lib"; # use system header files in dependency checking("system" +or "nosystem") $opt_s = "system"; # inform about any files that will require re-building, bu +t do not rebuild("rebuild" or "norebuild") $opt_r = "rebuild"; # do dependency checking on libraries("nolibcheck", "libch +eck", "warnlibcheck" or "fatallibcheck") $opt_lib = "warnlibcheck"; DIRS bld.example/example : ^helloworld\.c$ : { $CC -c $s; } Complex(Bld.example/example/Bld.example.exec-c): A well commented example of all of the features of a Bld file. + The code routines are all just stubs designed to illustrate a Bld file. Comments EVAL # this section will define perl variables to be interpolated i +nto DIRS section cmd fields # the compiler $CC = "clang"; # mandatory defined variables # the target to build e.g. executable, libx.a, libx.so $bld="exec-c"; # cmd used in perl system() call to build $bld target - re +quires '$bld'(target) and '$O'(object files) internally $bldcmd = "$CC -lm -o \$bld \$O"; # space separated list of directories to search for librar +ies $lib_dirs = "example/lib /usr/lib /lib /usr/local/lib"; # use system header files in dependency checking("system" +or "nosystem") $opt_s = "system"; # inform about any files that will require re-building, bu +t do not rebuild("rebuild" or "norebuild") $opt_r = "rebuild"; # do dependency checking on libraries("nolibcheck", "libch +eck", "warnlibcheck" or "fatallibcheck") $opt_lib = "fatallibcheck"; # some examples of variables that will be interpolated into DI +RS section cmd fields $INCLUDE = "-I bld.example/example/include"; $LSOPTIONS = "-l"; # "a" or "b" to conditionally compile main.c $COND = "a"; DIRS # this section will have either {cmds} cmd blocks or '[R] dir: +regex:{cmds}' specifications # example of use of conditional compilation bld.example/example/C : ^main\.c$ : { # can have comments here too if [ "$COND" == 'a' ]; then $CC -S $INCLUDE $s; fi if [ "$COND" == 'b' ]; then $CC -O4 -S $INCLUDE $s; fi } # example of execution of a bare block of cmds - '{' and '}' m +ay be on separate lines { ls $LSOPTIONS; } # the cmd field may be put on another line(s) and indented bld.example/example/C : ^g\.x\.C$ : { $CC -c $INCLUDE $s; } # all three fields - dir, regex and cmd - may be put on separa +te lines(even with extra blank lines). # directories may have embedded blanks('a b'). bld.example/example/C/a b : ^m\.c$ : {$CC -c $INCLUDE $s;} # example of regex field that captures multiple source files(h +.c and i.c) and example of a # cmd field with multiple cmds - white space is irrelevant(a c +hange should not cause a re-build) # example of cmd fields with multiple cmds(ls and $CC) bld.example/example/C : ^(h|i)\.c$ : { ls -l $s; $CC +-c $INCLUDE $s; } # example of assembler source # Note: the $CC compile produces .o output by changing the c t +o an o. # the as output needs to be specified by the -o option. bld.example/example/C : ^main\.s$ : {as -c -o main.o $s;} # are applied to all subdirectories of the specified dir field +(right after the 'R') R bld.example/example/C/y : ^.*\.c$ : {$CC -c $INCLUDE $s;} bld.example/example/C/x : ^t\.c$ : {$CC -c $INCLUDE $s;} bld.example/example/C/z : ^(w|w1)\.c$ : {$CC -c $INCLUDE +$s;} # cmd blocks may execute multiple cmds(ls and pwd) { ls -lfda; pwd; ls; } FILES ~/bld directory files: bld - the bld perl script bld.rm - script to clean the bld directory bld.README - for first point of contact quick start Bld - the bld file which controls bld and the construction of +a target Bld.gv - the file of global values imported into the Bld file(unu +sually used only for multi-target builds) Bld.sig - the signature(SHA1) file created from the Bld file bld.info - information about the bld bld.warn - warnings from the bld bld.fatal - the fatal msg that ended the bld ~/bld directories: Bld.<project>/<version> - has all files controlling <project> <versio +n>s blds and bld target output files bld.<project>/<version> - source code for <project> <version>s aux - template scripts directory for <project> bl +ds ~/bld/aux files: aux/bld.<project> - template copied to Bld.<project>/<version> +directories to bld multi-target projects aux/bld.<project>.rm - template copied to Bld.<project>/<version> +directories to clean multi-target projects ~/bld/Bld.<project>/<version> files: bld.<project> - for initiating single target, multi-t +arget or all target blds of a <project> bld.<project>.rm - for initiating single target, multi-t +arget or all target clean of a <project> bld.<project>.targets - list of all <project> targets bld.<project>.README - <project> README bld.<project>.install - <project> install script bld.<project>.script.<script> - scripts called by the Bld.<project>.< +target> files Bld.<project>.<target> - the Bld file for each <project> <targ +et> Bld.gv.<project> - global values imported into all Bld.< +project>.<target> files Bld.sig.<project>.<target> - the signature(SHA1) file for each <pr +oject> <target> bld.info.<project>.<target> - the bld.info file for each <project> +<target> bld.warn.<project>.<target> - the bld.warn file for each <project> +<target> bld.fatal.<project>.<target> - the bld.fatal file for each <project> + <target> BUILDING SINGLE TARGETS (a target is an executable or library(static or shared)) 1. Construct the Bld file - see below for a Bld file example and s +ee the Bld.example directory for multiple examples. A Bld.gv file is not needed f +or a single target. Since there are no args or options to bld and no environment va +riables or rc files are used, nothing else needs to be done. 2. Execute './bld'. This will rebuild the target and create/updat +e the Bld.sig signature file. The bld.info, bld.warn and bld.fatal files will be creat +ed. 3. Use './bld.rm' to clean the bld directory. BUILDING MULTI-TARGET PROJECTS (a target is an executable or library(static or shared)) 1. Pick a name for the project e.g. git, svn, systemd. 2. In the main bld directory(the location of the bld script) creat +e a bld.<project> directory. Create another directory, bld.<project>/<version>, +that describes the version of the code it will hold e.g. systemd-208. Any number +of these version directories may be created to maintain different versions of th +e code. Unpack the source code in the version directory e.g. bld.systemd/systemd-2 +08/<systemd-208 src code>. 3. Create a Bld.<project> directory. Create another directory, Bl +d.<project>/<version>, that describes the version of the code it will maintain e.g. sy +stemd-208. There should be one version directory for each version of the code be +ing maintained. These directories will hold: a. all of the Bld.<project>.<target> and Bld.gv.<project> f +iles that control the construction of each project target b. all of the target bld output files: bld.info.<project>.<target>(information describing t +he bld) bld.warn.<project>.<target>(bld warning msgs) bld.fatal.<project>.<target>(bld fatal msgs) c. the bld scripts and script control files d. all of the targets 4. Create a Bld.<project>.<target> file for each project target. +These control the bld for each and only that target. If there are variables that may + be defined globally over the entire project then set them in the Bld.gv.<project>(o +ne per project) file. The Bld.gv.<project> file will be included in each Bld.<project +>.<target> file before it is evaluated. All Bld.<project>.<target> files require six +variables to be defined. These are: e.g. # mandatory defined variables # space separated list of directories to search for librari +es $lib_dirs = ""; # use system header files in dependency checking("system" o +r "nosystem") $opt_s = "nosystem"; # inform about any files that will require rebuilding, but +do not rebuild("rebuild" or "norebuild") $opt_r = "rebuild"; # do dependency checking on libraries("libcheck", "nolibche +ck", "warnlibcheck" or "fatallibcheck") $opt_lib = "nolibcheck"; # the target to built e.g. executable, libx.a, libx.so $bld="accelerometer"; # cmd used in perl system() call to build $bld target - req +uires '$bld'(target) and '$O'(object files) internally $bldcmd = "$CC $LIBSLINKOPTS -o \$bld \$O -LBld.systemd/sys +temd-208 -ludev -lsystemd-shared -ludev-core -lm -lrt -ldl"; The definition of these variables may be spread between the Bld +.<project>.<target> and the imported Bld.gv.<project> files. See the Bld FILE FORMAT section below for detailed instructions + on Bld file construction. 5. Create a bld.<project>.targets file with all of the project tar +get names, one to a line. e.g. # each line is a valid target argument to bld.<project> e.g +. './bld.<project> target' # and will build one(that named) library or executable. th +e order of target bld's # is important. if 'bld.<project> --all' is used to bld al +l targets then dependencies # must be built first, that is, libraries that executables +depend on must be built at # the start. # libsystemd-shared.a # libsystemd-login.so # systemd-cgls ... The order is important; if an executable depends on a library t +hen build the library first. 6. Copy the 'bld.<project>' and 'bld.<project>.rm' files from the +aux directory to the source code directory and rename them for that particular project. These s +cripts use the <project> part of the script name for building output files. 7. Run the ./bld.<project> script with the --all option or one or +more target names from the bld.<project>.targets file. This will bld all targets or the selected targets. The: bld.info.<project>.<target>(information describing the bld) bld.warn.<project>.<target>(bld warning msgs) bld.fatal.<project>.<target>(bld fatal msgs) files will be created. Examine these for the results of each t +arget bld. All of the bld.fatal.<project>.<target> files should be empty if everything bld's OK. 9. Run './bld.<project> --all' again to rebuild. If everything bu +ilt successfully the first time then this run will indicate that everything is up to date. 8. Use './bld.<project>.rm [--all] [target, target, ...]' to clean + up. 9. Use './bld.<project>.install' as root to install the project. 10. Examine the ./bld.<project>.README for project specific inform +ation. NOTES 1. bld assumes that a source will build a derived file e.g. .o files +in the same directory and have the same root name as the source. a subsequent Bld file {} b +lock can rename and move the .o files. 2. bld assumes that all targets in multi-target bld's will be uniquel +y named - all targets go into the same project directory before distribution for installati +on. installation can then rename and distribute targets in any way desired. 3. Some projects(rarely) violate either or both of these target namin +g or object file naming/location requirements, but reconstructing these projects wi +th bld should be relatively easy e.g. systemd. 4. bld executes cmd fields({}) in the bld directory and then moves al +l created files to the source directory. 5. A bld run may be ^C interrupted. When a normal uninterrupted run +is completed the Bld.sig file is rewritten using only those files that were include +d in the build. Thus, if files were added or deleted this would reflect in the new + Bld.sig file. If a run is interrupted, all files read in from the original Bld.sig fi +le and any new files already built are written out to the Bld.sig file. This ensures t +hat new files already built and old files not yet examined will not be re-built. This m +ay result in some entries in the Bld.sig file that no longer exist, but this will be + corrected at the end of the next normally completed run. 6. The Bld.sig signature file is automatically created and updated. +It contains one line for each source, one line for each header file, optionally one lin +e for each library and one line for the executable. The header file lines and the execut +able line have two fields: the file name and its signature. The source lines have fo +ur fields: the file name, the signatures of the source file, the command use to build +the source, and the object file. The user can modify this file to force the re-build +of files by altering the signature or even by deleting a line, however, any modificatio +n to a source or header file, or build command string will do the same thing. Removing th +e Bld.sig file will re-build the entire target. 7. A non C or C++ source will be re-built if its build command has be +en changed or the source file itself has been changed. The re-built output will be +put back in the directory where the source came from. The assumption is that it wi +ll be a file that will then act a source for subsequent build steps e.g. a lex or yacc th +at will produce a C file as output which will then later need to be compiled. Bld fil +e cmd({}) blocks can modify this. CALL TREE Calls to imported functions from public CPAN modules are omitted. Ca +lls to some bld coded routines are omitted: opt_help() system_error_msg() warning() fatal() sourcesort() bld Bld_section_extract() init_blddotinfo() dirs_pro() cvt_dirs_to_array() expand_R_specification() accum_blddotinfo_output() var_sub() var_sub() var_sub() variable_match() var_sub() var_sub() var_sub() read_Blddotsig() var_sub() var_sub() src_pro() file_sig_calc() buildopt(); tgtextorfile(); hdr_depend() file_sig_calc() rebuild_src_bool() tgt_signature() rebuild_target_bool() file_sig_calc() file_sig_calc() rebuild_exec() file_sig_calc() file_sig_calc() multiple_sigs() sig_file_update() DIAGNOSTICS Diagnostics are either warnings(calling warning($msg)) or fatals(call +ing fatal($msg)). Warnings and fatals are exclusive; specific conditions cause warnings + and specific other conditions cause fatals. There is no overlap. Warnings are co +nditions that may be of interest e.g. same header file in multiple locations or multipl +e -c specifications in a source compilation cmd, but which should not impact the full bui +ld of the project. Fatals are conditions that either will necessarily force the terminat +ion of the build e.g. a source file compile failure, or that indicate the construction + of the Bld file has some instructions that are contradictory or missing necessary ele +ments e.g. same source file matched twice in Bld file DIRS section specifications. W +arnings are written to the bld.warn file only. Fatals are written identically to standar +d out and the bld.fatal file. Warnings all start with 'WARNING:' and fatals all st +art with 'FATAL:'. There are no other types of diagnostic msgs. INCOMPATIBILITIES None Known BUGS AND LIMITATIONS None Known SEE ALSO Do: perldoc 'bld.<project>' perldoc 'bld.<project>.rm' 'make' and it's difficulties - in FEATURES AND ADVANTAGES above GITHUB RELEASES https://github.com/rahogaboom/bld bld-1.0.4.tar.gz - entirely perldoc updates. example projects - +bld-1.0.4-git.tar.xz, bld-1.0.4-svn.tar.xz and bld-1.0.4-systemd.tar.xz. bld-1.0.3.tar.gz - moved four more routines from bld to BldRoutin +es.pm. added new project version builds for git(git-2.3.0.tar.gz) and svn(subversion-1.8.1 +1.tar.gz). improved error reporting. multiple versions of git and svn are now built. the na +ming convention for example project release files has changed - bld-1.0.3-git.tar.xz, bld- +1.0.3-svn.tar.xz and bld-1.0.3-systemd.tar.xz. bld-1.0.2.tar.gz - added capability to set Bld file EVAL section +variables for interpolation into DIRS section directory specifications. improved fatal() ou +tput. updated git/svn/systemd builds to Fedora21. doc improvements. improved exports +/imports of variables/functions between bld/BGC.pm/BldRoutines.pm. Why are there no releases beyond the latest three? For now, I on +ly intend to maintain and answer questions about the most recent releases. This may change in future. AUTHOR Richard A Hogaboom richard.hogaboom@gmail.com LICENSE and COPYRIGHT and (DISCLAIMER OF) WARRANTY Copyright (c) 1998-2014, Richard A Hogaboom - richard.hogaboom@gmail.c +om All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are + met: * Redistributions of source code must retain the above copyright notic +e, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright no +tice, this list of conditions and the following disclaimer in the document +ation and/or other materials provided with the distribution. * Neither the name of the {organization} nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "A +S IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PUR +POSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUEN +TIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOOD +S OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOW +EVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIA +BILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF +THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. : +-)
DBIx::Class recursive subquery construct
No replies — Read more | Post response
by maruhige
on Aug 15, 2014 at 14:38

    Hello Monks,

    My first encounter with DBIx::Class's subquery construct went so well (and my need of n subqueries so great) that I tried a programmatic way of creating nested subqueries, shared here.

    Context is simply a tag search function, where all the tags are in one table, with an intermediate 2 integer column table to the post table. Tags are retrieved from the tag table in ascending order of weight to ensure the smallest possible start set, getting progressively smaller.

    sub recursive_subquery{ my($schema,$join_column,$key,$values) = @_; my ($value1) = shift(@$values); my $sub_query =$schema->search({ $key => $value1, }); for(@$values){ my $tmp_query = $schema->search({ $join_column => { -in => $sub_query->get_column($join_column)->as_query }, $key => $_, }); $sub_query = $tmp_query; } return $sub_query; }

    And a short example of using it manually in Catalyst:

    $c->stash->{rows} = $c->model('DB::Post')->search({ 'posts.post_id' => { -in => recursive_subquery( $c->model('DB::TagCloud1'), ,'post_id', 'tag_def_id',[ 16042, 190712, ])->get_column('post_id')->as_query }, });

    As an aside, the performance of this highlighted the unhappy fact that mysql tends to evaluate sub queries from the outside-in, so you may want to reverse the weighting in that setup.

Draw a Square With Perl!
4 direct replies — Read more / Contribute
by Dipseydoodle
on Aug 14, 2014 at 10:41

    Good morning monks. Today I figured I'd post this little script I wrote. It's not that cool and you can propbably point out errors in my writing/syntax style, so feel free to yell at me :)

    #!/usr/bin/perl # Put on your nerd glasses and draw a square! use strict; use warnings; my $balancex = 10; # width my $repeatx = $balancex; #don't change repeatx!!! use balancex instead +. my $repeaty = 10; # height do{ while($repeatx > 0){ print ". "; # change the period to print another character, but ke +ep the extra space. $repeatx -= 1; } print "\n"; $repeatx += $balancex; $repeaty -= 1; } until ($repeaty == 0); #corrected by Athanasius & AppleFritter

    No doubt this could be done in much fewer lines, or even as a one-liner :P But as it says it draws a square with periods, and is just fun to look at.

Check popular review sites for new reviews.
1 direct reply — Read more / Contribute
by wrinkles
on Jul 28, 2014 at 19:25

    This script checks select pages on some popular review sites for the latest review, and writes the date of the most recent review from each site to a file. Each time it is run, it checks against the previous results and sends an email notification with the date and link to page(s) with fresh reviews.

    "mailx" was used to send email. I suspect that this may not be available in Windows, I tested only on Mac OS X and Ubuntu.

    The following script has the pages hard-coded, as it was written for my school. Those pages (and your email addresses) could easily be replaced to suit your requirements.

    I found "The 10-minute XPath Tutorial" ("Automating System Administration with Perl, 2nd ed.) very helpful in understanding XPath. Thanks also to the help of fellow perl monks!

    By the way, "EB" and "MA" are shorthand for two separate campuses within our school.

    Update 2014-07-28 - I ran perlcritic and fixed some potential problems

    #!/usr/bin/env perl use strict; use warnings; use utf8; use Text::CSV; use Carp; use LWP::Simple qw(get); use Text::Unidecode qw(unidecode); use HTML::TreeBuilder::XPath; # Email Settings my %email = ( to => 'me@example.com,you@example.com', subject => 'New ECDS reviews found' ); # Reviews subroutine and URLs to check my $review_sites = [ { site => 'Yelp', sub => \&yelp_checker, review_pages => { 'EB' => 'http://www.yelp.com/biz/encinitas-country-day-school-encinitas?sort_b +y=date_desc', 'MA' => 'http://www.yelp.com/biz/encinitas-country-day-school-encinitas-2?sort +_by=date_desc' } }, { site => 'GreatSchools', sub => \&gs_checker, review_pages => { 'MA' => 'http://www.greatschools.org/california/encinitas/9670-Encinitas-Count +ry-Day-School/?tab=reviews' } }, { site => 'PrivateSchoolReview', sub => \&psr_checker, review_pages => { 'MA' => 'http://www.privateschoolreview.com/school_ov/school_id/ +2039' } }, { site => 'Kudzu', sub => \&kudzu_checker, review_pages => { 'MA' => 'http://www.kudzu.com/m/Encinitas-Country-Day-School-135 +71675' } }, { site => 'MerchantCircle', sub => \&mc_checker, review_pages => { 'MA' => 'http://www.merchantcircle.com/business/Encinitas.Country.Day.School.7 +60-942-1111?sort=created&dir=desc' } } ]; # Default date if no record my $default_date = '00-00-0000'; # Month name to number conversion my %month = ( January => '01', February => '02', March => '03', April => '04', May => '05', June => '06', July => '07', August => '08', September => '09', October => '10', November => '11', December => '12' ); # Where is the reviews file? my $reviews_filepath = "reviews.txt"; # Where is the alert message file? my $msg_filepath = "msg.txt"; # Slurp hash from reviews file my $old_reviews = hash_from_csv($reviews_filepath); my %new_reviews; # Iterate through each site for my $review_site (@$review_sites) { my $pages = $review_site->{review_pages}; # iterate through each campus html and collect xpath nodes while ( my ( $campus, $url ) = each %$pages ) { my $html = get $url or croak("Can't reach $url $!\n"); $html =~ s/([^[:ascii:]]+)/unidecode($1)/ge; my $tree = HTML::TreeBuilder::XPath->new; $tree->parse($html) or croak("Parse failed: $!\n"); my ($date) = $review_site->{'sub'}->($tree); # create hash keys from campus and review site names my $campus_site = $campus . '_' . $$review_site{'site'}; push( @{ $new_reviews{$campus_site} }, $date ); push( @{ $new_reviews{$campus_site} }, $url ); } } # Write message if new reviews my $msg = ''; while ( my ( $item, $data ) = each %new_reviews ) { unless ( $$old_reviews{$item}[0] eq $$data[0] ) { $msg .= "New review on $$data[0]: \n $$data[1]\n"; } } # Save message. open my $fh, ">:encoding(utf8)", "$msg_filepath" or croak("cannot open $msg_filepath: $!"); print {$fh} $msg or croak("Can't print message:\n$msg\n$!"); close $fh; # Write new review data to file. hash_to_csv( \%new_reviews, $reviews_filepath ); # Email message if exists send_email($msg) if length($msg); ######## SUBROUTINES ####### # import old data from file sub hash_from_csv { my $filepath = shift; open my $fh, "<:encoding(utf8)", "$filepath" or croak("cannot open $filepath: $!"); my $csv = Text::CSV->new( { binary => 1 } ); my %hash; map { $hash{ shift @{$_} } = $_ } @{ $csv->getline_all($fh) }; close $fh; return \%hash; } # write new data to file sub hash_to_csv { my ( $hash, $filepath ) = @_; open my $fh, ">:encoding(utf8)", "$filepath" or croak("cannot open $filepath: $!"); my $csv = Text::CSV->new( { binary => 1, eol => "\n" } ); for ( keys %$hash ) { my $colref = [ $_, $$hash{$_}->[0] ]; $csv->print( $fh, $colref ); } close $fh; return; } # send email notifications sub send_email { my ($body) = @_; open my $pipe, '|-', '/usr/bin/mailx', '-s', $email{subject}, $ema +il{to} or croak("can't open pipe to mailx: $!\n"); print $pipe $body; close $pipe; croak("mailx exited with a non-zero status: $?\n") if $?; return; } # extract date of most recent review from GreatSchools tree sub gs_checker { my $tree = shift; my $xpath = '//div[contains(@class,"media mbs")]/div[(@class="author small make-99 +9999 fl pbn mbn")]'; my $dates = $tree->findnodes($xpath); # dates returned as 'month dd, yyyy' my $date; $date = $$dates[0]->as_trimmed_text() if ( $$dates[0] ); if ( $date =~ /(\w{3,9})\s+(\d{1,2}),\s+(\d{4})/ ) { $date = $3 . '-' . $month{$1} . '-' . $2; } return ( $date || $default_date ); } # extract date of most recent review from Yelp tree sub yelp_checker { my $tree = shift; my $xpath = '//meta[@itemprop="datePublished"][1]'; my $dates = $tree->findnodes($xpath); # dates returned as 'yyyy-mm-dd' if ( $$dates[0] ) { return $$dates[0]->attr('content'); } else { return ( $$dates[0] || $default_date ); } } # extract date of most recent review from PrivateSchoolReview tree sub psr_checker { my $tree = shift; my $xpath = '//meta[@itemprop="datePublished"][1]'; my $dates = $tree->findnodes($xpath); # dates returned as 'yyyy-mm-dd' if ( $$dates[0] ) { return $$dates[0]->attr('content'); } else { return ( $$dates[0] || $default_date ); } } # extract date of most recent review from Kudzu tree sub kudzu_checker { my $tree = shift; my $xpath = '//div[@class="review_post_date"]/p/span[@class="rp-da +te"]'; my $dates = $tree->findnodes($xpath); # date returned as 'mm/dd/yyyy' my $date; $date = $$dates[0]->as_trimmed_text() if ( $$dates[0] ); if ( $date =~ /(\d{1,2})\/(\d{1,2})\/(\d{4})/ ) { $date = $3 . '-' . $1 . '-' . $2; } return ( $date || $default_date ); } # extract date of most recent review from MerchantCircle tree sub mc_checker { my $tree = shift; my $xpath = '//span[@itemprop="datePublished"][1]'; my $dates = $tree->findnodes($xpath); # dates returned as 'Month dd, yyyy at hh:mm PM' my $date; $date = $$dates[0]->as_trimmed_text() if ( $$dates[0] ); if ( $date =~ /\s*(\w{3,9})\s*(\d{1,2})\s*\,\s*(\d{4})\s+at\s+\d{1,2}\:\d{2} +\s+[AP]M/ ) { $date = $3 . '-' . $month{$1} . '-' . $2; } return ( $date || $default_date ); }

Add your CUFP
Title:
CUFP:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others surveying the Monastery: (6)
    As of 2015-03-30 10:43 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      When putting a smiley right before a closing parenthesis, do you:









      Results (637 votes), past polls