Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

The Monastery Gates

( #131=superdoc: print w/ replies, xml ) Need Help??

Donations gladly accepted

If you're new here please read PerlMonks FAQ
and Create a new user.

New Questions
Looking for ideas: Converting a binary 'flags' field into something human readable
1 direct reply — Read more / Contribute
by Monk::Thomas
on Jul 07, 2015 at 17:17

    Hello fellow monks

    I wrote a parser library for a specific class of binary files (resource files for a video game). It converts the file into a human readable data structure. (hashes of hashes of array of hashes kinda thing; data fields that are only relevant for parsing the binary data streams are stripped from the result)

    One of the data types it must be able to handle are 'flags' - a variable length sequence of bytes, where the actual value is uninteresting, the interesting part is whether a certain bit (flag) is set or not, e.g if a record is deleted or compressed or has a certain property. It seems like they are mostly exactly 1, 2, 4 or 8 bytes long, so I could easily use an unsigned integer value. However there are 2 things that bug me:

    • what to do if there turns up to be a flags field which does not match integers? (e.g. 6 or 10 bytes)
    • is there a better way to represent the 'flag/bit'-nature of the value?

    My ideas:

    One could emulate a '6 Byte Flags' field by reading uint32 + uint 16 and then manually calculate the combined integer value. Did anyone say kludge/wart? Yeah. Looks like one.

    Other representations I can think of could be 1110111100001 (which could get _extremely_ long) or a hash like:

    %flags = ( 'is_deleted' => 0, # a known flag 'is_compressed' => 1, # another known flag '2^15' => 1, # a bit that is set but unknown );
    (unknown bits with value 0 are not listed in order to conserve space)

    Your ideas?

A restful back end
5 direct replies — Read more / Contribute
by Random_Walk
on Jul 07, 2015 at 02:07

    Good day fellow monks,

    I am starting a new project, with a clean slate, this is a wonderful and rare delight.

    I need to create a back end web service and being a man of the times, I plan to use a REST style API.

    The back end needs to provide:

    • User Signup
    • User Authentication
    • Order creation
    • Payment
    • Job Acceptance
    • Geolocation of assets
    There will be more, but that's some of the main features.

    I do not have to write it in Perl, but that is where I know my way the best, so I am minded to make it so.

    I have started looking at Mojolicious, but then thought I'd back up a step and ask for some advice.

    • Is Perl still a good language to use, or am I a dinosaur?
    • Is Mojolicious a good way to build a REST service
    • What other frameworks would you folk recommend I look at
    • Does anyone know of a nice open source project providing some set of the above features, that could get me started?
    • Any other warnings or guidance from those with experience in REST

    Many thanks for your time

    Pereant, qui ante nos nostra dixerunt!
Question about communicating with a running perl script
3 direct replies — Read more / Contribute
by Amblikai
on Jul 06, 2015 at 13:44

    Hi folks, i'm not even sure where to begin on this one (which makes it hard to search for solutions)

    Basically i have a perl script which runs in the background performing various operations on an SQL database using DBI (love that module!).

    Separately, i have a website which is essentially just displaying the information in the sql database. It can however, modify the contents of the SQL database.

    The perl script running in the background has various "steps" which it goes through. I'd like to be able to control these steps through the web front end.

    The obvious first thought i had was polling a table in the SQL database but i wondered if there was a simpler or more elegant solution?

    Can anyone point me in the right direction?

    Thank you all!!

Subroutines not exporting? Why?
6 direct replies — Read more / Contribute
by no21
on Jul 06, 2015 at 10:06

    I have to be doing something dumb to not realize what the problem is here.

    I have a script with the following:

    use lib '/home/username/testmodules/'; use strict; use warnings; use My::Util; my x = get_file_name( ... );

    The following file exists:


    The file basically contains the following:

    package My::Util; use strict; use warnings; use Exporter; our @ISA = qw(Exporter); our @EXPORT = qw( get_file_name ); __END__ sub get_file_name { ... } 1;

    When I run the script, I get:

    Undefined subroutine &My::Util::get_file_name called

    I've googled this problem and found many, many pages discussing this very topic, and from everything I can tell I'm doing things correctly.

    It's acting like the subroutines from the module are not being exported, but I am using other modules successfully and (I appear to be) exporting their functions the exact same way.

    Please expose my ignorance. I MUST be missing something.

XSub won't write to memory file
1 direct reply — Read more / Contribute
by syphilis
on Jul 06, 2015 at 04:39
    I have (for demonstration purposes) an XS sub that prints "hello" to a filehandle.
    It will print to STDOUT, STDERR and to any real filehandle (opened for writing) that gets passed to it.
    But, if I pass it a filehandle to a memory file, then nothing gets written to the referenced scalar.

    Here's the demo script:
    use warnings; use strict; use Inline C => Config => BUILD_NOISY => 1; use Inline C => <<'EOC'; void to_FH(FILE * stream) { fprintf(stream, "hello"); fflush(stream); } EOC my ($got1, $got2); $got1 = get_string_1(); chomp($got1); # just in case ... print "OK 1\n" if $got1 eq "hello"; $got2 = get_string_2(); chomp($got2); # just in case ... print "OK 2\n" if $got2 eq "hello"; sub get_string_1 { # Write to a temporary file open TEMPFILE, '+>', undef or die $!; to_FH(*TEMPFILE); seek TEMPFILE, 0, 0; my $ret = <TEMPFILE>; close TEMPFILE; return $ret; } sub get_string_2 { # Write to a memory file my $out; open MEM, '+>', \$out or die $!; to_FH(*MEM); #close MEM; # Makes no difference return $out; # returns undef ... why ? } __END__ For me, outputs: OK 1 Use of uninitialized value $got2 in scalar chomp at line 24. Use of uninitialized value $got2 in string eq at line 25.
    So ... sub get_string_1 works fine and passes the string to the temporary file associated with the filehandle, from which I successfully retrieve what was written.

    With sub get_string_2, my expectation is that the string "hello" will be written to the sub's $out - but that's not happening, and $out remains uninitialized.

    What needs to be done here, to get it working as I expect ?

    I also tried an alternative version of the XSub:
    void to_FH(PerlIO * stream) { FILE * stdio_stream = PerlIO_exportFILE(stream, NULL); fprintf(stdio_stream, "hello"); fflush(stdio_stream); PerlIO_releaseFILE(stream, stdio_stream); }
    But this made no discernible difference to the behaviour.

DBIx::Dump Title
1 direct reply — Read more / Contribute
by ScottM
on Jul 05, 2015 at 12:58
    Hi Monks, is there a way to add a Title to my worksheet created with DBIx::Dump? TIA
missing values when using Parallel::ForkManager
1 direct reply — Read more / Contribute
by plagent
on Jul 03, 2015 at 01:51
    I used Parallel::ForkManager to scan the motifs in DNA string pairs. The script is the following:
    my $multi_cpu_num = 13; my @ciona_pwm_array = getCionaPWM(); my $output_temp = 'temp.ciona.temp'; my $output_temp_handle = FileHandle->new(">output_temp"); my $output_handle = FileHandle->new(">ciona.matrix.result"); my $result_buffer = undef; foreach my $id (1..100) { my $seq_1 = 'gatctatcgtagcagcta'; my $seq_2 = 'agtctacatgctacgatg'; my $pm = Parallel::ForkManager->new( $multi_cpu_num ); PWM: for my $pwm (@ciona_pwm_array) { $pm->start and next PWM; my $count = count_pwm_num($pwm,$seq_1,$seq_2); if($count != 0) { $output_temp_handle->print($id); $output_temp_handle->print("\t$pwm\t1\n"); } else { $output_temp_handle->print($id); $output_temp_handle->print("\t$pwm\t0\n"); } $pm->finish; } $pm->wait_all_children; } $output_temp_handle->close; # catch the Parallel::ForkManager output # the temporary output looks like that # a A 1 # b B 0 # c C 1 # a B 1 # in real code , I used 1, 2,3, instead of a, b, c ,d. my $reopen_handle = FileHandle->new($output_temp); while(my $line = $reopen_handle->getline) { chomp $line; my @buffer = split/\t/,$line; $result_buffer->{$buffer[0]}->{$buffer[1]} = $buffer[2]; } $reopen_handle->close; unlink 'temp.ciona.temp'; # print output_header_information :row 1 in program result table; $output_handle->print("id"); foreach my $pwm (@ciona_pwm_array) { $output_handle->print("\t$pwm"); } $output_handle->print("\n"); # print the content of program result table foreach my $id (1..100}) { $output_handle->print("$id"); foreach my $pwm (@ciona_pwm_array) { my $count = $result_buffer->{$id}->{$pwm}; $output_handle->print("\t$count"); } $output_handle->print("\n"); } $output_handle->close; sub ciona_pwm_num {} sub getCionaPWM {] exit(0);
    There are three parameters for the function count_pwm_num. The first one is $pwm, which is from list @ciona_pwm_array and is the motif feature. The second and third are DNA strings. The function is to calculate the motif number (how many?)using motif feature $pwm in the DNA strings. The aim of the function count_pwm_num is to report the motif number from both strings. The following table is the report from the program. However, I calculated the same DNA strings 100 times ( the first for-loop) and used same motif list (@ciona_pwm_array, the second for-loop), there is always missed motif(s) in the report. For example, row a (first row), the program reported all six motif number from A-F. row b missed the number of the motif A (because they were same DNA strings, the output should keep the same). row c missed the number of motif B.row d missed the number for motif A and B. It seemed the missed motif numbers were randomly selected. So my question is: why count_pwm_num randomly skip some $pwm and do not perform the function of count_pwm_num?

    program result table:

    id	A	B	C	D	E	F
    a	1	1	0	1	1	0
    b		1	0	1	1	0
    c	1		0	1	1	0
    d			0	1	1	0
    e	1	1	0	1	1	0
    Expected result
    id	A	B	C	D	E	F
    a	1	1	0	1	1	0
    b	1	1	0	1	1	0
    c	1	1	0	1	1	0
    d	1	1	0	1	1	0
    e	1	1	0	1	1	0
Max of 23 and 27 is 23?
6 direct replies — Read more / Contribute
by luxgladius
on Jul 02, 2015 at 13:35
    Here's the output of an actual debug session. Can anybody explain this?
    DB<10> p $#l 27 DB<11> p $#F 23 DB<12> p max($#l, $#F) 23 DB<13> p max(27,23) 27 DB<14> p 0+$#l 27 DB<15> p max(0+$#l, $#F) 27 DB<16> p max($#l, 0+$#F) 23
    So the second-to-last line tells me the solution, but I have no clue what the actual problem is. For what it's worth, I'm using 64-bit Activestate Perl 5.20.

    Edit: The max is from List::Util.

Hang on STDOUT
2 direct replies — Read more / Contribute
by DanEllison
on Jul 02, 2015 at 08:55

    I'm using Strawberry Perl in a Windows 8 environment. I have a scheduler I have written in Perl that I have been using for a couple of years. Its a threaded application that runs more than 1500 jobs every night, so its doing a lot of the same thing over and over.

    The main engine spawns worker threads to run the individual jobs and reports some progress to STDOUT. I have had issues in the past where STDOUT would somehow get corrupted and the engine would stop reporting progress but continue to run and I would eventually find my progress messages in one of the job log files.

    I haven't seen that issue in a while, but lately I find the engine has simply hung. It writes messages to both STDOUT and to a log file, and I've been adding debugging left and right, and it simply appears to be hanging on prints to STDOUT.

    Autoflush is on and I can see single character updates, but here's an example:

    sub message { if ($options{verbose} && $options{verbose} >= ($_[1] || 0)) { if ($inprogress) { print "\n"; $inprogress = 0; } print "<"; my @time = localtime; print ">"; printf "%04d-%02d-%02d %02d:%02d:%02d %s\n", $time[5] + 1900, $time[4] + 1, $time[3], $time[2], $time[1], $time[0], $_[0]; } }
    $_[0] is the message I want to print. I thought maybe I was hanging trying to get the local time, but no. I put the < and > prints on either side of the localtime, but it hung between the > and the printf of the message.

    Has anyone had experience with this?

Make sin wave with GD
1 direct reply — Read more / Contribute
by Anonymous Monk
on Jul 01, 2015 at 12:59
    Hi. I using GD package to make wave. But it does not work.
    use GD::Simple;
    $i = GD::Simple->new(400,400);
    ($x1,$y1) = (0,0);
    for ($x2=0; $x2<=400; $x2+=10) {
        $y2 = sin($x2)*400;
        ($x1,$y1) = ($x2,$y2);
    print $i->png;
    How do I get right value for y2? Thank.
Win32::API in perl v5.20.2
5 direct replies — Read more / Contribute
by BillKSmith
on Jul 01, 2015 at 11:43
    I recently upgraded from Active State Perl 5.16.2 to 5.20.2 on my Win7 (64-bit) system. Most went well except for one script (which had been in daily use for a few years. It now fails with the message:
    Can't locate object method "new" via package "Win32::API" at C:\USERS\ +BILL\PERL\ LIB/ line 15.

    I have attached the module referenced by the message. The code originally came from a post ( in this forum

    Notes: In the install procedure, I saved and restored the ppm module configuration with ppm profile save/restore. While attempting to diagnose the problem, I noticed that a newer version of Win32::API was available on ppm. Installing the newer version did not change the result at all.

    Would someone please suggest a way to resolve this issue.

    I also have an unrelated issue with this “upgrade” because ActiveState has removed Perl/Tk from their ppm repository.

Testing a .pl script
5 direct replies — Read more / Contribute
by perl_help26
on Jul 01, 2015 at 10:26

    I have a .pl script with the following structure:

    use X; use Data::Dumper; sub func1 {....} sub func2 {....} until (should_stop()) { ..... .... .. }
    I am trying to build a test module for this pl script. Is there a way to export the functions WITHOUT running the while loop? Preferably, I do not want to split the functions into a .pm file. Thanks ...
New Meditations
The problem of documenting complex modules.
7 direct replies — Read more / Contribute
by BrowserUk
on Jul 05, 2015 at 04:41

    This is meditation; but I also hope that it might start a discussion that will come up with (an) answers to what I see as an ongoing and prevalent problem.

    This has been triggered at this time by my experience of trying to wrap my brain around a particular complex module; but I don't want to get into discussion particular to that module, so I won't be naming it.

    Suffice to say that CPAN is replete with modules that are technically brilliant and very powerful solutions to the problems they address; and that deserve far wider usage than they get.

    In many cases the problem is not that they lack documentation -- often quite the opposite -- but more that they don't have a simple in; a clearly defined and obvious starting point that gives a universal starting point on which the new user can build.

    And example of (IMO) good documentation is Parallel::ForkManager. It's synopsis (I've tweaked it slightly to remove a piece of unnecessary fluff):

    use Parallel::ForkManager; my $pm = Parallel::ForkManager->new($MAX_PROCESSES); foreach my $data (@all_data) { # Forks and returns the pid for the child: my $pid = $pm->start and next; ... do some work with $data in the child process ... $pm->finish; # Terminates the child process }

    is sufficient to allow almost anyone needing to use it, for almost any purpose, to put together a reasonable working prototype in a dozen lines of code without reading further into the documentation.

    It allows the programmer to get started and move forward almost immediately on solving his problem -- which isn't "How to use P::FM" -- and only refer back to and utilise the more sophisticated elements of P::FM, as and when he encounters the limitations of that simple starting point.

    As such, the module is successful in hiding the nitty-gritty details of using fork correctly; whilst imposing the minimum of either up-front learning curve or infrastructural boiler-plate upon the programmer; who has other more important (to him) things on his mind.

    Contrast that with something like POE which requires a month of reading through the synopsis of the 800+ modules in that namespace POE::*, and then another month of planning, before the new user could put together his first line of code. As powerful as that module, suite of modules; dynasty of modules is, unless you have the author's help, and lots of time, getting started is an extremely daunting process. In that respect (alone perhaps), POE fails to enable a 'simple in'.

    And before anyone says that it is unfair to compare those two modules -- which maybe true -- the purpose was to pick extremes to make a point; not to promote or denigrate either.

    Another module that I know I should have made much more use of in the type of code I frequently find myself writing, is PDL. I've tried at least a dozen times to use PDL as a part of one of my programs; and (almost) every time I've abandoned the attempt before ever writing a single line of PDL, because I get frustrated by the total lack of a clear entry point in to the surfeit of documentation.

    There's the FAQ, and the Core; and the Index; and the QuickStart; and the Doc; and the Basic; and the Lite; and the Course; and the Philosophy; and the pdldoc; and the Tips; and ... I'm outta here. I'm trying to write my program, which does a little math on some biggish datasets that would benefit from being vectored, but life's too short...

    Again; the underlying code is brilliant (I am assured), and it isn't a case of a lack of documentation; just a mindset that says: "this is PDL in all its glory, power and nuance. bathe yourself in its wonderfulness and wallow in its depth". Oh, and then when you've immersed yourself in its glory, understood its philosophy, and acclimated its nuance, then you can get back to working out how to use it to solve your problem.

    And that's a real shame; and a waste.

    I'm not sure what the solution is. I do know that the modules I use most List::Util, Data::Dump, threads etc. I have rarely ever had to look at the documentation; their functionality has (for me) become an almost invisible extension of Perl itself, and only the occasional (perhaps you forgot to load "sum"?) reminds me that they aren't.

    Of course, what they do is essentially pretty simple; but that in itself is a perhaps a clue.

    I do know that (for me) the single most important thing in encouraging my use of a module is being able to C&P the synopsis into my existing program, tweak the variable names, and have it do something useful immediately.

    In part, that comes down to a well designed API; in part, to well-chosen defaults; and part having a well-chosen, well-written synopsis that addresses the common case; with variable names and structure that make it obvious how to adapt that synopsis for the common case. Once I have something that compiles and runs -- even if it doesn't do exactly what I need it to do; or even what I thought it would do from first reading -- it gives me a starting point and something to build on. And that encourages me to persist. To read the documentation on a as-I-need-to basis to solve particular problems as I encounter them.

    Over a decade ago, I posted My number 1 tip for developers.; and this is the other side of that same philosophy. Start simple and build.

    And that I think has to be the correct approach to documenting complex modules. They need to:

    1. Offer a single, obvious, starting point. The in.
    2. That needs to be very light on history, philosophy, jargon, technical and social commentary and background. And choice.
    3. It needs to offer a single, simple, well-chosen, starting point, that requires minimal reading to adapt to the users code, for the common case.
    4. It then needs to offer them a quick, clear, simple path to solving their problem.

    What it must not do:

    • It mustn't present them with 'a bloody great big list of entrypoints/methods'.
    • It mustn't offer them a myriad of choices and configuration options.
    • It mustn't take them on a deep immersion in the details of either algorithms or implementation.
    • It mustn't present them with either "Ain't this amazing" nor "Ain't I clever" advert.
    • It mustn't waste their time with details of your personal preferences, prejudices, philosophies and theologies.

    If you want programmers to use your modules, you need to tell them what (the minimum) they *NEED TO KNOW* to get started. And then give a clear index to the variations, configurations and extensions to that basic starting point.

    Achieve that, give them their 'in', with the minimum of words, fuss or choice, and they'll come back for all the rest as they need it.

    This is ill-thought through and incomplete, so what (beyond risking offending half the authors on CPAN) am I trying to achieve with this meditation?

    I'd like to hear if you agree with me? Or how you differ. What you look for in module documentation. Examples that you find particularly good; or bad.

    It'd be nice to be able to derive from the thread, a set of consensus guidelines to documenting moderate to complex modules -- that almost certainly won't happen -- but if we managed to get a good cross section of opinions on what makes for good and bad documentation; and a variety of opinions of the right way to go about it; it might provide a starting point for people needing to do this in the future.

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
    I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!
New Monk Discussion
Attribute Removed When Post is Listed in a PerlMonks Section
1 direct reply — Read more / Contribute
by kcott
on Jul 05, 2015 at 08:50


    Last week I wrote Syntax-highlight Non-Perl Code for HTML which I posted in Cool Uses for Perl.

    The first line contained the abbreviation CRPG. I provided a tooltip for this using the title attribute, like so:

    <span title="Computer Role Playing Game">CRPG</span>

    The tooltip showed up in both the [preview] and [create] renderings. Everything looked fine and I thought no more about it.

    That post was front-paged and I've just noticed that the version in The Monastery Gates listing (under New Cool Uses for Perl) had no tooltip for CRPG.

    I checked the source HTML in Syntax-highlight Non-Perl Code for HTML which had, as expected:

    <span title="Computer Role Playing Game">CRPG</span>

    I then checked the source HTML in The Monastery Gates which had, not as expected:


    And now, while previewing this post, I also notice that the listing in Cool Uses for Perl has no tooltip either (source HTML: <span>CRPG</span>).

    That's now potentially becoming somewhat confusing. To summarize (and clarify):

    Node ID Node Name Tooltip Source HTML
    1132421 Syntax-highlight Non-Perl Code for HTML Yes <span title="Computer Role Playing Game">CRPG</span>
    131 The Monastery Gates No <span>CRPG</span>
    1044 Cool Uses for Perl No <span>CRPG</span>

    So, while that's two PerlMonks Sections that I was able to check, this could also be happening in other sections.


    Is the title attribute (and value) intentionally removed from the listings in PerlMonks Sections? If so, what is the rationale behind this? Are attributes other than title affected? Are elements other than span affected?

    If this attribute was not intentionally removed then, presumably, that indicates a problem somewhere in the process of listing a post in various PerlMonks Sections. If this is the case, can it be fixed?

    -- Ken

Log In?

What's my password?
Create A New User
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (7)
As of 2015-07-08 00:39 GMT
Find Nodes?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...

    Results (93 votes), past polls