Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Reliable software: SOLVED (was: Reliable software OR Is CPAN the sacred cow)

by powerman (Friar)
on Sep 15, 2006 at 11:47 UTC ( #573100=perlmeditation: print w/ replies, xml ) Need Help??

Update: I suggest you first looking at solution below, before reading complete thread.

Class, prepare your mouse pointers, take them near '--' and be ready to click 'Vote!' because I'm going to say something terrible about CPAN.

Few days ago I've asked about Reliable email parsing and that discussion uncover some points:

  • People doesn't need reliable perl modules, worksforme is enough.
  • People even doesn't understand what reliable solution is good thing which must be first priority goal (even if you can't reach this goal right now)!
  • CPAN == goodness. Use CPAN module - is only recommended solution for anything. If you say all existing CPAN modules for {any task here} is wrong because {any reason here} - you got a lot of '--', no matter is you right or not!
For me, ANY software MUST be developed with these priorities in mind:
  1. be reliable: it should do it work correctly (ex. for email parsing task it must support all features defined in RFCs related to email format plus be able to handle non-RFC-compliant emails produced by some buggy software)
  2. be secure: it should not allow unauthorized usage
  3. next priorities may vary from have a lot features to work fast or be cheaper or have intuitive interface, etc.
... because if software isn't doing exactly what it should or open way for hackers/viruses into my system I don't need it, no matter which (unreliable!) features it have!

I say "ANY software", but especially this important for all reusable things like perl modules and core things like OS kernel, perl itself, text editor or web server (i.e. all things which you use to develop your own application). Without this developing own reliable application become a nightmare because lower layers which you use isn't reliable itself and you spend huge amount of time detecting and fixing bugs there... or reimplementing these layers as part of your application. :(

Now about CPAN. 99.9% of CPAN modules developed without reliability and security in mind. They are "worksforme", feature rich, have ugly interfaces, bloated, clever, anything! But not reliable and secure.

So, if you wanna develop reliable application you can't use most of CPAN modules. List of "safe" modules may vary, but usually it include:

If this looks like "a lot" of modules then check them again: they all are low-level modules, each doing small simple task (probably except DBI)! All CPAN modules I seen which try to solve more complex tasks doesn't looks reliable and secure. A-L-L! Ok, I (still) hope there some exceptions which I missed, but this doesn't important.

So... for every high-level task (and low-level tasks which has no good enough CPAN module) you must develop custom solution. We all read: 'CPAN is goodness!'; 'Any your task already solved by reusable CPAN module, no need to reinvent it!'; 'Perl is better than other languages because only Perl has CPAN!'; etc. for YEARS...........................
This result in people stop thinking critically about CPAN, they believe is't goodness because it IS goodness, and that's all.

Now it's time for adducing some proof, small examples for people who think CPAN modules ARE reliable.

First example - executing external process task.
There already exists a lot of ways to execute command: system(), open(), `` (backticks), IPC::Open3, IPC::Run, IPC::Run3. In short, system() is good but doesn't allow interaction with running command, and all others doesn't handle signals correctly. Below is gore details, if interested.

While it's possible with system() to run command with custom filehandles instead of inheriting only STDIN/STDOUT/STDERR of current process, it has some limitations: - You can't use safe LIST form of system() and should execute your command using shell to have filehandle redirections like '2>&1' working. - You can't interact with running command using pipes. - If you need to give your non-STDIN/OUT/ERR filehandle or if you need to give more than 3 filehandles to command you should use fcntl() to modify close-on-exec flag and/or reopen your STDIN/OUT/ERR to needed filehandles. - User can't set timeout for command. There no way to setup alarm() for command. User can setup alarm() for his main process, but: 1) this isn't acceptable in module because user can already setup some alarm() before calling our module 2) this alarm() will interrupt system(), but not interrupt running command (and we can't kill it because we don't know it's pid) open() limitations: - open(), unlike system(), doesn't block SIGCHLD. This may be fine in user code, but in module this result in two problems if user has $SIG{CHLD} handler installed: 1) pid of process open()'ed in module will be delivered into user's SIGCHLD handler 2) $? status will not be available in module - open(), unlike system(), doesn't block SIGINT & SIGQUIT in main process so if you use open() instead of system() just to interact with executed command using pipe like: open(my $fh, 'some_foreground_command |') then both this command and your main process will receive these signals if user press Ctrl-C or Ctrl-\. - If command will exit while user print() into pipe SIGPIPE will kill main process if user don't block it. - All system() limitations also apply to open(), with only correction: you can interact with running command using pipe, but only single pipe - if you need more pipes then you should use IPC::Open3, IPC::Run or IPC::Run3. `command` (backticks) limitations: - No safe LIST form. - You can't interact with running command using pipes. - If you need to give your non-STDIN/OUT/ERR filehandle or if you need to give more than 3 filehandles to command you should use fcntl() to modify close-on-exec flag and/or reopen your STDIN/OUT/ERR to needed filehandles. - All open() limitations apply. IPC::Open3 limitations: - No signal handling at all, so all open() limitations about SIGCHLD, SIGINT, SIGQUIT and SIGPIPE apply. - No timeout. - Unable to use more than 3 filehandles. IPC::Run limitations: - No signal handling at all, so all open() limitations about SIGCHLD, SIGINT, SIGQUIT and SIGPIPE apply. IPC::Run3 limitations: - No signal handling at all, so all open() limitations about SIGCHLD, SIGINT, SIGQUIT and SIGPIPE apply. - You can't interact with running command using pipes.
My realization of this task have 110 lines of code and it's only realization I know which handle all nuances described in Stevens APUE book. You can check it by downloading POWER::Utils module from my website.

Second example - timers and timeouts.
If you use time() function for realizing timer or timeout, no matter CORE::time() or Time::HiRes::time() -- your code is unreliable. Why? Because. Because there NTP and /bin/date. And they may (and will!) change current time, both forward and backward. Only reliable way to realize timer/timeout in Linux - use CLOCK_MONOTONIC param for clock_gettime(2). This required syscall() until I've asked Time::HiRes author to add this feature, and since Time::HiRes 1.77 you've clock_gettime(CLOCK_MONOTONIC) feature. How many CPAN modules work with timers/timeouts? And is even single of them use CLOCK_MONOTONIC? Few months ago I've searched CPAN and found 0 such modules.

Third example - mailbox parsing (from my previous thread).
In short, mailbox can be in one of 4 formats: mboxo, mboxrd, mboxcl and mboxcl2. There no way to autodetect it. Reading from mailbox using wrong format lead to damaging messages. CPAN has many modules which has 'read mailbox' feature, but no one of them allow user to configure mailbox format BEFORE reading, and only one has note in documentation about these formats (and try to do it best autodetecting it, which is anyway impossible).

Fourth example - reliable eval().
Eval? What's the hell is wrong with eval!? Only one: eval() doesn't support one advanced perl feature: source filters. You can found my version of eval (6 lines) which compatible with internal perl eval but also support source filters in same POWER::Utils module.

Want more? Ok. Fifth example - using GPG.
CPAN contains a lot of modules for accessing GnuPG, but they all not reliable enough and some of them not secure. My version (see POWER::GPG module on my site) is module execute gpg with correct signal handling; use non-blocking pipes and multiplexing to avoid hang on processing huge files; never store sensitive information on hard drive in temporary files (one temporary file used for checking detached signature); use timeout to protect against gpg hangs. Details about existing CPAN modules below:

Limitations of other modules from reliability/security view: =over =item GPG - Doesn't handle SIGCHLD, SIGINT, SIGQUIT. - Hang on large files. - Parse unreliable STDOUT instead of reliable --status-fd. =item Crypt::GPG - Doesn't handle SIGINT, SIGQUIT. - Incorrectly handle SIGCHLD. =item GnuPG - Doesn't handle SIGCHLD, SIGINT, SIGQUIT. - Use deprecated shared memory interface. - Use temporary files to store sensitive information. =item GnuPG::Interface - Doesn't handle SIGCHLD, SIGINT, SIGQUIT. =back

And the last thing which drives me crazy. Perl isn't a new language, it activelly used in the world for more than 10 years. Perl, by design, is text processing language, used initially for system administration. Parsing emails is really needed task, it's text processing task and it's related to system administration. CPAN has huge amount of modules for this task, and no one of them support all email-related RFC. After so many years no one developed such module! AND NO ONE THINK IT'S NEEDED - at least comments to my question say so......... "Yeah, why you think you need reliable email parser, what you will use it for? Forget about it, man, you don't need it. Nobody need it!" :(

Comment on Reliable software: SOLVED (was: Reliable software OR Is CPAN the sacred cow)
Select or Download Code
Re: Reliable software OR Is CPAN the sacred cow
by talexb (Canon) on Sep 15, 2006 at 12:20 UTC
      For me, ANY software MUST be developed with these priorities in mind:
      1. be reliable: it should do it work correctly (ex. for email parsing task it must support all features defined in RFCs related to email format plus be able to handle non-RFC-compliant emails produced by some buggy software)
      2. be secure: it should not allow unauthorized usage
      3. next priorities may vary from have a lot features to work fast or be cheaper or have intuitive interface, etc.

    So, what's your thesis? Are you complaining that there are no Mail::* modules that follow 100% of the RFCs? Or that 99.9% of the modules on CPAN aren't reliable, secure, etc.? Or both?

    Well, this is just a reminder that this is the open source community: We're mostly volunteers. If you wanna write something that'll help you get your job done better, more efficiently, more safely, and if you want to sure that work with everyone else, terrific. If you want to use something that someone else has written, go for it. You can even make your own improvements and maybe even get those improvements included, with the thanks of the authour and the community.

    But if you're complaining that none of (for example) the Mail::* modules are totally compliant with the RFCs and want to blame someone, blame yourself for expecting the perfect solution for free. And if you want to do something about, use one of the existing modules and fix it up (see previous paragraph) or write your own, using the RFC. You may not encounter fame and fortune as a result, but if the module works well, the community will thank you and you may find yourself being bought a lot of beer.

    Finally, the requirement that a CPAN module have an intuitive interface escapes me a little -- you learn about the module interface by reading the module documentation. I don't know how a module's interface can be made .. intuitive. For example, DBI uses the 'connect' method to create a new database connection object. By the time someone pointed how non-OO this was (of course, it should be a 'new' method, everyone knows that), I'd already been using the 'incorrectly name' call to DBI for years, and thought nothing of it .. but it sure is intuitive.

    I'm working on preparing a module for CPAN right now, and it's hard .. because I haven't done it before .. but I thnk it'll be worth it, both for myself and for anyone out there who might use it.

    And boy, I'm looking forward to that beer. :)

    Alex / talexb / Toronto

    "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

      Are you complaining that there are no Mail::* modules that follow 100% of the RFCs? Or that 99.9% of the modules on CPAN aren't reliable, secure, etc.? Or both?
      Both.
      But if you're complaining that none of (for example) the Mail::* modules are totally compliant with the RFCs and want to blame someone, blame yourself for expecting the perfect solution for free.
      So, that's the problem? Money? Ok, no problem. Develop reliable module and sell it to people who need reliable solutions.

      But I don't think problem is in money. I think problem is in people's mind. Most people just don't think they should develop reliable software, then think "worksforme" is good enough. And my thesis is: this is terrible! Ok, you CAN develop "worksforme" software for yourself, but don't try to make it public by uploading it to CPAN.

      Finally, the requirement that a CPAN module have an intuitive interface escapes me a little
      I don't say that. Probably you misread something.
            Are you complaining that there are no Mail::* modules that follow 100% of the RFCs? Or that 99.9% of the modules on CPAN aren't reliable, secure, etc.? Or both?
          Both.

        OK then -- my suggestions is that you pick a module, retrofit it to make it reliable and secure (according to your standards) and post the results here on Perlmonks.

        We gladly await your results, and promise to buy you beer at the next YAPC.

        Alex / talexb / Toronto

        "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

        Guess what. People are using modules from CPAN. They work for them. They get the job done. Their employer gives them money and they get to feed their kids (or buy new gadgets for their computer(s)). I'm kinda inclined to say "worksforme" IS good enough.

        If you want to run a superCPAN containing only the perfect modules, go ahead. I'd rather have the wealth of "worksforme" modules than a few and far apart perfect ones. Sure it would be great if people paid more attention to safety. Preventing them from sharing something that works for them as well as for other people is not the way to improve that though.

        If you are so inclined, bug the authors to improve their modules (preferably telling them what's wrong and how to improve it), rate modules, keep a list of those that passed your scrutiny, but please do not frighten people away from sharing. The world is not perfect, learn to live with that.

        Ok, you CAN develop "worksforme" software for yourself, but don't try to make it public by uploading it to CPAN.
        Why? Could it be that you're expecting the wrong things of CPAN? Perl is useful in large and complex projects as well as in little Ineedthisbastardnow.pl's. Both are part of the Perl world and I expect both to be found on CPAN.

        Ordinary morality is for ordinary people. -- Aleister Crowley

      Well, this is just a reminder that this is the open source community: We're mostly volunteers. If you wanna write something that'll help you get your job done better, more efficiently, more safely, and if you want to sure that work with everyone else, terrific. If you want to use something that someone else has written, go for it. You can even make your own improvements and maybe even get those improvements included, with the thanks of the authour and the community.

      Note - I'm not specifically addressing talexb in this reply, but the OSS community in general.

      I think this attitude is a cop-out, since it is exactly counter to one of the original points about CPAN being a sacred cow. There are innumerable replies on this board and others with people saying, "Don't re-invent the wheel! Use Foo::Widget from CPAN!" And that's fine, we're allowed.

      But jumping to "Hey, it's free software and you get what you pay for!" if people gripe about it doesn't help the case for open source at all. Yeah, I know, we are all volunteers (well, mostly), and yeah, people do get what they pay for, but it's a horrible way to try and defend yourself.

      How many people around here gripe about Microsoft's ever increasing systems requirements for its operating system and office suite? I need a 8GHz Teraflop 90 core Whatsamadoodle just to run a word processor? That's BS!. And that's fine, gripe. But then how many people turn around and say that perl is a lightning fast language and it's just as good as C for most tasks (at least on the web) and things would just be a-okay if we threw some more hardware at it? If Microsoft gets chastized for ever increasing requirements, so should you.

      Same with open source. You can't sing the virtues of the wide open community with its own source code and legion of people poking the code and then turn around and say "We're all volunteers. That's what you get!" It firmly and completely negates the original benefits, I think.

      A previous boss used to really sneer about using "free software" for these reasons. The app we had was written in an old version of Mason that wasn't compatible with Perl 5.8. But, the API had changed enough that it would require some (unknown but probably semi-big) amount of work to upgrade to a current Mason. So the app is stuck on perl 5.6 and old mason. When he came on board and heard about this, the first thing he blamed was free software and how you can't trust anything and no one cares.

      Side note - I did point out to him things like .NET's API radically changing or Oracle support nightmares or whatever. Commercial software doesn't necessarily give you any more stability, reliability, or continuity. You just pay more for it. He wasn't convinced.

      I feel like I'm starting to ramble, so I'll close up. But I don't think that falling back on "It's free, it's what you get" helps the case.

        The point, as I see it, is that open source is a possible starting point in the journey of software development.

        No one's obligated to use open source. You are welcome to write your own stuff from scratch if you want. For me, however, open source is a heck of a good place to start, and I accept the reality that something I get from CPAN may not be complete, or bug-free, with the knowledge that I can make a difference by helping improve the module I'm using.

        And that's an important difference. I can make a change to Template::Toolkit if I want, for example. If I find a bug, contact the authour/maintainers, talk with the folks in the community, offer the patch, how can that not benefit everyone? I get something that works 1% better. So does the community.

        As everyone donates their 1%, we get a better and better product, and build a stronger and stronger community. Many eyeballs make for shallow bugs, and all that.

        Instead of fellow monk powerman ranting about how poorly the Mail::* modules fare in following the various RFCs and in security and reliability, I believe s/he should concentrate their energies on making it right.

        Alex / talexb / Toronto

        "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

        But jumping to "Hey, it's free software and you get what you pay for!" if people gripe about it doesn't help the case for open source at all. Yeah, I know, we are all volunteers (well, mostly), and yeah, people do get what they pay for, but it's a horrible way to try and defend yourself.
        I'd like to disagree, if only partly and on the interpretation. The Perl community has one big advantage: CPAN. But with this I don't mean the large (huge) base of code, but the people providing the community with it. CPAN lives from contribution, from critique, from honest opinions and bug reports. But it also feeds on the willingness of people to contribute. In this context, I find a comment that reads mostly like "Don't upload your crap to CPAN! I can't use that in my business!" pretty disrespectful.

        Nobody forces anyone to use any module from the CPAN (Well, your contract with your employer might, but you've signed it and took the job). You can use only those tools for the job that you find the right for the job, and it's up to you to either share the authors and communities opinions or disagree.

        In my opinion, the solutions to problem domains provided by CPAN for the workforme category are as invaluable as the larger modules that evolve into a better maturity and bring Perl forward. And in this consequence I'd have to answer the question if people should upload their worksforthem code or not with a loud and clear "Yes." Without these modules and contributions, you wouldn't have a solution either. But with them, someone else might. And who knows? Modules can be taken over, forked and even made reliable by other people if there's a need for it and someone's willing to give their time. That's a pretty important part of open source too.
        Same with open source. You can't sing the virtues of the wide open community with its own source code and legion of people poking the code and then turn around and say "We're all volunteers. That's what you get!" It firmly and completely negates the original benefits, I think.
        Yes but, ... what do you want us to say? It's just the truth. If someone doesn't want to write module XYZ himself, how can he expect anyone else to do it for him? And if he's wanting to contribute the module, what's the fuss?
        A previous boss used to really sneer about using "free software" for these reasons.
        Then I'd say he didn't understand "free software." It doesn't mean "same as if you buy it, but you don't have to pay for it." Freedom always includes the consequences. There is no one true solution.
        Commercial software doesn't necessarily give you any more stability, reliability, or continuity. You just pay more for it. He wasn't convinced.
        Exactly. I don't think the reason there lies in open source quality, tho, but more in peoples understanding of it.
        But I don't think that falling back on "It's free, it's what you get" helps the case.
        Well, to someone asking for more power in some areas of Perl development, I'm afraid it's the only thing I could think of to answer.

        Ordinary morality is for ordinary people. -- Aleister Crowley
        But jumping to "Hey, it's free software and you get what you pay for!" if people gripe about it doesn't help the case for open source at all.

        It doesn't help the naive case, but I don't care one whit about the naive case for open source or free software.

        The reason I believe that open source and free software can reach higher levels of quality and utility and applicability than proprietary software is because open licensing gives a wider community the opportunity to improve the software.

        The real response really ought to be "If you're part of the community, the quality and applicability and utility of the code is your concern too. Contribute." Contrarily, if you don't contribute in some means, you're very clearly not part of the community and your opinion about the code doesn't really matter.

Re: Reliable software OR Is CPAN the sacred cow
by davorg (Chancellor) on Sep 15, 2006 at 12:33 UTC
    This result in people stop thinking critically about CPAN, they believe is't goodness because it IS goodness, and that's all.

    I don't believe that this is an accurate summary of anyone's views on CPAN (or, at least, anyone who has looked seriously at CPAN).

    There is a _lot_ of code on CPAN. A lot of it is of very high quality and very useful. Some of it isn't. I don't think that's a problem with CPAN however, I see the low barrier to entry for CPAN as a feature. The Perl community has, in recent years, set up side projects to CPAN which allow users to see the "quality" of CPAN modules (as discussed by other users or determined by automated tests) and to give them more information on which to base their decision on whether or not to use a particular CPAN module.

    We all realise that there are problems with some CPAN modules. There are almost certainly still bugs to find in pretty much every CPAN module. When you find one of these problems, you have a choice. You can mutter vaguely about the problem (as you've been doing on this site for the last few days) or you can help the author of the module to fix the problem and thereby help make the module better. There are a number of ways that you can help:

    • Raise a bug report in RT (it would be nice if your bug included a failing test).
    • Review the module on CPAN Ratings
    • Annotate the POD to correct any error
    • Join the module's mailing list (if one exists) and explain the problem
    • Offer to fix the problem - provide a patch

    This is open source software. Of course, you're free to use it (or not use it) as you wish. No-one is going to force you to contribute. But it would be really good if you could help out. You've made it clear recently that you're not happy with the quality of CPAN's email handling modules. But as far as I can see, you have done nothing to improve the situation. Yes, you say you've written your own implementation, but why haven't you released it? Or why haven't you contacted the Perl email project to offer your help? Or given them concrete examples of the way that their current modules fail? It's very easy to complain, but harder to do something constructive.

    I don't believe that CPAN is a sacred cow. And I don't know anyone who does. I know that there's an awful lot of rubbish on CPAN. But I find rubbish there that I'd like to use then I don't waste time moaning about it, I do what I can to fix it. Why don't you do the same?

    --
    <http://dave.org.uk>

    "The first rule of Perl club is you do not talk about Perl club."
    -- Chip Salzenberg

      There is a _lot_ of code on CPAN. A lot of it is of very high quality and very useful.
      Agreed! But high quality here isn't equal to reliable. For example, if your MTA will have a lot of features, high quality code and be very useful, but... in some rare cases (poweroff, badblocks in hard drive) it will damage your mailbox - NOT because of some small bug (this always can happens), BUT because it author doesn't try to think about this issue and concentrate on features instead (which in turn result in different internal architecture which make fixing these issues much harder or even impossible). And same is for security - high quality application developed without security in mind is suxx, sorry.
      You can mutter vaguely about the problem (as you've been doing on this site for the last few days)
      No, sorry, I'm not mutter vaguely about some features I need, which not exists in current CPAN software. I'm yell about lack of reliability and security in 99.9% of CPAN!

      I'll be happy to send some bugreports or patches to existing modules, but I can't send patch which magically convert module developed without reliability and security in mind into reliable and secure module - it's much ease to develop own modules instead! (And this is what I'm doing - you can download a lot of my modules for free from my site.)

      When I found reliable&secure enough software with some bugs I always send bugreports or patches. Last example of such software was MatrixSSL C library and Crypt::MatrixSSL perl interface for it. I've send several bugreports and patches to MatrixSSL, and found their developers responsible enough. And I've sent 12 patches to Crypt::MatrixSSL, but it author doesn't reviewed them yet (they was send 3 months ago, so probably I will fork this module if author doesn't reply in few weeks more). Here ChangeLog, if interested:

        I'll be happy to send some bugreports or patches to existing modules, but I can't send patch which magically convert module developed without reliability and security in mind into reliable and secure module - it's much ease to develop own modules instead! (And this is what I'm doing - you can download a lot of my modules for free from my site.)

        Then I take back my commentary about bugpatching. Yet my core point remains -- if you have time to create modules, then either a) continue to help with bugpatching, b) put 'em on CPAN, so we can easily run and compare (and so others can test out your modules from the CPAN testing framework), and/or c) write actual documents to help us understand what you mean by reliable/secure coding.

        Otherwise, we'll never know what the heck you're talking about, because all you've done is rant. And rants rarely get anything done, but we're all impressed with solid, well-documented code and coding ideas.

        ----Asim, known to some as Woodrow.

Re: Reliable software OR Is CPAN the sacred cow
by perrin (Chancellor) on Sep 15, 2006 at 12:39 UTC
    Well, duh. It's very rare for two people's requirements to be exactly the same, so of course the existing modules don't all do exactly what you wish they did. Your definition of "reliable" seems to be "does precisely what I want."

    This is the part where you're getting people mad at you. If you run around shouting that people's code is "unreliable" rather than simply not covering some functionality you want (Source filters in eval? Give me a break!), you are certain to annoy them and make them less likely to help you. When you add injury to insult by not offering to help improve what you say is lacking in their modules, what do you expect them to do? Thank you and beg for more of your great wisdom?

    CPAN comes with a 100% money back guarantee. Feel free to cash in, or to fork some modules, or to help people fix modules, or whatever it is that you want to do. Meanwhile, the horribly broken e-mail modules that you're complaining about will go back to running most of the Internet's mail.

      Your definition of "reliable" seems to be "does precisely what I want."
      Pure nonsense. "Reliable" mean I can rely on this software: every documented feature work as expected in all cases. "Does what I want" mean have some features.

      There a lot of modules in CPAN which "does what I want, and much more", but I can't use them because I open their source and in few minutes/seconds discover bugs which nature say: hey, my author does't think about this, and that, and about that critical issue too while developing me. This in turn mean there no sense in patching/bugreporting, just because author doesn't have reliability and security in his goals, and I can't reach these goals in this situation!

      (Source filters in eval? Give me a break!)
      Hmm. Can you give me full list of perl features which... too complex, or too exotic and which may not be supported by eval()... or do()... or require()... or, better, let's remove them from perl at all! Eval manual say: "EXPR is parsed and executed as if it were a little Perl program.", and doesn't say "except source filters feature", so it MUST support everything which supported in "little Perl program". It doesn't. (I hit this bug when I experimented with html template engine based on source filters.)
      Meanwhile, the horribly broken e-mail modules that you're complaining about will go back to running most of the Internet's mail.
      YEAH. And that's terrible, if you ask me!!!

      Yesterday I've found bug in qmail/mutt: qmail store messages in mboxrd format, while mutt incorrectly convert them into mboxcl2 format. So my mailbox have messages in 3(!) different formats:

      1. mboxrd - incoming - just delivered by qmail
      2. mboxcl2 - outgoing - composed by mutt
      3. ??? - mboxrd incorrectly converted in mboxcl2 by mutt
      And you think it's ok? I don't think so... Additional fun here is what term 'mailbox format' has no more sense, because there now 'messages in different "mailbox" format in the single mailbox'. :-/

        "Reliable" mean I can rely on this software: every documented feature work as expected in all cases.

        What you're doing here is adding your own personal requirements and claiming that they are basic necessities. It's just not true. Not everyone needs IPC that supports pipes and lists. Not everyone needs timers that handle someone changing the clock under them. Not everyone needs mailbox parsers that handle all possible formats and survive power failures. So why should they spend their time implementing the features you want, when they already have the ones they need? Adding those features is time-consuming and expensive.

        Can you give me full list of perl features which... too complex, or too exotic and which may not be supported by eval()

        You picked a great one here. Your complaint about eval is that it doesn't handle source filters? My complaint about perl is that it does handle source filters. I think you're probably the only one on the planet who wants this feature in eval. And the lack of it shows some kind of moral corruption in all perl developers?

        So your definition of "reliable" is that the software has absolutely no bugs and is 100% secure? I'd love for you to provide a list of software ( any software, in any language ) that fits this. :)

        As I understand it your definition of "reliable" is a unicorn. A mythical thing that does not exist, but fun to talk about.

        Frank Wiles <frank@revsys.com>
        www.revsys.com

        This in turn mean there no sense in patching/bugreporting, just because author doesn't have reliability and security in his goals, and I can't reach these goals in this situation!

        I claim pure, unadulterated bovine leavings.

        If you can't fix bugs in a piece of software, you're not much of a programmer. Honestly, I'm about ---><-- from putting you in the Noisy Internet Crank bucket forever, and that's after reading three messages from you.

        How about dropping the histrionic Smartest-Person-In-The-Room syndrome and actually attempting to work with other people, rather than inventing your own little playrooms? 'cuz you know, some of the people who actually built some of the systems you're dumping on might just care a little bit about this kind of abuse.

Re: Reliable software OR Is CPAN the sacred cow
by blue_cowdawg (Prior) on Sep 15, 2006 at 13:08 UTC
        1. be reliable: it should do it work correctly (ex. for email parsing task it must support all features defined in RFCs related to email format plus be able to handle non-RFC-compliant emails produced by some buggy software) 2. be secure: it should not allow unauthorized usage 3. next priorities may vary from have a lot features to work fast or be cheaper or have intuitive interface, etc.

    Are you trying to tell me that your own code is 100% bug free and does all the above (or some analog thereof) 100% of the time?

    Gimme a break! I know that I've probably written on the order of 1,000,000 lines of code over the last 35 years (proably more than that, I-dunno, the number ain't the point) and I've run into more WTFs in my own code. Subroutines that "shoulda worked" that didn't for one reason or another, forgotten switch cases, token types in a parse I wasn't expecting, you name it.

    You're complaining about something that was written by volunteers. That's silly. If it's broke and you have a fix for it, gently and politely contact the module's author(s) and let them know you have a fix. They might even incorporate it in a future release of the module.

    What I've described in a nutshell is the way Open Source is supposed to work and what is supposed to be the strength in Open Source. This is the difference, at least in my mind, between free, as in "freedom of speach" and free, as in "free beer."

    If all you do is take from the Open Source world and don't at least offer to give back in some form or another then, in my humble opinion, you've blunted your right to complain. Even if your not offering code to fix a problem, you could at the very least offer up to the author(s) your situation as a test case of where the OSS code in this case is going wrong.

    Hey... if code were 100% reliable 100% of the time us System Admistrator types wouldn't have jobs! :-D

    Just my US $0.02 worth....


    Peter L. Berghold -- Unix Professional
    Peter -at- Berghold -dot- Net; AOL IM redcowdawg Yahoo IM: blue_cowdawg
Re: Reliable software OR Is CPAN the sacred cow
by derby (Abbot) on Sep 15, 2006 at 13:36 UTC

    I never thought of CPAN as a sacred cow ... just the giant who's shoulders I stand on. I think a rite of passage in the Perl Community is when you can separate the wheat from the CPAN chaff. (All right ... that's way too many trite phrases for one post).

    -derby
Re: Reliable software OR Is CPAN the sacred cow
by Fletch (Chancellor) on Sep 15, 2006 at 13:39 UTC
    AND NO ONE THINK IT'S NEEDED - at least comments to my question say so......... "Yeah, why you think you need reliable email parser, what you will use it for? Forget about it, man, you don't need it. Nobody need it!" :(

    I think you need to re-read the replies a bit harder. They mostly boil down to "Try module X or Y; if those don't work contact the maintainers with test cases which exercise the problems you think there are".

    • davorg: "And if they don't, then the project members seem pretty open to receiving bug reports and patches."
    • davorg: "If you have specific areas where you know the existing modules have problems, then why not get involved with the Perl Email Project and help them fix those problems."
    • dave0: "Well, given that I use quite a few of those modules on a daily basis, I find it hard to believe that they're all "far away" from compliance. Do you have any specific issues to point out?"

    They're not saying live with the bad software, they're saying if you've got problems show how to reproduce them and let the maintainers fix them. If they don't, you're free to put your own überperfect code up on CPAN to shame them in the glorious open source meritocracy that it is.

Re: Reliable software OR Is CPAN the sacred cow
by b10m (Vicar) on Sep 15, 2006 at 13:40 UTC
    For me, ANY software MUST be developed with these priorities in mind:
    1. be reliable: it should do it work correctly (ex. for email parsing task it must support all features defined in RFCs related to email format plus be able to handle non-RFC-compliant emails

    POWER::Email is funny though, coming from mr. RFC-is-the-law:

    "Simple (and non RFC compliant) email sending"
    --
    b10m

    All code is usually tested, but rarely trusted.
Re: Reliable software OR Is CPAN the sacred cow
by dave0 (Friar) on Sep 15, 2006 at 13:43 UTC
    Wow, what a rant. Given that I have a limited number of hours in the day, I'll ignore most of it, but this part right here:
    Parsing emails is really needed task, it's text processing task and it's related to system administration. CPAN has huge amount of modules for this task, and no one of them support all email-related RFC. After so many years no one developed such module! AND NO ONE THINK IT'S NEEDED - at least comments to my question say so......... "Yeah, why you think you need reliable email parser, what you will use it for? Forget about it, man, you don't need it. Nobody need it!" :(
    needs a response. I guess you miss the entire point of the Perl Email Project, as pointed out to you earlier? The project aims to identify the current best-of-breed email modules in CPAN, and:
    1. Recommend them as the best option for those tasks
    2. Develop and improve them further
    So, here's at least one group thinking critically about CPAN, and working to improve it, in the particular area you're ranting about. The modules aren't perfect, nor will they ever be -- no software is -- but they can get a lot closer, and that's what we're trying to do.

    If you actually want better email modules, it would be a good idea to (politely) file bugs on those modules for whatever it is you claim is broken. Ideally, providing a patch, or a test case, but at minimum a description of the bug and how to reproduce it.

Re: Reliable software OR Is CPAN the sacred cow
by Albannach (Prior) on Sep 15, 2006 at 14:56 UTC
    Just to add my vote, I don't know anyone who treats the CPAN as a sacred cow, and I've yet to hear anyone say that even a single module in the CPAN is perfect.

    However, in my experience it is generally a far far better use of my time to pull pieces out of the CPAN than write them myself. Sure I may find spelling mistakes, bugs, or incomplete solutions, or even security issues, but I'm way ahead of the DIY or commercial options in that I have the source, and I have a community of other users who are generally more than passingly interested in hearing of problems and helping with solutions.

    For the record, I upvoted you on this thread because, despite your somewhat hostile tone, I think this discussion is a good one to have now and again, and I particularly agree that people shouldn't just put all their personal modules on the CPAN in some misguided quest for glory. I'd encourage you to add your best work though. If it is as good as you say, it would be both an important contribution the the community, and a good example of the care that should be taken in publishing modules. And I dare say that some users may offer corrections to your work from which you will benefit.

    I think that this is a case where yelling from outside is less effective than coming inside and lending a hand.

    --
    I'd like to be able to assign to an luser

Re: Reliable software OR Is CPAN the sacred cow
by powerman (Friar) on Sep 15, 2006 at 15:12 UTC
    I'll answer to all above comments here to avoid duplication.

    Most of your misunderstand me. Probably because my initial post was too emotional and overloaded with details. Here is short summary:

    • I'm NOT complaining about CPAN modules at all! Maybe sometimes is sounds like it, don't know, but I don't try to say this.
    • This node was just a question: is there exists reliable email parser module in CPAN (including RFC compatibility, because I consider this isn't optional feature for reliable email parser)? The answer was "No.". Ok, no problem.
    • Current node is meditation about right attitude to development: goals and priorities required to develop useful software. I say "useful" because, as I say before, if software isn't reliable and secure it isn't useful for me.
    So, we can discuss how may (10, 30) years of experience we've; we can discuss why I complain about CPAN modules instead of sending many more patches/bugreports than I send now; why my own modules non RFC compliant; etc. But I don't think this have sense.

    I wrote this to discuss another thing: what you think about right attitude, goal and priorities used while developing reusable software like CPAN modules. Do you agree reliability and security must be 1st and 2nd priorities, or not and why?

    blue_cowdawg: Are you trying to tell me that your own code is 100% bug free and does all the above (or some analog thereof) 100% of the time?
    Of course, no. But I'm always (at least in last ~6-7 years :)) trying to develop everything with reliability and security in mind first, and all other things next. My success in this area depends on task and my experience X years ago when I developed it.
    Fletch: I think you need to re-read the replies a bit harder. They mostly boil down to "Try module X or Y; if those don't work contact the maintainers with test cases which exercise the problems you think there are".
    Yeah, you right. But I speak about attitude. There no replies like: "Yeah, sadly, but there no reliable email parsers on CPAN now. Most promising is Email::* branch, you can contact it author, maybe he interesting in making his module compliant to all RFC you've listed here... But possibly it's just another 'worksforme' module, just designed to be 'recommended best practice' replacement to all existing modules."
    BTW, I've subscribed to PEP maillist and mail there my question, but looks like it dead.
    b10m: "Simple (and non RFC compliant) email sending"
    Yep. At least I include this note in documentation. This function was developed many years ago (in 2000-2001)... few months ago I wish to add GPG support and email parsing support, and found it realization terrible. I've checked CPAN. :( Then I've spend 2 weeks to carefully read and analyse all RFC related to this topic... I've begin developing parser, but then 'russian summer holydays' happens. :) So now I should return to this work... probably reread some RFC, etc. Complex, and distasteful task. So, I've tried to ask fellow monks - maybe I missed some module when checking CPAN few months ago. Sadly, but not, I don't missed anything. If I wish to have reliable email parser, I must develop it from the ground. :(
    dave0: The project aims to identify the current best-of-breed email modules in CPAN
    Good aim. But far away from mine. It's hard for me to believe in possibility to improve complex enough software developed without reliability/security in mind. Can you believe in reliable&secure Sendmail or BIND? I can't, that's why I use DjbDNS and Qmail... And looking for Apache replacement.
    Jenda: I'm kinda inclined to say "worksforme" IS good enough.
    Albannach: ... far far better use of my time to pull pieces out of the CPAN than write them myself. Sure I may find spelling mistakes, bugs, or incomplete solutions, or even security issues, but ...
    So, looks like we've two clean votes against 'reliable' and for 'worksforme'. I say "against" because if you agreeable to use 'worksforme' modules in your project then you even don't try to develop 'reliable' software. 'worksforme' works in both directions - if you think it's ok to use it then you think it's ok to develop it.
    Anyway, I consider these two comments only 'on topic' for now.
      BTW, I've subscribed to PEP maillist and mail there my question, but looks like it dead.
      That would probably be because the PEP people read Perlmonks and are either a) replying to you here, or b) not wanting to get involved in an unproductive flamewar with you when it's clear you just want to rant.

      I sadly fall into the category of a), but am now retreating to b).

        :-/ When I say 'dead ml' I mean 'dead ml' - no emails in maillist for more than 24 hours at all. I doesn't mean: I don't received reply on _my_ email :~(.
      Most of your misunderstand me.

      May I make a suggestion? That's not because we're dense. It's because you're bandying about words like "reliable" and "RFC compliant" without defining what you mean by them. And no, your code doesn't count, because, frankly, you're sitting on it. Why is it not on CPAN, if you've made such great strides in the programming realm?

      I read your original post yesterday. You utterly confused me, and I suspect a lot of people, by saying it must conform strictly to the RFCs...except that it must also understand mails that are not compliant with the RFCs.

      I've seen no links to bug reports from you on these modules. No precise definitions as to the problems you see, so we all can follow along -- few of us have the time to parse through a near-dozen RFCs and the 600+ modules in question, and you, apparently, already have this information. No commentary that you're willing to talk to the various maintainers of the modules about how you might improve them -- which is, after all, the point of Open Sourcing one's code. It's really not to come on a public source, like this, and bash the hard work of many before you.

      Let me ask you this -- have you actually talked to any of the developers, or asked for insight as to why they didn't conform to the RFCs? They are, after all, only Requests For Comments. I had a similar issue when I was developing an application to work with ICal files -- the RFC for ICal, and the reality on the ground with regard to Outlook's ICal files, was two slightly different things -- and it was folks from MS that wrote the damn spec! Naturally, I opted for interoperability over adherence to spec, and I strongly suspect you'll find the various Mail maintainers have done similar.

      If. You. Ask. And no, asking on Perlmonks does not equate to asking maintainers of modules.

      There no replies like: "Yeah, sadly, but there no reliable email parsers on CPAN now.

      Sure there are. The problem is, you have a very narrow, and likely unrealistic, definition of reliable. That's why we keep not answering your question; we're trying, very nicely, to point out the reality, based on people using these modules in Real Life situations -- the kinds where security holes, non-RFC compliant emails, etc., are ripped apart very, very quickly. You can either take that advice, or insist on conformance to specs and buzzwords. Your choice, mate.

      Ranting on here about "reliability" (which you've also failed to define, aside from some arguments about "Object Orientation" and "Security First!") isn't going to solve the problem, either. They are, in the end, just words -- as Microsoft's "attention" to Security in their products prove. Following the RFCs does NOT equate to a secure product -- and if you don't mean to co-mingle the two, please write so we can understand what you mean.

      If you think we're idiots who cannot write code, point out the lines of code that suck in the mail business. That way, we can actually solve problems, rather than play "whack the mole" with your issues and suggestions.

      I’m sorry for the rant, and I tried to be as nice as possible. But I’m utterly frustrated that you’re not listening to very good advice, the kind I wish I’d had access to when I first started working with Mail in Perl. I strongly recommend a listen, and deep re-read, of the commentary both here and in your original post. And then to ask deep questions, over insisting that your programming knowledge exceeds those who wrote the various modules, as well as those of us on Perlmonks.

      EDITED to clarify programming issues

      ----Asim, known to some as Woodrow.

        The problem is, you have a very narrow, and likely unrealistic, definition of reliable.
        Hmm. Very interesting. Maybe you right, maybe... but ... can you explain what's wrong with my definition of reliable?

        As least few people in the world develop reliable, by my definition, software - DJB, as example. For me it looks like problem isn't in "unrealistic, definition of reliable", but in people who doesn't try to develop really reliable software. They doesn't try, and so they doesn't got it.

        Anyway, I'm really interested in your opinion on this topic, so I hope you'll reply. Only few monks reply "on topic" here (and I agree it mostly my fault - I've mixed too many different topics and emotions in otiginal post), so every "on topic" reply I really appreciate.
        I'm here not to just rant my opinion, but mostly to calibrate it.

        That's why we keep not answering your question; we're trying, very nicely, to point out the reality, based on people using these modules in Real Life situations -- the kinds where security holes, non-RFC compliant emails, etc., are ripped apart very, very quickly.
        I can't understand.
        If you think existing modules are reliable, because they support all possible cases, both RFC compliant and Real Life - then why you doesn't reply on my yesterday quesion with names of such modules?
        Or you think existing modules are reliable, even if they doesn't support some emails (which comply to RFC!) because authors/users of these modules doesn't received such emails?
        Or you mean something else?

      I think your question is an interesting one:

           Do you agree reliability and security must be 1st and 2nd priorities, or not and why?

      I must admit, I fall into the 'works for me™*' school of thought -- but I'm open to being gently encouraged or corrected. Your original 'rant' and subsequent posts have not (yet) inspired me to change (mostly because they are long, somewhat rambling, and I don't have a lot of extra time today). I was surprised to see that your initial node is (or was, at the time of this comment) still in positive reputation territory ... because I think you come across as complaining or rebuking but not really offering a solution. Can you succinctly sum-up your arguments for why you think the bar of reliable and secure software should be raised, and how you propose to raise the bar on CPAN or within this community? What solution are you offering?

      Most of us are not against producing more reliable and secure software ... but you have to gently convince us that:

      1. we are not already doing so
      2. we (both individually and as a community) will gain something tangible by doing so
      3. there is a workable path to improving in this direction

      I'm a very pragmatic person, and I'm generally rewarded by delivering results on a day-to-day and week-to-week basis, which is partly why I love Perl. If I'm going to redirect bandwidth toward more secure and reliable software and away from delivering those immediate results, I'll need some justification.

      *works for me = Works within the context of my current environment. Currently, that means I am developing for internal users within the semi-safe confines of an intranet.

        Motivation:
        • I'm unhappy with current quality of 99.9% software both in Windows and Linux worlds. I just try to improve this situation by developing as reliable and secure software as I can.
        • This is more interesting and complex task than develop 'works for me' sort of software.
        • I'm lazy... so I hate to spend a lot of time debugging script X, just to find it not working because if bug in module A, used by module B, used by module C which I've finally used in my script X. To avoid/minimize such situations modules A, B and C much be reliable. (Actually, this was main reason why I've switched from developing under Windows to Linux many years ago - 3 days of debugging my Deplhi application until I've reinstalled Windows and found there no more bug now (reinstalling Delphi on day 2 won't help).) I wish spend my time thinking about right way to solve something and writing code, not for debugging. This, in turn, mean my code should be more reliable and less buggy, that's all.
        "we are not already doing so": just re-read my original post, there a lot of examples.
        "we (both individually and as a community) will gain something tangible by doing so": see motivation above
        "there is a workable path to improving in this direction": just change your goals to "reliability and security must be 1st and 2nd priorities" and keep this in mind while developing
      Can you believe in reliable&secure Sendmail or BIND? I can't, that's why I use DjbDNS and Qmail... And looking for Apache replacement.

      Ah, you're one of those people. Can I say it? Sure I can. I've been running both Sendmail and BIND on many systems, for over 10+ years, with tens of thousands of users and I sleep just fine. Never had a reliablity problem or a security problem.

      Worrying about security and reliablity are good things, don't get me wrong. Worrying about them TOO MUCH, at the expense of everything else is like only drinking pure distilled water and eating brown rice because anything else MIGHT cause cancer. You can do it, but please stop bitching because everyone else doesn't wear the same tin foil hat you are wearing.

      Frank Wiles <frank@revsys.com>
      www.revsys.com

        How much is TOO MUCH is depends.

        I'd like idea to move from Linux to Inferno OS each time I re-read Stevens APUE book or POSIX. But it's TOO MUCH for me, at least for now.

        But replacing Sendmail with Qmail and sleep MUCH MORE fine is ease and NOT TOO MUCH, for me. Replacing mc with mc-mp once, and forget about any bugs in mc also NOT TOO MUCH. Spend time once learning about GrSecurity, PaX, SSP and PIE, and then convert my system into Hardened Gentoo - NOT TOO MUCH. Use runit to supervise all services and forget about all problems with unreliable /etc/init.d/ script - NOT TOO MUCH. Etc. etc. etc...

        ... carefully develop own System() function and forget about all issues related to hang processes and unreliable signal handling in perl modules which need to execute some external process - not too much for me, sorry.

      I already recommended it, but I'll recommend it again.

      Read Worse is Better. And the followup article. And try to understand it.

      It is all about how the Lisp world's attempt to put reliability and security first resulted in nobody using Lisp. And it therefore gives an argument for not putting reliability and security first.

Re: Reliable software OR Is CPAN the sacred cow
by zentara (Archbishop) on Sep 15, 2006 at 16:13 UTC
    CPAN has huge amount of modules for this task, and no one of them support all email-related RFC. ....... AND NO ONE THINK IT'S NEEDED - at least comments to my question say so......... "Yeah, why you think you need reliable email parser, what you will use it for? Forget about it, man, you don't need it. Nobody need it!" :(

    You sound so intelligent and knowledgable on this topic, so why don't YOU write the module for the uber-superior absolutely foolproof email parsing.


    I'm not really a human, but I play one on earth. Cogito ergo sum a bum
Re: Reliable software OR Is CPAN the sacred cow
by Booger (Pilgrim) on Sep 15, 2006 at 16:35 UTC
    I probably shouldn't bother replying to this because I know it's just a flame... but ....

    I think you're a foisting idealistic concepts on what is almost an entirely volunteer operation (I say this knowing that there are a few fortunate souls out there who are able to convince their employer to fund development that eventually finds its way to CPAN). I've spent quite a few years working with and for non-profit organizations that depend heavily on volunteers and these experiences have shaped my understanding of volunteer-oriented organizations.

    The vast majority of volunteers give freely of their time and volunteering is only a part of their otherwise busy lives. You can't expect a volunteer to produce the same quality of work as say a professional engineer who is being paid to design a bridge. Sure, some volunteers will produce work equal to or sometimes even better than said engineer but this isn't something you should expect from them. Rather, you should be thankful that they're contributing in the first place!

    Secondly, a volunteer-based organization WOULD NOT EXIST were it not for the many volunteers performing their often thankless jobs. When working with volunteers you must be prepared to work alongside those who are highly skilled, those who are not and all manner of individuals, personalities and talents.

    Most importantly, don't castigate those who give freely of their time. You'll quickly discover that volunteers resent this and will flee from any organization that does. Volunteers aren't thanked enough.

Re: Reliable software OR Is CPAN the sacred cow
by tilly (Archbishop) on Sep 15, 2006 at 18:04 UTC
    I believe that someone has never read or understood the classic essay Worse is Better. There is also a good followup discussion. Do so.

    As for me, I am plenty aware of the shortcomings of CPAN. For instance one of the modules that you point to as being reliable, Carp, is reliable in large part because I made it so. (And I shudder at how much buggy code there is out there which is crap because it used the old $Carp::CarpLevel interface, which I had to jump through hoops to support because people were misusing it even though it was undocumented.) However I am also aware that most people who come asking for a solution simply won't come up with something better than what is on CPAN. Therefore for them the best answer is to use CPAN.

    Heck, I know that I generally won't come up with something better than what is on CPAN without doing a lot of work. If it isn't really important to me, then I should use CPAN as well. And I usually do.

    If this bothers you, then you should not be using commodity software. I would suggest not using Windows, OS X or Linux. OpenBSD might be up to your standards.

    A random note. You are unhappy that eval doesn't "support" source filters. Well I have news for you. If you care about reliable software, then in any language with a syntax that is more complex than Lisp, you should not use source filters. If you don't understand why, then you are not going to succeed in writing reliable software.

Re: Reliable software OR Is CPAN the sacred cow
by radiantmatrix (Parson) on Sep 15, 2006 at 18:19 UTC

    I think you're missing a huge part of the picture of OSS development, and what CPAN is. CPAN is a way to share code.

    Someone, one day, needed to parse mail messages from an internal system, and wrote a module to do it. It works reasonably well, but doesn't consider all of the edge cases. Still, though, they spent hours just getting what they have to work, so they upload it to CPAN so that no one else will have to duplicate that work.

    Yes, this means that most of CPAN isn't of the highest quality. That doesn't mean that the advice to "go find a CPAN module to do your task" isn't valid: it just means that it won't save you 100% of the work you need to do.

    You claim to have, on your website, better-quality versions of several modules which you felt were incomplete. The point behind CPAN is that it's collaborative: have you sent patches (with tests) to the module maintainers so that other people can benefit from your improvements?

    When we say "don't reinvent the wheel, use a CPAN module", we mean that you shouldn't re-do the work others have done. If no CPAN module quite cuts it, then download the closest one and build on it to complete your solution. Ideally, then, also contribute your improvements back to the CPAN module.

    The responses to your questions about mail parsing were not as you phrase them. They were "don't start from scratch, a lot of work has already been done and is available on CPAN: start there".

    <radiant.matrix>
    A collection of thoughts and links from the minds of geeks
    The Code that can be seen is not the true Code
    I haven't found a problem yet that can't be solved by a well-placed trebuchet
Re: Effective communication OR Is POWERman the sacred cow
by tye (Cardinal) on Sep 15, 2006 at 18:37 UTC

    Calling all of your modules POWER::* gets you plonked into the "kooks who don't 'get' CPAN" category. All modules must be well named. 99.9% of your modules have horrid names. Nobody of powerman cares at all about effective communication. A module that is 100% reliable is no good if nobody can find it and nobody wants to use it because the author comes across as a ranting loon.

    Complaining about the quality of what is on CPAN when none of your modules are on CPAN also costs your credibility points. If you think CPAN can be done so much better, then please prove it by actually producing a module that would be admired by the Perl community and could serve as an example of how to do CPAN modules right. But please don't upload modules named POWER::* and really don't upload a module with a totally useless name like POWER::Utils.

    Actually, I'm not sure that you uploading modules to CPAN is a great idea. I'm glad that you put a lot of stress on reliability and identify some features that can be important if you need very reliable solutions. However, reliability isn't the only desirable feature of a CPAN module and I don't think you'd be good at meeting the other desirable features. Good Enlish documentation is required. If all of your modules require POWER::Utils then I suspect you don't have a good enough grasp on modularizing parts of software. Then naming modules isn't an easy task and it is one that is very often not done well enough on CPAN, and I worry that your contributions wouldn't be an improvement on this front.

    Please do contribute to CPAN. But I think it would be better if you contributed to CPAN by, as clearly and politely as possible, identifying where existing modules fail in ways you consider important and working with the maintainers of those modules to get improvements made. I currently get frustrated that CPAN seems to stress ownership too much and contribution too little, so it will require your best diplomacy and communication skills to get your contributions successfully into existing CPAN modules. The best case would probably be if some module authors would let you write code (perhaps provided to them as patches). Alternately, you could find someone to help you with your weaknesses and produce reliable code while they ensure that it is more modular, better named, and better documented so that it would make a good CPAN module.

    I think you raise some important points. But I don't think you've raised them in a way that is going to do anyone any good. Software can't just be reliable. It must also "work for me", else it won't get used and all of that work making it reliable is wasted.

    I agree that CPAN in general would benefit from more attention to issues of reliability. It would be great if you could focus your energy on this point to actually get this improved reliability up on CPAN where people will find it and be able to use it is not just reliable but also "works for them". Doing that isn't always easy. I certainly can find many examples of where I haven't done this.

    - tye        

      You've just listed all reasons why I don't uploaded best of my modules to CPAN yet. POWER::* names and interdependencies.

      This happens because these modules was initially developed as non-free software, so POWER::* namespace was used to avoid possible conflicts with current and future CPAN modules.

      Now licence of these modules was changed to default perl module licence, and I'm spend time trying to fix interdependencies and then I plan to rename them to CPAN recommended names and upload to CPAN. Such operation has side effect on lot of existing scripts/projects which use these modules using POWER::* names, so I need some time to fix this first.

Re: Reliable software OR Is CPAN the sacred cow
by shmem (Canon) on Sep 15, 2006 at 18:58 UTC

    Dear powerman,

    almost all has been said so far in response to your valuable posting, but one point is missing.

    print "Howdy, world!\n";

    is one reason why people code in perl, which is really two, as expressed elsewhere:

    • you can babble in perl, drawl, jargonize, extemporize, speak volumes, drone, singsong, ad-lib, or eloquently express yourself with mastery; and
    • as the saying goes, "perl is the language to get your job done before your boss fires you." Corollary: "you do a good job as long as it's worth doing it well" (something like this is in the first Camel Book).
    Your way of speaking seems like trying to lay down the law.

    Another reason for coding perl is aptly demonstrated by

    $=[!print$=]=!map{$==$=[$#z]+=$=[@z]/@z,($=[@z]%=1+$#z--).=0}@z=@=for@ +==(1)x pop
    which snippet calculates the Euler constant (1st argument is number of digits - credit is due to a wizard on the #perlgolf irc channel (not me :-)).

    Another way goes like this:

    use HTTP::Cache::Transparent (); use LWP::Simple qw(get); use Compress::Zlib (); HTTP::Cache::Transparent::init({BasePath => "/tmp/http-cache-transpare +nt-$>"}); my $gzipped = get "http://www.perl.com/CPAN/misc/lwall-quotes.txt.gz"; my $quotes = Compress::Zlib::memGunzip($gzipped); my @quotes = split /\n%%\n/, $quotes; print $quotes[rand @quotes], "\n";

    Yet another splendid way of writing perl are the gems Erudil has posted.

    A flavour of perl coding which I particularly like and greatly admire is Lingua::Romana::Perligata --

    #! /usr/local/bin/perl -w use Lingua::Romana::Perligata; adnota Illud Cribrum Eratothenis maximum tum val inquementum tum biguttam tum stadium egresso scribe. vestibulo perlegementum da meo maximo . maximum tum novumversum egresso scribe. da II tum maximum conscribementa meis listis. dum damentum nexto listis decapitamentum fac sic lista sic hoc tum nextum recidementum cis vannementa da listis. + next tum biguttam tum stadium tum nextum tum novumversum scribe egresso. cis

    With these examples I'm trying to mark (some) cornerstones of what perl is, and all of these are good uses of perl and splendid ways to make life interesting.

    Ah, and don't let us forget the Acme namespace - that's all more Monty Python than any so-called python (scnr).

    <update why="the point">
    None of these coding examples are in any way meant to be reliable in other sense than "if it ain't broke, don't fix it".
    </update>

    I've upvoted your post, partly because it does adress a valid concern, partly because it rouses a good discussion, and partly because it's good if from time to time a pilgrim comes back to the monastery to shout

    YOU ARE ALL SINNERS! REPENT!
    at the fellow monks sitting at the table. I hope you'll not loose your harsh tone and your engagement in the quest (feel free to set up the perlgems.org domain - it's untaken at the moment - to publish the Carat weight of CPAN modules you come across :-), but don't go down the wrong way, don't end up like Jorge of Burgos who died poisoned by his own wrath.

    Anyways - if you want really reliable code - you probably shouldn't use perl at all. Perl comes with all the ambiguities natural languages have. You should take the right tool for that, C perhaps, or assembler, and scrutinize the libraries your code uses, the underlying OS and the chips that OS runs on.

    Last thing - CPAN isn't a holy cow at all. It's more aptly described as an ecosystem - there are so many interdependencies of modules, which reliably infest each others with parasites, pollution, bugs and even sudden death, cot death... there's no way to stand that but with the firm belief in your own open eyes, skill, and the good intent of your fellow coders.

    Your good intent is highly welcome .

    --shmem

    _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                                  /\_¯/(q    /
    ----------------------------  \__(m.====·.(_("always off the crowd"))."·
    ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
      Actually, I'd say that CPAN *is* a Holy Cow...

      As in, "Holy Cow! Look at all that code!"
      Or "Holy Cow! I didn't think of doing it *that* way!"
      Or "Holy Cow! I can actually talk to the person that wrote this!"

      As many have pointed out (and I'm paraphrasing), CPAN isn't some faceless corporation where you are only a consumer of product... CPAN is *real people*.

      And that's a good thing.

      -- WARNING: You are logged into reality as root.
        It's people. CPAN is made out of people. They're making our code out of people. Next thing they'll be breeding us like cattle for code. You've gotta tell them. You've gotta tell them! CPAN is people!
Re: Reliable software OR Is CPAN the sacred cow
by Unanimous Monk (Sexton) on Sep 15, 2006 at 19:45 UTC
    I wrote this to discuss another thing: what you think about right attitude, goal and priorities used while developing reusable software like CPAN modules. Do you agree reliability and security must be 1st and 2nd priorities, or not and why?

    If that is truly what your asking, you sure went out of your way to hide it...

    Reliability and security must be 1st and 2nd priority to all CPAN authors? Of coarse not, generalizing any statement like that is folly. Blindly deciding to make reliability and security a priority without taking into account the purpose of a module is going to result in poorer programming in the long run. Each module's priority should be based on its purpose.

    For example, why should the author of Acme-Apace-Werewolf make reliability and security a priority of his module? Technically, it doesn't even need to compile to serve its purpose.

Re: Reliable software OR Is CPAN the sacred cow
by eric256 (Parson) on Sep 15, 2006 at 20:54 UTC

    First Point:Reliability and securtiy mean different things in different contexts to differnt people. Depending on the different modules, no that isn't and probably shouldn't be the first goal. If a module is reliable and secure but unusable what have you gained?

    Second point: Module authors goal match the goals of the project they were developing when they decided to code the module. Which normaly falls in the "get it done" category along side the "works for me" category. These don't mean the module wont be secure and reliable, it simply means that the level of security and reliability will match the goals of the project, not some preset goals for releasing software.

    Final Point: If cpan was started with only "secure" and "reliable" modules with no room for immature modules then you significantly raise the bar of entry. People who would have published modules would not have, they wouldn't have learned and grown an we wouldn't have the better modules they latter produce or the modules inspired by their first attempt. CPAN and perl would be poorer for the loss not richer, at least that's how it looks from here.


    ___________
    Eric Hodges
Re: Reliable software OR Is CPAN the sacred cow
by Anonymous Monk on Sep 16, 2006 at 00:56 UTC
    Bloody DJB fanboys and their useless rants! You guys pop up every now and then on any programming-related mailing list or forum, spew forth about how nobody understands "secure" programming except for you and the celestial DJB and then vanish again because it takes you six months to code a secure way of opening a file.

    Answer me this, oh powerful one, if your esteemed deity's way of programming is so ultimately superior why does noone (not even the vocal members of the qmail and djbns mailing lists) use the code as he wrote it? They all use extensions and enhancements someone else added (which by the somewhat insane license DJB uses would be illegal if anyone cared) or rewrite the whole shebang themselves because otherwise these apps might have been useful last decade but certainly are not now.

    So yeah, go ahead, program your own POWER::SUPER::IAMGOD:: namespace as long as you like and revel in your superiority. The rest of us will happily go on using modules that actually do the work. Feel free to climb down from your tower whenever you want to contribute to the real world.
      why does noone (not even the vocal members of the qmail and djbns mailing lists) use the code as he wrote it?
      You wrong about "noone", but this isn't a point, so... Because people need "features". Most of such features can't be coded in reliable&secure way because of their nature. DJB prefer to not use such features, at least until somebody invent secure&reliable way to develop them. Anyway I suppose while DJB's software with few patches is less secure&reliable than without these patches, it's still much more secure&reliable than most of other software. AND - you've choice: use more reliable less features version or less reliable more features version. I think this is goodness.

      BTW, I'm partially agree with you - developing really reliable&secure software sometimes require too much time. And this is one of reasons why I writing here - I wish to calibrate my feeling about this, to avoid developing reliable&secure email parser in next 2 years.

Re: Reliable software OR Is CPAN the sacred cow
by bigmacbear (Monk) on Sep 16, 2006 at 01:05 UTC

    There is an observation that applies to all sorts of engineering, research, and product development:

    Good. Fast. Cheap. Choose only two.

    What happens is that in the real world there are always trade-offs between quality, development time, and cost. Essentially, there is a finite pool of resources available to a project, and if you need more of one thing, at least one of the other two (and possibly both) will suffer.

    On a commercial project, if you need reliability*, you will have to pay for it either in a longer development cycle or in paying more programmers (or both). If you need working software yesterday, you can hire an army of programmers or skimp on the reliability of edge cases. And if you want the cheapest software on the planet -- free (as in beer) software -- both development time and quality will suffer as a result.

    In essence, our pool of resources is limited to the available time of the volunteers who maintain that particular project. Therefore, if the quality of a particular module isn't up to your standards, you will need to look at getting that project the resources it needs, and the surest way to do that is to do the work on it yourself and contribute to the community. In terms of the trade-off above, Good trumps Fast and Cheap in your opinion.

    That said, most programmers (myself included) are quite content to use what is already available and live with the limitations. There's nothing wrong with that, it's simply an issue of priorities -- Cheap trumps Good and Fast every time.

    *Or you're working on something that will kill people if any bugs are left in the final product, which if that is the case you shouldn't be using Perl.

Re: Sacred Cow (was:Reliable software OR ...)
by Intrepid (Deacon) on Sep 16, 2006 at 01:17 UTC

    powerman wrote (Sep 15, 2006):

    Few days ago I've asked about Reliable email parsing and that discussion uncover some points:
    • People doesn't need reliable perl modules, worksforme is enough.
    • People even doesn't understand what reliable solution is good thing which must be first priority goal (even if you can't reach this goal right now)!
    • CPAN == goodness. Use CPAN module - is only recommended solution for anything. If you say all existing CPAN modules for {any task here} is wrong because {any reason here} - you got a lot of '--', no matter is you right or not!

    I downvoted this post -- not the poster -- because it signifies an erroneous mental process; the poster reached incorrect conclusions. I downvote those incorrect conclusions. In brief here's what's incorrect (and btw I've gone through and upvoted the other reply nodes that say the same thing ;-).

    • People at Perlmonks "don't do anything in reply to questions but say use such-and-such CPAN module for that" (paraphrased). (BTW, paraphrasing isn't "putting words in someone's mouth", especially when the original words were barely parseable English to begin with. Nonetheless I will of course welcome and acknowledge corrections from the OP if he feels that I got anything wrong).

      It's a grossly inaccurate generalization, is what's wrong. The very diverse Perlmonks membership comes up with many different sorts of recommendations in response to questions.

    • People at Perlmonks "claim all CPAN modules are reliable solutions". Nowhere. Never. Well, hardly ever. In fact it's a basic and established understanding among knowledgeable Perl people that CPAN contributions vary widely in quality, from the sublime to the absurd. If that informal but widely referred-to (directly or indirectly) understanding never percolated through to the OP, that's too bad. Maybe it does need to be stated more often. But in direct answer to the direct question "Can I rely on it just because it comes from the CPAN?", I've heard no one ever reply "Of course" or anything like it.

    • "People even doesn't understand what reliable solution is" =>
      "People don't even understand what [a] reliable solution is" ...

      Wrong because it is also grossly generalized, but partly true because it is grossly generalized ;-). Perlmonks possess all sorts of degrees of understanding about what quality is, in perl code. Some here have demonstrated the highest awareness of the value of workmanship and pride in one's craft, and some here fall far short of that. Some haven't yet really tried to show what they've got it in them to accomplish, because, perhaps, they fear to fall short of an ideal. In general, though, one thing that characterizes the concerns of people in the Perlmonks community is quality and correctness, far more than you'll find in nearly any other Perl-oriented project or site anywhere.

      I'll just point out that one thing which demonstrates one's own pride of workmanship is the degree to which one makes efforts to create clear and readable documentation for one's own projects, and by extension, to ask questions or comment to others in clear and readable lingua franca of hackerdom (that is, English). A NNES* could ask for someone with better skills to proofread his-or-her writeup before posting. That would set a good example, would show walking-the-walk instead of just talking-the-talk.

      * NNES: Non Native English Speaker

    As for what the OP (powerman) got mostly right:

    • Yes, "they all are low-level modules, each doing small simple task" has considerable truth to it. And that actually makes sense, and is in keeping with the original Unix philosophy of having a rich toolkit of many small tools which is each specialized and optimized for its task (and then connect them together into larger assemblies as needed). Perl came from the Unix world. It's sadly true but also perhaps not unexpected that quite a few modules that work on a higher level, which are on CPAN, are not of such high quality. Some are not maintained -- they workedforher for the original author but that author wasn't fully committed to portability and bug-stomping (which can end up being nearly a full-time job in itself, or seems that way at certain moments).

      I say "ANY software", but especially this important for all reusable things like perl modules and core things [...
      All Perl modules are not "equally reusable". That's a mistaken idea. The basic notion of modularity in Perl is simply the facilitation of extending the language to accomplish different tasks in better ways. "Tasks" in turn are rationally categorizable into "less" or "more" specialized. The more specialized the task, by definition the fewer users need to do it. No?

    There are problems with CPAN and over-reliance on it, I agree. I've seen it grow worse recently. Newer modules are being released which, while not especially low-level (i.e. they are more specialized) are nonetheless of interest to a lot of people, and these modules in turn are relying on prerequisites (dependencies) on CPAN that are failing tests on at least some platforms and aren't being fixed. There's a "quantity over quality" attitudinal problem on the part of some of the most prolific recent CPAN contributors that is degrading the overall reliability of the Perl-CORE+CPAN system and it shows up most negatively where the modules in question pertain to Perl admin and development itself, i.e. building, packaging or installing Perl modules in various ways. These tools need to be the most tested and robust (IMHO) because getting all other CPAN offerings relies on them to be working. I don't know what to do about this trend. Partly I hope it will just correct itself.

        Soren A / somian / perlspinr / Intrepid

    -- 
    Words can be slippery, so consider who speaks as well as what is said; know as much as you can about the total context of the speaker's participation in a forum over time, before deciding that you fully comprehend the intention behind those words. If in doubt, ask for clarification before you 'flame'.
Re: Reliable software OR Is CPAN the sacred cow
by CountZero (Bishop) on Sep 16, 2006 at 08:48 UTC
    Did your version of Perl come with a guarantee that the CPAN-modules would conform to your idea of a code repository? Mine didn't.

    I'm sure there are other programming languages that come with promises of inherent reliability and security ("Eiffel" springs to mind), so why don't you go there and preach your gospel to the converted over there?

    My experience is that the vast majority of the Perl community can work reliably with CPAN, so it becomes a matter of "I'm not wrong the whole world is wrong".

    CountZero

    "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

Re: Reliable software
by powerman (Friar) on Sep 16, 2006 at 13:52 UTC
    I am much obliged to all monks here. Now I've answer which I looking for in my initial post!

    My initial post was very incorrectly formulated, but that's because to formulate question correctly you need to know at least half of answer, which I did't know at that time.
    My initial post (and most comments) was deliberately designed as rant, because I've feeling I need much more than usual 3-5 comments to find answer. I'm sorry about this, but it's worked! :)

    Now, in hope this helps somebody in the future, I'll repeat my question (better formulated and without rant), with answer.

    Q: Why not develop everything with maximum possible reliability and security? Isn't world become better if our OS will be designed like Inferno and all software will be as reliable and secure as DJB's?
    A: Because there different goals in development:

    • If you need to "have your work done", then "works for me" is right way.
      This way your software will be as reliable and secure as spec require, no more and no less. If you wish to improve overall software quality in the world in this case, you must try to increase spec requirements to reliability and security (i.e. go talk with your boss/customer).
    • If you need to get maximum possible reliability&security, like DJB do, then you need to have a lot of time, experience, and be able to change spec if it require something you can't develop as reliable&secure as you wish.
      This is more like intellectual game. Such sort of development doesn't even guaratee you'll "have your work done" at some point, a good example of this is unfinished DJB's Internet Mail 2000 project. Also it doesn't guarantee your software will be useful for somebody, because people may need features which you've removed from spec (and that's why we've so many custom patches for Qmail).
    There also exists slightly more abstract answer. If you experienced enough, then you able to develop software per ANY spec. If spec will require DJB-like quality - no prob. If spec will require low quality cheap software developed very quickly - no prob. This way we've no more different goals, just different specs. And it's always up to you, agree to THAT spec, or not, and look for different work instead.
      My initial post (and most comments) was deliberately designed as rant, because I've feeling I need much more than usual 3-5 comments to find answer. I'm sorry about this, but it's worked! :)

      I don't know about the rest of the Monks here, but for myself? I respect those who respect me. And ranting on here, during my time, to "get a rise"? To get a "full answer". That's not respecting the other people on here. Especially as I take time from my actual job to answer questions on here, many times.

      If you had asked for help in a respectful manner, you would have received a ton of replies that would have pointed you to source after source. If you had chosen the path of asking honest questions, rather than posturing, you would have received honest and detailed answers that would have not only enlightened you, but saved you time and energy.

      I'm going to save my reply to your questions to me for another day, perhaps. I'll close this by passing on something I keep in my Personal Nodelet, Spirit of the Monastery. I can only recommend a reading, and pondering, of it's words, as I do, from time to time, before posting.

      ----Asim, known to some as Woodrow.

      My initial post was very incorrectly formulated, but that's because to formulate question correctly you need to know at least half of answer, which I did't know at that time.

      You've answered your own question why people put unreliable and not-secured code in CPAN: because to formulate something correctly you need to know at least half of answer, which you did not know at that time. People put code in CPAN for different reason and different purpose and different contexts. Therefore their priorities are different, while in your case, it's reliability and security.

      My initial post (and most comments) was deliberately designed as rant, because I've feeling I need much more than usual 3-5 comments to find answer. I'm sorry about this, but it's worked! :)

      You've just made a very bad reputation of yourself by doing this. There are a lot nicer ways of getting a lot of comments from fellow monks than what you've done here. Social skill is a lot more important than programming skill when you are in a community, online or not.

      -cheepy-

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://573100]
Approved by talexb
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (5)
As of 2014-07-12 10:34 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    When choosing user names for websites, I prefer to use:








    Results (239 votes), past polls