Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re (tilly) 4 (gotcha): Supersplit

by tilly (Archbishop)
on Dec 31, 2000 at 13:33 UTC ( #49090=note: print w/ replies, xml ) Need Help??


in reply to Re (tilly) 3: Supersplit
in thread Supersplit

I hate having to say, "Oops".

Everything that I said about BEGIN is true. But my promise that Perl 5.8 will have a version of Carp which fixes the begin-time Exporter issue is incorrect. Here are some details on the situation.

Suppose that you have 3 packages, A, B, and C. Suppose that B and C both inherit from A but C does not inherit from B. Today when C makes a call into B and then B makes a call into A which generates an error, Carp will not label the error as coming from A, B, or C, but instead will report it from whoever called C. In 5.8 the error will in this case be reported from C because Carp has not been told that B trusts C.

This is good because the fact that B inherits from A is an internal implementation detail which C should not have to know about. What is not good at the moment is that the trust relationship is synonymous with "inherits". But the code for Carp has been written in a way where this can be fixed by just replacing the trusts_directly() function in Carp::Heavy with something smarter. This is all intended.

What I had not thought through, though, is the situation where C calls a method in B, which actually is inherited from A. Now when Carp looks at the information, what it sees from caller is that C called a function in A. But C trusts A, so that call is trusted. In fact if the call generates an error, then C should be fingered because B does not trust C, and where the error is reported from should not depend on that implementation detail. But the critical piece of information required is not reported by caller, on a method call we don't know what call was intended, we only know what function was called. If we had a way to get that information, then set $called to the appropriate thing in the short_error_loc() function in Carp::Heavy, and the underlying problem would go away in Perl 5.8. But unless someone feels motivated to champion this on p5p, that won't happen and in this situation the error will continue to be incorrectly reported.

This last situation is exactly what we get with Exporter and playing games with @ISA during compile time. If you do not play the games, then while your code is being compiled you do not inherit from Exporter during your use invocations, so you can be correctly fingered as causing errors. If you do set @ISA bright and early, your use of other packages will go to Exporter's import() function, and then when Carp tries to figure out the error it will decide not to finger you because what it sees is you calling a function in a package you inherit from.

An incidental note. Capturing information about method calls in caller would allow that information to be properly propagated in Perl's SUPER mechanism. This is a necessary step towards having Perl's SUPER mechanism play nicely with multiple inheritance. Of course making that change in Perl 5 raises backwards compatibility issues, so likely a new keyword would need to be created. That would require even more championining. Since I am personally of the opinion that multiple inheritance leads to overly fragile and complex design issues, I won't be seen caring much either way, but some others might...


Comment on Re (tilly) 4 (gotcha): Supersplit
(tye)Re2: Supersplit
by tye (Cardinal) on Jan 01, 2001 at 11:20 UTC

    First, my comment about using qr// was specific about when writing modules. If you are writing modules then you should consider other peoples' environments. So I think the minor effort to support what are still quite common versions of Perl to be good manners.

    After all of the above, I am even more convinced that:

    • Inheritance should not be used for Exporter.
    • use base and initializing static variables in BEGIN blocks is a good idea.
    • Having Carp.pm pay attention to inheritance is a bad idea.

    The documentation for Carp.pm is quite simple, saying that the error will be reported as coming from the caller (not from the caller of the caller of the caller etc. for as long as it takes to get out of your inheritance heirachy). I think this is a much better idea.

    My first run-in with Carp.pm skipping more than one level of the call stack kinda made sense since it allowed your module to have a helper routine that didn't check all of its arguments closely and could pass on bad arguments to an underlying routine and in that specific case, it makes sense to have the error report two levels up. Well, sorta. At least it doesn't help much to have an error reported in your helper routine.

    I say "sorta" because I'd better be pretty careful in my underlying routine to word my error message clearly. It doesn't do much good to have code fred("bar") report "_fetchField() called with undefined field name". So I think that in most cases, even the best case I that can come up with, skipping more than one level of call stack doesn't really give a better message than just simply always skipping exactly one level would do.

    I think it makes much more sense to let those specific error messages (that are very carefully worded and carefully chosen such that they really were caused more than one level up and will be understood when reported more than one level up) should call carp() in such a way that carp() knows how far up the stack to climb.

    Now skipping the whole inheritance heirarchy is just madness. So everyone who uses HTML::ParseTree (or whatever some module is that expects the user to use inheritance in order to make use of the module) should be getting their programming errors popped up to the next higher package instance?

    I can see wanting carp() to skip a couple levels of inheritance heirarchy in very rare cases where I have several packages that are developed together, probably in a single source file, and some specific arguments are not checked until we get a few levels deep. Then only errors about those specific arguments should be allowed to bubble up the appropriate level.

    But just because I inherit from a class certainly doesn't mean that I should avoid checking my arguments for validity before passing them along to that class. But this seems to be what the designers of Carp assume, no? If I inherit from someone else's package, I really do want to be told when I use their package wrong. I will write my code such that errors passed in by users of my code will be detected by my code, not blindly passed on to the other module.

    So the fact that Exporter has a "bug" in that it doesn't take into account the bad design flaw of Carp and this bug can be worked around by making the inheritance tree inconsistant over time, isn't really a good argument for encouraging such inconsistancy.

    I think you may be too caught up in the whole Carp mess. (:

    And all of this just enforces my commitment that lots of things go on between declaring @ISA and run-time initializing @ISA and it doesn't make sense to leave your inheritance in an inconsistant state during all of this processing.

    Sorry, I don't have time right now to dig up the easy way to break things when you don't use BEGIN or use base. But several other people got convinced enough that patches were issued such that BEGIN blocks were actually documented as the proper way to initialize your package variables like $VERSION and @ISA.

            - tye (but my friends call me "Tye")
      I think we are agreed on the qr// idea. I just wanted to mention that it was possible, not advocate its use.

      So now on to the more substantial BEGIN issue.

      Having read and thought about what you wrote, the following points summarize my current position fairly well:

      1. I form my own opinions. Telling me what other people have been convinced of is a waste of time. Give me examples and reasons please.
      2. Complaints about poorly written error messages don't go very far with me.
      3. Our differing views about debuggers explain to some extent our disagreements on error messages.
      4. There are valid reasons for having a standard way to have error messages reported from elsewhere. There needs to be some flexibility in what that elsewhere is.
      5. It is a bad idea to make wrappers know a lot about what they are wrapping or vice versa.
      6. It is a bad idea to make the programmer have to be explicit about where exactly the error should be reported from.
      7. Inheritance is the wrong handle to use for deciding where errors get reported. But it is at least a relative of the right concepts.
      8. I still think that having user code play cute games with the internal definitions of what happens during run-time and compilation is a Bad Idea.

      More on those points.

      1. I form my own opinions. Telling me what other people have been convinced of is a waste of time. Give me examples and reasons please.

        This is how I have always been. The fact that other people think something doesn't tell me why I should think that. Perhaps when I think about it I will come to a different conclusion. More importantly, knowing why I believe what I do is more important than having some rote rules to learn from.

        For instance Perl 5.6 introduces the idea of 'our'. A lot of people - including many who I know are better at both Perl and programming in general than I am, think that it is a great idea. However that didn't stop me from thinking about it and deciding to respectfully disagree.

        Now perhaps it is a great idea. If so then by forming my own opinion and putting it forth I get to find out something about how I don't understand programming in a useful way. The opportunity for that is more important to me than having a stupid opinion from time to time. And not doing that leads to Cargo Cult thinking.

        So rather than tell me that some example out there convinced some people to have the official documentation give me the examples. Besides which I note that the current version of Perl (ie what I see in 5.7 snapshots) does not recommend using BEGIN blocks. Or suggest base. So not everyone can have been convinced.

      2. Complaints about poorly written error messages don't go very far with me.

        This should be a, "Well, duh" item. In Perl there are many different options for reporting an error. You can die. You can warn. From Carp you can choose to carp, croak, cluck, and confess. The only one of these that I have not used in production code is cluck. Each one makes sense in different places, and what message you should write depends strongly on which one you are using.

        Complaining that people don't know how to write useful error messages sounds to me like a user education issue. If you don't know how to write an informative message then it doesn't matter what function you use, your errors will be uninformative at best and seriously misleading at worst. So learn to write better messages.

        For the record from most to least frequently I use confess, croak, die, warn, carp, and cluck. When I am writing stuff that others will use I move croak and carp up in that list.

      3. Our differing views about debuggers explain to some extent our disagreements on error messages.

        I was trying to figure out why we disagree. And this is the best explanation I came up with.

        As we discussed in Are debuggers good?, I prefer to write good error messages and then depend on them. You prefer to use the debugger. This difference in philosophy affects our view of error messages. If the error message gives a good place to start in the debugger, it is likely useful for you. I want it to give me me much more than that. I want it to finger a piece of code and give me information on why that code in particular is likely to be the cause of the problem.

        As a result I demand far more out of my error messages than you do. So I put (and expect others to put) far more energy into what is reported where and when. And into deciding what is reported how. Which leads to an interesting cycle. Because I put that energy in, I constantly find that errors are pinpointing my exact problem in a useful way, and I do not have the desire to use the debugger. The same happens in reverse for you.

        Or at least that is the best theory I can come up with at the moment.

      4. There are valid reasons for having a standard way to have error messages reported from elsewhere. There needs to be some flexibility in what that elsewhere is.

        There is a world of difference between where an error is found and what the most useful place to report it from is. What both croak and carp are for is having a library report an error from the context of a user. If the library does this then it is up to the library to provide enough context about the error to make the report useful. (The lazy way to provide context is, of course, to just confess to your sins.)

        The reason for this is that whoever is maintaining the user-code should not need to know anything about the internals of the library to figure out the code. And a real-world useful library is unlikely to just belong in one package. For instance look at LWP. So if you encounter a problem (eg you cannot resolve the site you are trying to connect to) that has nothing to do with your library, deciding where to report it by reporting the error as soon as you get out of your current package is likely to be useless.

      5. It is a bad idea to make wrappers know a lot about what they are wrapping or vice versa.

        Very often people want to wrap something else. For instance XS wrappers to allow people in Perl to use C libraries, the DBI wrapper to give a wide variety of DBD drivers a common Perl interface, so on and so forth. This is both a common thing to do and often a good design idea. This should be encouraged, not discouraged.

        When you insist that the wrapper should know every possible thing that can go wrong inside the thing wrapped and handle all of them, you have just made wrapping a much more complex thing to do, and are discouraging wrappers. If I wrap a socket connection with an interface that makes ftp requests transparent, and then someone uses my code, there is no reason on Earth that I should have to do my own tests that the connection can be made or won't break. Not only is there no good way for me to do that, but I am reusing code that (hopefully) already does that for me - why should I reinvent the wheel?

        Conversely the thing wrapped shouldn't need to care that it is wrapped. After all you have an interface into it, not vica versa. If it is written well it should not be making assumptions about how it is being used. If it is making such assumptions then it is preventing itself from being easily reused.

      6. It is a bad idea to make the programmer have to be explicit about where exactly the error should be reported from.

        Well you know my opinion about the stupid misuses of $Carp::CarpLevel in the past. First of all when proper coding depends on programmers always getting things right, that is guaranteed to go wrong often. For instance the most common security whole in every year of the last ten has remained the basic buffer overflow. It isn't hard. Don't put more into a data structure than it has room for. But people simply cannot get it right. (But the ever growing popularity of languages like Perl that do dynamic allocation and so prevent buffer overflows may someday change this.)

        Well the one thing I don't want to see messed up on a regular basis is error reporting. OK, you have to ask for errors to be reported. But if you require people to also have to synchronize information correctly, they will get that wrong. Guaranteed. They will do things like hard-code the level. (Which Carp and perldb.pl both do with $Carp::CarpLevel.) Or they will write their own nifty functions to figure out where they think the error should come from. (warnings.pm does this.) And they will get it wrong. Do you want proof? Just look at how regularly the Perl core has messed up on this given the opportunity! And the Perl core is supposedly filled with competent people!

        In particular woe betide the poor joe who wants to wrap a library that for no particular reason chooses to be special and do it differently!

        Software design involves a constant balancing act between what complexity needs to be exposed in your interface, and what interface aspects are going to be the cause of ongoing problems. In this case my strong opinion is that asking for libraries to know enough about their environment and how they are used that they can correctly decide where it is reported from is just begging for a nightmare.

      7. Inheritance is the wrong handle to use for deciding where errors get reported. But it is at least a relative of the right concepts.

        I am not a mind reader. I am not even positive who wrote the original Carp. However I had to think through a lot of the decisions made, and while I have to say that the original was a hack, it was a hack that caught a lot of important ideas.

        The use of inheritance in Carp captures the following insight. It should not matter whether a method was inherited or overridden. That is an implementation detail. The rewrite takes that basic insight and makes its application more consistent. But it is still retained. If it matters to you whether a given method was over-ridden or inherited, then perhaps inheritance is not what you are looking for.

        As you rightly point out, this does not generally hold. Here are several classes of issues:

        • The module (like HTML::ParseTree) expects you to inherit and subclass from it.
        • You have a has-a relationship rather than is-a. (This is very common in wrappers, and we both agree that using has-a is usually cleaner than relying on is-a.)
        • The library (which is complex enough to be spread across several packages) is not object-oriented.
        I claim it as a bug in Carp that it does not have a more flexible declaration mechanism for handling these cases. I don't think that coming up with one is by any means impossible. However when dealing with reasonably mature packages (eg stuff off of CPAN) it will often turn out that if package A inherits from package B then they shouldn't really be reporting errors in each other.

      8. I still think that having user code play cute games with the internal definitions of what happens during run-time and compilation is generally unwarranted, and for this case in particular is a bad idea.

        This is where we started. I am personally a fan of trying to KISS. Have what people are likely to do pretty much by default work well. If you can, make it work right to have people consistently do the simple thing. If you can't then make it a big deal.

        Now I have no idea that if code that doesn't play cute BEGIN games relies on code that does, you can get all sorts of fun games. But my answer to that is to not play cute games in the first place, not to play more.

        And in this case in particular you will guaranteed get wrong behaviour from both old and current versions of Perl (as well as versions likely to be released in the near future) by playing games with BEGIN. So Don't Do That. And if you do do that and get hurt, Don't Complain.

      OK, so this is far, far more than I intended to write, but it hopefully is interesting. I don't think that the concept of Carp is broken. I am emphatically unconvinced that BEGIN games are justified. And I have just said far more on that than I should. :-)

        I prefer to write good error messages and then depend on them. You prefer to use the debugger.

        ...over writing good error messages? No. And I never said that. That is quite a big jump you've made ): I just said that I prefer using a debugger over adding temporary trace.

        But I did figure out some of our disagreement. I'm really only talking about Carp::croak(), which I feel was meant for what I call "precondition failures". In many other languages, the simple preconditions for a function are that you specify the right number of arguments and that they each have the correct type. These are checked at compile time.

        Perl doesn't offer such compile-time preconditions most of the time (rarely you can reasonably use prototypes for this) so a Perl programmer who includes lots of error checks is going to generate fatal errors when a function gets the wrong number of arguments, for example.

        This type of error is extremely likely to just be a simple coding error in the immediate caller of the function (more on this later). So it makes sense to have Carp::croak() report the error as having come from the immediate caller and including a stack back trace is not likely needed.

        From the documentation, this seems to be what Carp::croak() does and what it was intended for. But it appears that the documentation is wrong and Carp::croak() looks at inheritance to decide how far up the call stack to go.

        Now, I can see some use for generating warnings that seem to come from the user code and can see using inheritance to figure that out (imperfect but not too unreasonably so)1. But a fatal error is a different matter.

        For the module to try to kill the script, there must be a serious problem. These usually amount to precondition failures or assertion failures. Simple precondition failures can be pinned on the immediate caller (modulo wrappers to be discussed shortly) because the source of the error is almost always right there. Complex preconditions should include a stack trace (which can skip the current function context if you like). Assertion failures should also include a stack trace. That is because we may suspect that the user code is at fault but we shouldn't delete useful information about the state of the module even though we don't think that this is the source of the problem. If we know that the user code is at fault, then we should be returning a failure indication (or throwing an exception), but not trying to kill the script.

        I have this tiny suspicion that Carp::croak() used to work this way but then had patches applied that didn't include documentation changes. But that is just a tiny hunch.

        Now, Perl doesn't have a preprocessor so you often end up with simple wrappers in your module (that don't check their arguments before passing them on) so the simple preconditions might be due to code more than one level up the call stack. Since a module should be tested before it is released, there shouldn't be any precondition failures that are due to code in the same module. So it makes sense to bubble up within the module.

        Now, for some huge packages of modules like LWP or CGI, there are lots of modules developed together. Well, CGI never croak()s. LWP croaks a whole 6 times. Either force it to specify that each or all of its calls to Carp::croak() should skip more than one package or have Carp.pm use inheritance along with the fact that all of the package names fall under the same hierarchy. The few big packages of modules shouldn't be setting the behavior for Carp.pm when it is easy (like it is) to find cases that this behavior breaks for.

        I didn't look at LWP long enough to see whether any of the croak()s could have originated from user code and passed through a different module/package of LWP (thus requiring an @ISA scan to find the user code and the proper place to report the error). But I did notice that several of the error messages would be pretty confusing if this did happen.

        In some ways, I don't think what Carp.pm is doing is horribly wrong. But it is starting to sound to me like you are "fixing" Carp.pm without a design document and that is part of our problem.

        So Exporter complaining about %EXPORT_TAGS being set up incorrectly should be throwing a precondition failure which shouldn't (by default) look at (just) inheritance. "Fixing" the problem by lying about your inheritance during a window of time is what sounds like "playing games" to me.

        What you keep calling "playing cute games" is simply initializing variables close to when you declare them and before you might use them. This is often easy in Perl. But for "static" variables, doing so currently often requires the very simple step of putting the initialization into a BEGIN block. Nothing fancy nor tricky nor in the slightest way "cute". (:

        And in this case in particular you will guaranteed get wrong behaviour from both old and current versions of Perl

        Nope. Never have and probably never will. I'd only get wrong behavior if I'd also made stupid mistakes about setting up %EXPORT_TAGS or such. And then I'd have to do that where I have multiple packages inheriting from each other. Yep, I pretty much guarentee that I won't ever run into that. Even if I did, I'd find those during module testing where the error being reported in the wrong place is pretty minor.

        Not initializing statics at compile time has bitten me several times, for different reasons each time. The only specific cause that I recall is circular module dependancy which isn't something you run into often but then that only accounts for one of my bites.

        I've seen this advocated by several people, so I'm sure they've been bitten by it too. I'm not telling you to trust their decision about whether or not to do this, I'm telling you that several people have been bitten by it enough that they decided to change. You have one case of the reverse that I know I don't care about. I have one specific case that I suspect you don't care about. But there are quite a few cases out there on my side.

        Plus, your case really seems to me to be a case of two wrongs making a right. Carp.pm gets the inheritance tree wrong which prevents its other bug from firing. I consider it evidence that you should initialize statics in BEGIN blocks, because look at all that can happen during that window!

        Well you know my opinion about the stupid misuses of $Carp::CarpLevel in the past. First of all when proper coding depends on programmers always getting things right, that is guaranteed to go wrong often.

        Um, so if programmers have to get things right then sometimes they won't and then things will go wrong. Is that a tautology? Yeah, $Carp::CarpLevel was a terrible design choice. But having a per-module equivalent of $Carp::CarpLevel as well as a perl-call equivalent would be useful and fairly easy to do. And if it designed well and documented, then programmers will very often get it right. Somehow you think that you setting the one true behavior, that you will get it right more often than programmers will? Only if the programmers are given a horrid, undocumented way to do it. Don't let how bad $Carp::CarpLevel was prevent you from designing and implementing a good solution.

        Now, you can certainly choose to let this one obscure bug prevent you from coding defensively when it comes to uninitialized static variables; but I think that would be a mistake. :)

        1 I'm not a big fan of warnings from modules. Warnings from a module can be quite useful to the script writer (aka module user) but are usually delivered to the script user (non-programmer). The script writer can trap them with $SIG{__WARN__}, for example, but that will also catch warnings that are meant for the script user. So if I want to send a programmer warning from a module, I'll provide a delivery mechanism other than warn.

                - tye (but my friends call me "Tye")
Re: Re (tilly) 4 (gotcha): Supersplit
by Anonymous Monk on Jun 13, 2001 at 23:33 UTC
    In caller you have file and line number. Go ahead and parse the call yourself if you need more info.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://49090]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (11)
As of 2014-09-23 09:44 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (216 votes), past polls