Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Will the real Internals module please stand up?

by dragonchild (Archbishop)
on Mar 01, 2009 at 05:00 UTC ( #747247=perlquestion: print w/replies, xml ) Need Help??
dragonchild has asked for the wisdom of the Perl Monks concerning the following question:

After shaving a yak, I ended up in Hash::Util, specifically in lock_keys(). Noticed heavy use in the PurePerl version of Internals, specifically Internals::SvREADONLY(). Never having heard of it, I looked it up. Lo and behold, although SvREADONLY() was used by the XS, it wasn't exposed by the XS. Bopping around a bit, I found use of a bunch of functions in the Internals:: namespace that aren't exposed by Internals. And, furthermore, it seems like these functions are available without use-ing Internals itself.

So, the question - what's in Internals:: and where can I get some documentation on it? Sounds like all sorts of goodies are in there and I wanna get me some of that! :-)

Update: Per Anonymonk's reply, apparently a whole bunch of seemingly-unrelated things are provided in universal.c and that's just the list for 5.8.9. In 5.10.0's universal.c, there's twice the list. (Scroll down halfway looking for /^XS\(/). Why are these functions all stuffed together in universal.c? Where are they documented? If they're exposed, I would think they would be documented somewhere.

My criteria for good software:
  1. Does it work?
  2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

Replies are listed 'Best First'.
Re: Will the real Internals module please stand up?
by ikegami (Pope) on Mar 01, 2009 at 05:28 UTC
    >perl -wle"while(($n,$g)=each %Internals::) { print $n if *$g{CODE} }" hash_seed hv_clear_placeholders HvREHASH rehash_seed SvREADONLY SvREFCNT

    ActivePerl v5.10.0 build 1001 built for MSWin32-x86-multi-thread


    #!perl -wl my @todo = *::; while (@todo) { my $st = shift(@todo); my @sl = sort values %$st; print for map /^\*(.*)/, grep *$_{CODE}, @sl; push @todo, grep !/\*main::main::/, grep /::$/, @sl; }

    ActivePerl v5.10.0 build 1001 built for MSWin32-x86-multi-thread:

    ActivePerl v5.8.8 build 824 built for MSWin32-x86-multi-thread

    This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi:

    DynaLoader::boot_DynaLoader Internals::HvREHASH Internals::SvREADONLY Internals::SvREFCNT Internals::hash_seed Internals::hv_clear_placeholders Internals::rehash_seed PerlIO::get_layers Regexp::DESTROY UNIVERSAL::VERSION UNIVERSAL::can UNIVERSAL::isa attributes::bootstrap utf8::decode utf8::downgrade utf8::encode utf8::is_utf8 utf8::native_to_unicode utf8::unicode_to_native utf8::upgrade utf8::valid PerlIO::Layer::NoWarnings PerlIO::Layer::find
      Only code there
      $ perl -le'print for %Internals::' SvREFCNT *Internals::SvREFCNT hv_clear_placeholders *Internals::hv_clear_placeholders hash_seed *Internals::hash_seed SvREADONLY *Internals::SvREADONLY HvREHASH *Internals::HvREHASH rehash_seed *Internals::rehash_seed
        How is that better? The output is less readable and it introduces an assumption.
Re: Will the real Internals module please stand up?
by Anonymous Monk on Mar 01, 2009 at 05:31 UTC

      Why isn't it documented? How official is this? For example, can I rely on

      our $VERSION = version::qv('1.0.0');

      without doing use version; first? (Ignore compatibility with 5.8.)

        I don't know, appears to be a case of "some patch got accepted"
        can I rely on our $VERSION = version::qv('1.0.0'); without doing use version; first?
        In counter-example, the utf8 and Win32 doc specifically say what can be assumed to be available without loading the module, so unless you can find a similar guarantee in the version doc, I think you'd be overly trusting to do so.
Re: Will the real Internals module please stand up?
by ysth (Canon) on Mar 01, 2009 at 19:39 UTC
    Undocumented things in universal.c should not be depended on; they should only be used by core modules. They aren't documented on purpose, to allow them to be changed whenever and however necessary. For those purposes, the code is good enough documentation.

      Some are documented

      version::newundocumented availability
      version::qvundocumented availability
      version::booleancompletely undocumented
      version::vcmpcompletely undocumented
      version::noopcompletely undocumented
      version's UNIVERSAL::VERSION overrideundocumented availability
      utf8::native_to_unicodecompletely undocumented
      utf8::unicode_to_nativecompletely undocumented
      Internals::SvREADONLYcompletely undocumented
      Internals::SvREFCNTcompletely undocumented
      Internals::hv_clear_placeholderscompletely undocumented
      Internals::hash_seedcompletely undocumented
      Internals::rehash_seedcompletely undocumented
      Internals::HvREHASHcompletely undocumented
      PerlIO::get_layersundocumented availability
      re::is_regexpundocumented availability
      re::regnameundocumented availability
      re::regnamesundocumented availability
      re::regnames_countundocumented availability
      Tie::Hash::NamedCapture::undocumented availability

      The table applies to 5.10.0. I omitted methods except those in UNIVERSAL (since those have a global effect).


      documentedThese functions are documented as always available.
      impliedThe documentation for these functions implies they are always available.
      undocumented availabilityThese functions are documented, but their availability isn't.
      completely undocumentedThe documentation of the modules for these functions don't mention them.

      Interestingly, version claims to override UNIVERSAL::VERSION, but there is only one function. The "override" is always in effect.

      Interestingly, the following bindings are absent:

      • re::regmust
      • re::regexp_pattern

        What do you mean about "bindings are absent"?

        Most of this stuff, in particular availability, is not documented directly for a reason. So for instance Tie::Hash::NamedCapture is used to implement named captures. It is loaded automagically by the internals when you do a $+{foo} style lookup. It uses a bunch of the re:: functions, which are exposed via universal because they are then compiled into the core executable and thus do not require an additional library to be loaded. Now I could have implemented the entire thing without using any perl level modules or exposing any subs, but that would have limited the options available to the programmer which I felt was not the Perl way.


Re: Will the real Internals module please stand up?
by ikegami (Pope) on Mar 01, 2009 at 19:49 UTC
    The utf8:: functions can't be loaded by use utf8; since that's a pragma. I guess universal.c was a speedier alternative to making a module just to expose the utf8:: functions.
      They could have been included in the module, the same way that there are utility functions in the charnames pragma that you can load without triggering the pragma behavior via "use charnames ()" or "require charnames".

      So either it was just a "speedier alternative", as you say, or it was desired that the module not need to be loaded. I can't guess which.

        One reason to put things in universal.c is it allows you to use these functions in miniperl. If they are used by things that are run by miniperl or they are used to test perl then they need to be available early. Another reason is that functions are used to implement parts of perls internals, and therefore need to be around. An example is some of the re:: routines that are used to implement named captures. They need to be usable without loading the re dll, which also contains a full debugging enabled version of the regex engine.


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://747247]
Approved by ikegami
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (9)
As of 2017-02-20 20:09 GMT
Find Nodes?
    Voting Booth?
    Before electricity was invented, what was the Electric Eel called?

    Results (302 votes). Check out past polls.