Will the real Internals module please stand up?

[dragonchild]

After shaving a yak, I ended up in Hash::Util, specifically in lock_keys(). Noticed heavy use in the PurePerl version of Internals, specifically Internals::SvREADONLY(). Never having heard of it, I looked it up. Lo and behold, although SvREADONLY() was used by the XS, it wasn't exposed by the XS. Bopping around a bit, I found use of a bunch of functions in the Internals:: namespace that aren't exposed by Internals. And, furthermore, it seems like these functions are available without use-ing Internals itself.

So, the question - what's in Internals:: and where can I get some documentation on it? Sounds like all sorts of goodies are in there and I wanna get me some of that! :-)

Update: Per Anonymonk's reply, apparently a whole bunch of seemingly-unrelated things are provided in universal.c and that's just the list for 5.8.9. In 5.10.0's universal.c, there's twice the list. (Scroll down halfway looking for /^XS\(/). Why are these functions all stuffed together in universal.c? Where are they documented? If they're exposed, I would think they would be documented somewhere.

[ikegami]

>perl -wle"while(($n,$g)=each %Internals::) { print $n if *$g{CODE} }"
hash_seed
hv_clear_placeholders
HvREHASH
rehash_seed
SvREADONLY
SvREFCNT

ActivePerl v5.10.0 build 1001 built for MSWin32-x86-multi-thread

Update:

#!perl -wl
my @todo = *::;
while (@todo) {
   my $st = shift(@todo);
   my @sl = sort values %$st;
   print for map /^\*(.*)/, grep *$_{CODE}, @sl;
   push @todo, grep !/\*main::main::/, grep /::$/, @sl;
}

ActivePerl v5.10.0 build 1001 built for MSWin32-x86-multi-thread:

Read more... (2 kB)

ActivePerl v5.8.8 build 824 built for MSWin32-x86-multi-thread

Read more... (1143 Bytes)

This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi:

DynaLoader::boot_DynaLoader
Internals::HvREHASH
Internals::SvREADONLY
Internals::SvREFCNT
Internals::hash_seed
Internals::hv_clear_placeholders
Internals::rehash_seed
PerlIO::get_layers
Regexp::DESTROY
UNIVERSAL::VERSION
UNIVERSAL::can
UNIVERSAL::isa
attributes::bootstrap
utf8::decode
utf8::downgrade
utf8::encode
utf8::is_utf8
utf8::native_to_unicode
utf8::unicode_to_native
utf8::upgrade
utf8::valid
PerlIO::Layer::NoWarnings
PerlIO::Layer::find

[Anonymous Monk]

Only code there

$ perl -le'print for %Internals::'
SvREFCNT
*Internals::SvREFCNT
hv_clear_placeholders
*Internals::hv_clear_placeholders
hash_seed
*Internals::hash_seed
SvREADONLY
*Internals::SvREADONLY
HvREHASH
*Internals::HvREHASH
rehash_seed
*Internals::rehash_seed

[ikegami]

How is that better? The output is less readable and it introduces an assumption.

[ysth]

That includes statically linked modules, not just things from the perl core.

[Anonymous Monk]

They're exposed by UNIVERSAL
http://cpansearch.perl.org/src/RGARCIA/perl-5.10.0/universal.c

[ikegami]

Why isn't it documented? How official is this? For example, can I rely on

our $VERSION = version::qv('1.0.0');

without doing use version; first? (Ignore compatibility with 5.8.)

[ysth]

can I rely on our $VERSION = version::qv('1.0.0'); without doing use version; first?

In counter-example, the utf8 and Win32 doc specifically say what can be assumed to be available without loading the module, so unless you can find a similar guarantee in the version doc, I think you'd be overly trusting to do so.

[Anonymous Monk]

I don't know, appears to be a case of "some patch got accepted"

[ysth]

Undocumented things in universal.c should not be depended on; they should only be used by core modules. They aren't documented on purpose, to allow them to be changed whenever and however necessary. For those purposes, the code is good enough documentation.

[ikegami]

Some are documented

UNIVERSAL::isa	implied
UNIVERSAL::can	implied
UNIVERSAL::DOES	implied
UNIVERSAL::VERSION	implied
version::new	undocumented availability
version::qv	undocumented availability
version::boolean	completely undocumented
version::vcmp	completely undocumented
version::noop	completely undocumented
version's UNIVERSAL::VERSION override	undocumented availability
utf8::is_utf8	documented
utf8::valid	documented
utf8::encode	documented
utf8::decode	documented
utf8::upgrade	documented
utf8::downgrade	documented
utf8::native_to_unicode	completely undocumented
utf8::unicode_to_native	completely undocumented
Internals::SvREADONLY	completely undocumented
Internals::SvREFCNT	completely undocumented
Internals::hv_clear_placeholders	completely undocumented
Internals::hash_seed	completely undocumented
Internals::rehash_seed	completely undocumented
Internals::HvREHASH	completely undocumented
PerlIO::get_layers	undocumented availability
re::is_regexp	undocumented availability
re::regname	undocumented availability
re::regnames	undocumented availability
re::regnames_count	undocumented availability
Tie::Hash::NamedCapture::	undocumented availability

The table applies to 5.10.0. I omitted methods except those in UNIVERSAL (since those have a global effect).

Legend:

documented	These functions are documented as always available.
implied	The documentation for these functions implies they are always available.
undocumented availability	These functions are documented, but their availability isn't.
completely undocumented	The documentation of the modules for these functions don't mention them.

Interestingly, version claims to override UNIVERSAL::VERSION, but there is only one function. The "override" is always in effect.

Interestingly, the following bindings are absent:

• re::regmust
• re::regexp_pattern

[demerphq]

What do you mean about "bindings are absent"?

Most of this stuff, in particular availability, is not documented directly for a reason. So for instance Tie::Hash::NamedCapture is used to implement named captures. It is loaded automagically by the internals when you do a $+{foo} style lookup. It uses a bunch of the re:: functions, which are exposed via universal because they are then compiled into the core executable and thus do not require an additional library to be loaded. Now I could have implemented the entire thing without using any perl level modules or exposing any subs, but that would have limited the options available to the programmer which I felt was not the Perl way.

[ikegami]

The utf8:: functions can't be loaded by use utf8; since that's a pragma. I guess universal.c was a speedier alternative to making a module just to expose the utf8:: functions.

[ysth]

They could have been included in the module, the same way that there are utility functions in the charnames pragma that you can load without triggering the pragma behavior via "use charnames ()" or "require charnames".

So either it was just a "speedier alternative", as you say, or it was desired that the module not need to be loaded. I can't guess which.

[demerphq]

One reason to put things in universal.c is it allows you to use these functions in miniperl. If they are used by things that are run by miniperl or they are used to test perl then they need to be available early. Another reason is that functions are used to implement parts of perls internals, and therefore need to be around. An example is some of the re:: routines that are used to implement named captures. They need to be usable without loading the re dll, which also contains a full debugging enabled version of the regex engine.