Refresh a Module

[Timka]

I have a simple module called My.pm:

package My;
print "Loading...\n";
sub Run { print "Running\n" }

If I were to try to load, remove it from the symbol table, and reload it (as done by Module::Refresh::refresh_module):

use My;
delete $INC{"My.pm"};
undef &My::Run;            # Why do this?!
delete $My::{Run};
require My;
eval "My::Run()";
My::Run();

Then I'd see output like this:

Loading...
Loading...
Running
Undefined subroutine &My::Run called at -e line 1.

Question 1: why does the undef statement makes the next require fail?

I can get around that by simply removing that line:

use My;
delete $INC{"My.pm"};
delete $My::{Run};
require My;
eval "My::Run()";
My::Run();

Then I'd see output like this:

Loading...
Loading...
Running
Running

Now to the interesting part:

use My;
<STDIN>;                       # NEW
delete $INC{"My.pm"};
delete $My::{Run};
require My;
eval "My::Run()";
My::Run();

During the pause, update My.pm to have a different output message.

Then press enter to continue:

Loading...

Loading...
Running again
Running

One step further, during the pause, remove the Run function and observe:

Loading...

Loading...
Running again

Question 2: How to reload a module without having to use an eval (like eval "My::Run()")

Thanks,

~Tim

[Fletch]

Module::Reload prossibly is what you're looking for.

Edit: Derp I just actually read the whole thing and see you'd mentioned that module. I need moar caffeine.

[Timka]

I actually tried Module::Refresh

I will check Module::Reload (if I had not already.)

I recall seeing another module, but it was simple and would not remove existing functions.

[Timka]

Yea, Module::Reload just deletes from %INC and does require again.

That does not remove any subs that no longer exist in the updated file.

[afoken]

Question 2: How to reload a module without having to use an eval (like eval "My::Run()")

Reloading a module will always end in a string eval, like loading a module does. Remember or read the docs:

• use Some::Module ...; is exactly equivalent to BEGIN { require Some::Module; Some::Module->import(...); }
• require Some::Module; behaves like a 50 line wrapper around do "$prefix/Some/Module.pm" searching for Some/Module.pm along @INC and updating %INC.
• do "$prefix/Some/Module.pm" behaves much like $text=readFile("$prefix/Some/Module.pm"); eval $text;, searching @INC and setting %INC (and $@ and perhaps $! on error).

So, in EVERY use and require, a string eval is hidden. And that string eval will happen unless the requested module has already been loaded. If you load or reload a module at runtime, there will always be a string eval. So a simple eval "require Some::Module" does not hurt much. Of course, you can manually search for the module and do a require "$prefix/Some/Module.pm", but it will still hide a string eval.

---

Another way to reload updated modules is to just re-execute your script. Store $^X, $0 and @ARGV at startup, then when you feel the need to reload modules, run something like exec { $^X } $^X,$0,@ARGV; die "Re-exec failed: $!";. This will completely restart your script, loading all modules again. Of course, there are limitations:

• Your operating system may not support exec, in that case, perl may provide a rough emulation (Windows - see exec in perlport).
• If your STDIN comes from something like a socket or a terminal, the data you have already processed is lost.
• Some operating systems do not flush output file handles on exec, so some output may be lost. Again see exec in perlport.
• If your script generates some "startup noise" (e.g. write a header to STDOUT), that will probably be repeated unless you find a way to detect a re-exec. Adding a flag to %ENV or the stored copy of @ARGV might be an idea.
• All variables are lost. You must be able to regenerate the state of your program from the available input at the time of exec. If you can't, saving essential state information to a file (JSON, Storable) before exec and reading it back after exec might be an idea.

Alexander

[Corion]

LanX presented this topic at the Frankfurt Perl Workshop last Sunday and we discussed it. The main point I find missing is, that the redefinitions can only affect code, and not data. This makes the whole thing a non-starter for me, since for example reloading a class/object package will only do what I want if I don't declare new accessors and don't restructure my existing object tree:

my $info = My::Config->new( Load( 'config.yaml' ));

# in My::Config
package My::Config 0.01 {
    use Moo 2;
    has 'include_files' => (
        is => 'lazy',
        default => sub { [] },
    );
};

# After update:
# in My::Config
package My::Config 0.01 {
    use Moo 2;
    has 'plugins' => (
        is => 'lazy',
        default => sub { [] },
    );
    has 'include_templates' => (
        is => 'lazy',
        default => sub { [] },
    );
};

If the above happens, you will still have an object $info that looks like:

$info = {
    include_files => [],
};

... but the other code will never access the old data.

I can imagine cases where I want both, blowing away the old data, and keeping the old data, and I think there is no good "automatic" way of determining what I want in each case.

In limited situations, for example where programmer-users are allowed to rewrite some plugin code that gets hot-reloaded, and that plugin code is required to act in a functional way, that is, having no state of its own, reloading a module can still work.

[LanX]

I'm not using Moo(se) but this looks like you are changing the interface after reload.

Obviously (for me) you must respect the contract, when replacing code in a live system.

Regarding data, there is a reason why languages like Haskell require their functions to be "pure", i.e. to have no side effects or inner states.

Regarding Perl modules, a reload would need to handle not only functions, but also with package variables being used outside the module and/or lexical closure variables carrying state.

I could think of a solution with a proxy module shadowing all this, but this l also depends on the concrete requirements.

And practicality is different for development and production scenarios..

Last but not least, I can't really comment on OO frameworks. If their inner workings make it impossible to make data persistent, there is not much which can be done.

[LanX]

A function call does the stash lookup when it's compiled, and compile time inside the eval (and consequently require) is runtime.

Deleting the Stash entry after the call is compiled doesn't help, because the reference is cached.

But you can always redefine/redeclare the sub. An already compiled call will execute the new code.

If you are looking for a way to destroy a sub in a way to make an already compiled call break, I'm very sceptical about the wisdom of this concept.

FWIW: you can also try delete $::{'My::'} to delete the whole package. But this won't touch already referenced elements, resp. compiled calls.

[LanX]

> Deleting the Stash entry after the call is compiled doesn't help, because the reference to the typeglob is cached.

Ran a quick test on my phone:

Looks like the typeglob˛ is looked up and the reference is stored in the OP tree. See \&main::t in #4.

That's why further manipulation will produce different results if the original typeglob isn't targeted. °

(Concise is able to backengineer the original name or is doing other introspection)

~ $ perl -MO=Concise  -e'sub t {}; t()'

6  <@> leave[1 ref] vKP/REFC ->(end)
1     <0> enter v ->2
2     <;> nextstate(main 3 -e:1) v:{ ->3
5     <1> entersub vKS ->6
-        <1> ex-list K ->5
3           <0> pushmark s ->4
-           <1> ex-rv2cv sK/1 ->-
4              <#> gv[IV \&main::t] s ->5
-e syntax OK

For comparison, that's what's happening when the sub is unknown at compile time

gv[*t] which most likely means "look up the symbol t at runtime"...

~ $ perl -MO=Concise  -e't();'                       
6  <@> leave[1 ref] vKP/REFC ->(end)
1     <0> enter v ->2
2     <;> nextstate(main 1 -e:1) v:{ ->3
5     <1> entersub[t2] vKS/TARG ->6
-        <1> ex-list K ->5
3           <0> pushmark s ->4
-           <1> ex-rv2cv sK/1 ->-
4              <#> gv[*t] s/EARLYCV ->5
-e syntax OK
~ $

°) especially deleting the entry in the namespace prevents the compiler to find the old typeglob.

˛) or is it only the CODE slot? How reliable is concise here???

[Danny]

Can you elaborate or provide a reference? It's still unclear to me why My::Run() doesn't get redefined here:

require My;
print "a: "; My::Run(); # a: Running
undef &My::Run;
require My;
print "b: "; My::Run(); # Undefined subroutine &My::Run called at ...

[LanX]

Well yes, undef &sub is a way to destroy a sub. Again I doubt the wisdom of doing so.

I'll try to come up with some example code tomorrow...°

Though others might be faster ;)

°) My guess is that most of the confusion is caused by the deletes in the stash

[LanX]

Like already explained this is not only touching complicated concepts like symbol table, execution phases and typeglobs, but might also depend on internal implementation details.

It would help to know, what you expect to happen, when a module is "refreshed".

Best in an SSCCE with a test suite, because words (and perldocs) are fuzzy.

And keep in mind that functions of a module might also have been exported to other namespaces or their reference is stored elsewhere.

Now, are those "links" to older functions expected to ...

• survive
• being disabled
• redefined

... ?

Please clarify!

[LanX]

To sum the thread up:

Question 1: why does the undef statement makes the next require fail?

Like already explained, it's the delete of the typeglob aka symbol which is causing the problem,

This works for me

use lib '.';

use My;
eval "My::Run('eval')";
My::Run('normal');


delete $INC{"My.pm"};


# --- Why do this?

undef &My::Run;

# 1.  Avoids warning redefined
# 2. will deactivate function if missing in new module


# --- Why DON'T do this

# delete $My::{Run};

# If you destroy the symbol, Perl won't know how to handle optimies op
+code

require My;
eval "My::Run('eval')";
My::Run('normal');

---- Loading My at 1727478470.8689 ... 
Running Run(eval) loaded at 1727478470.8689
Running Run(normal) loaded at 1727478470.8689


---- Loading My at 1727478470.87299 ... 
Running Run(eval) loaded at 1727478470.87299
Running Run(normal) loaded at 1727478470.87299

Question 2: How to reload a module without having to use an eval (like eval "My::Run()")

IMHO both, normal call and eval should work. Also importing symbols.

The following solution seems to work and should probably be a module on CPAN.

Basic idea is to never use use from the beginning. I.e. don't load at compile time. Use runtime "reuse" at the beginning of execution.

Like that the calls in the OP-Code will never optimize to refs and hence always check the symbols in the STASH, without weird side-effects.

use v5.12;
use warnings;
use lib ".";


ReUse::reuse("My" => 1,2,3);
My::Run("normal");
eval 'My::Run("eval")';

ReUse::reuse("My" => 1,2,3);
My::Run("normal");
eval 'My::Run("eval")';



package Test;
my $data = [1..5];


say "\n\n---- ReImporting from module";
ReUse::reuse("Data::Dump" => qw/pp/);
pp($data);                              # test import

say "\n\n---- ReImporting from module";
ReUse::reuse("Data::Dump" => qw/pp/);
pp($data);                              # test import





exit;

# --- this should probably be a proper CPAN Module

package ReUse;

sub reuse {
    my ($module, @imports) = @_;


    # --- purge %INC
    my $inc_key = "$module.pm";
    $inc_key =~ s(::)(/)g;
    my $oldpath = delete $INC{$inc_key};


    # --- build code



    # --- require at runtime
    my $PACKAGE = (caller)[0];
    my $REQUIRE = $inc_key;

    eval <<"__CODE__";
        package $PACKAGE;
        require "$REQUIRE";
__CODE__
    if ($@) {
        die qq(REQUIRE of "$module" failed with: <$@>);
    }


    # --- import at runtime
    my $IMPORTS = join ",", map {"'$_'"} @imports;

    eval <<"__CODE__";
        package $PACKAGE;
        $module->import($IMPORTS);
__CODE__
    if ($@) {
        die qq(IMPORT OF "$module->import($IMPORTS)" failed with: <$@>
+);
    }


    #  --- warn if module's path changed
    # TODO make optional
    if ($oldpath and $oldpath ne $INC{$inc_key}) { # untested TODO
        warn "Path of $module changed";
    }

}

---- Loading My at 1727477140.49975 ... 
Running Run(normal) loaded at 1727477140.49975
Running Run(eval) loaded at 1727477140.49975


---- Loading My at 1727477140.50415 ... 
Running Run(normal) loaded at 1727477140.50415
Running Run(eval) loaded at 1727477140.50415


---- ReImporting from module
[1 .. 5]


---- ReImporting from module
[1 .. 5]

NB: I didn't bother to purge the old STASH with the old symbols to have a clean start, because:

• Don't think you need that in your use case.
• here are dragons, if you took references of the old code, it would survive
• you never specified the edge cases
ATM I think deleting the namespace and/or undef-ing the subs and vars should be optional and decided by the user.

Last but not least: my My.pm (put in the same dir like above code).

package My;
use v5.12;
use warnings;
no warnings "redefine";

use Time::HiRes qw(time);

my $loadtime = time();
say "\n\n---- Loading " . __PACKAGE__ . " at $loadtime ... ";
sub Run { say "Running Run(@_) loaded at $loadtime" }

HTH

[Timka]

Besides the perl5 c/h mainly repo,

Would I find these esoteric feature in a perldoc page?

(Thanks everyone btw to your much provided input and enlightenment.)

[LanX]

First of all "refreshing a module" makes as much sense as "refreshing a family", it's not simply done by remarrying another partner.

You're the one who has to deal with the resulting patchwork. ( Unless you took care to have a very limited relation without kids and common property. Or unless you think you can still show up with a spare key and sleep in the old bed and you don't care about the mess ... Etc... etc ...)

I discussed it with my local group and everybody had another understanding what refreshing a module is supposed to engulf. I begged you to clarify what you want, alas you didn't.

Secondly there is a reason why implementation details remain undocumented features:

> Since the suppliers of the software usually consider the software documentation to constitute a contract for the behavior of the software, undocumented features are generally left unsupported and may be removed or changed at will and without notice to the users

Thirdly I already pointed you to documentation and listed the keywords. Most is found in perlsub and perlmod and my code was based on it.

Now if you just copy code you don't understand from modules meant for different ends and using undocumented features , ...

... Well then you shouldn't complain! :)

Deleting the stash is like burning down the marriage registry. It's still bigamy if you marry again.

Simply copying expert code doesn't make you smarter. Especially if the original authors had other goals in mind and your goals are fuzzy.

Happy reading of the docs. :)