Re^7: Module::Install hacking

What happens when one of your renamed dlls changes?

These dll's are part of the MinGW compiler - they will only change when I switch to a different version of MinGW. (That *will* happen, but hopefully not with great frequency.)

When I do upgrade my compiler from the current 4.5.2 to (say) 4.6.3 then the renamed dll will change from libgcc_sis_452.dll to libgcc_sis_463.dll.
I don't see any problems with that - packages that need libgcc_s_452.dll will still install $Config{installsitebin}/libgcc_sis_452.dll, and those that need libgcc_sis_463.dll will install $Config{installsitebin}/libgcc_sis_463.dll.
Eventually, as time goes by,all of the ppm packages will need libgcc_sis_463.dll, and there'll be a 100Kb libgcc_sis_452.dll sitting in the users $Config{installsitebin} folder that is no longer needed. (I think I can live with that :-)

Similarly for libstdc++-6.dll (which is the *only* other dll I rename). It, too, is a MinGW compiler dll - and I'll be renaming it to something other that cpp++-6_sis.dll when I upgrade my MinGW compiler.

I didn't properly stress it in my previous post, but these 2 dlls are the *only* dlls that I rename with my x86 builds.
With my x64 builds there are also only 2 dlls that I rename - libgcc_s_sjlj-1.dll (which is the mingw64 counterpart of mingw's libgcc_s_dw2-1.dll) and libstdc++-6.dll (same name as mingw's corresponding file).

I install two of your packages each has a dependency upon a different version of one or more of your renamed dlls. Whichever order I install them in, one of them gets the wrong thing

Shit !! - I hope you're referring to a hypothetical situation here :-)
So far, there has been no version change. So, such a situation should not yet have arisen, and should not ever arise so long as I adhere to the principle I've just outlined (of giving different versions different names).

1. shipping these dlls under their real names, with each package that uses them, and place them in the same directory as the dll(s) that use them

This is precisely what I do with *all* dll's except the 3 mentioned above - and I used to it with those 3 as well. But there's a problem with that approach wrt those 3 dlls.
Let's say I'm building the FOO extension with my x64 gcc-4.7.0 compiler, and I put a copy of its libgcc_s_sjlj-1.dll in blib/arch/auto/FOO (alongside blib/arch/auto/FOO/FOO.dll). That works fine in most places, but if someone installs my FOO binaries on Strawberry Perl, the libgcc_s_sjlj-1.dll that I packaged never gets loaded.
This happens because Strawberry Perl itself loads libgcc_s_sjlj-1.dll - only *it* will load Strawberry's 4.6.3 version of libgcc_s_sjlj-1.dll, thereby preventing the libgcc_s_sjlj-1.dll that I provided from being loaded by FOO.
Unless the 2 versions of libgcc_s_sjlj-1.dll have an identical set of entry points (and they don't), you've got trouble.
It's this problem that led me to adopting my current approach.

With x86 perls, this type of problem is less likely, but still possible.
To phrase it in general terms - the approach you've outlined in 1. is not guaranteed to work if perl itself loads a dll that has the same name as a dll that I've placed in blib/arch/auto/FOO. And, in the real world of x64 Strawberry Perl, it's guaranteed to fail.

One thing, however ... I've just been checking ... and it seems that mingw perl itself doesn't load libstdc++-6.dll. That being so, there's no reason to apply the same treatment to that dll. (I'll check on that and phase it out if I can.)

Similarly with your second option, I think:

Win32::LoadLibrary( 'libgcc_s_dw2_1.dll' ) or die $^E;
[download]

will either die or fail to load the the dll that I want if a dll named 'libgcc_s_dw2_1.dll' has already been loaded by perl itself.

In summary, as long as I limit this to the 3 (hopefully-reduceable-to-2) dll files specified above, I'm pretty confident I can be handle this without much drama. (Famous last words ;-)

Cheers,
Rob

Comment on Re^7: Module::Install hacking Download Code

Replies are listed 'Best First'.
Re^8: Module::Install hacking by BrowserUk (Patriarch) on Sep 14, 2012 at 12:41 UTC
Let's say I'm building the FOO extension with my x64 gcc-4.7.0 compiler, and I put a copy of its libgcc_s_sjlj-1.dll in blib/arch/auto/FOO (alongside blib/arch/auto/FOO/FOO.dll). That works fine in most places, but if someone installs my FOO binaries on Strawberry Perl, the libgcc_s_sjlj-1.dll that I packaged never gets loaded. This happens because Strawberry Perl itself loads libgcc_s_sjlj-1.dll - only it will load Strawberry's 4.6.3 version of libgcc_s_sjlj-1.dll, thereby preventing the libgcc_s_sjlj-1.dll that I provided from being loaded by FOO. Flying by the seat of my pants here -- and I have to leave for while, so verifying this will take some time -- but that shouldn't happen! (Part of) The point of side-by-side assemblies is that, the same dll, located from different paths via the DLL Search Order, should allow both to be loaded without conflict. I tend to think of it -- though it is probably a simplification of the reality -- as incorporating the path into the libraries load-name. In a nutshell, what I think should happen in your scenario, is that Strawberry Perl loads (say) 'c:\StrawberryPerl\bin\libgcc_s_sjlj-1.dll', and then at runtime, one of your modules loads 'c:\StrawberryPerl\site\lib\auto\Package\Name\libgcc_s_sjlj-1.dll',. And the two -- even if identical, are loaded into memory at different base addresses and -- the same or different -- everything works. If you know different, I defer; but that's my understanding at this point. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP Neil Armstrong	[reply]
Re^9: Module::Install hacking by syphilis (Archbishop) on Sep 14, 2012 at 13:38 UTC
If you know different, I defer; but that's my understanding at this point I certainly don't claim to know a lot about this - but it was our (mine and aero's) experience that one of my past x64 builds of PDL wouldn't work (because of missing entry point) on x64 Strawberry unless we either: a) Replaced Strawberry/c/bin/libgcc_s_sjlj-1.dll with my 4.7.0 version of the same file or b) employed the current hack. I did ask about this on the Strawberry list - and it was there that I was put on to the hack that I've used. At the link you provided, I see: If a DLL with the same module name is already loaded in memory, the system checks only for redirection and a manifest before resolving to the loaded DLL, no matter which directory it is in. The system does not search for the DLL. There are no manifest files involved (afaik) ... and the quote I've just provided suggests to me that the already-loaded dll will therefore be used. Admittedly I'm not sure what "redirection" is .... or what "resolving to the loaded DLL" exactly means. (What dictionary do they use in Redmond ?) Thinking a bit more about it ... I guess that it could have been the case that the libgcc_s_sjlj-1.dll that I had provided from my 4.7.0 compiler was simply located in the wrong place. (But I think we checked that wasn't happening.) I'll re-visit this tomorrow with a much simpler test case than PDL - just to be sure. Cheers, Rob	[reply]
Re^10: Module::Install hacking by BrowserUk (Patriarch) on Sep 15, 2012 at 00:10 UTC
In theory, if a dll that is part of an XS module is dynamically loaded and has a dependency upon another dll that is already loaded (whether by the main perl.exe, perl5xx.dll, or some other previously loaded XS dll), then you can prevent that loaded dependency from being 'reused', by adding a xxx.dll.local file in the same directory as the xxx.dll. Hm. That's as clear as mud. I'l try again :) perl.exe uses perl5x.dll which uses dependee.dll. Module MyXS.pm uses MyXS.dll uses dependee.dll. But it may need a different version of it to perl5x.dll. In the normal way of things, when MyXS.dll is runtime dynamically loaded as a result of `use MyXS;` statement, when the OS loader is loading MyXS.DLL, it sees it dependency upon dependee.dll, see that the process already has (a copy of) that loaded, and doesn't look further. However, if whilst loading MyXS.dll, it see a file MyXS.dll.local in the same directory, it should then ignore the existing in-memory copy of dependee.dll and look for (another copy) of it in the same directory as MyXS.dll, and use that for doing the fix-ups. That, by every interpretation I can find is what should happen. But ... I cannot make it work! `myApp\myApp.c myApp\myApp.exe myApp\myDll.c myApp\myDLL.dll myApp\Dep.c myApp\Dep.dll myApp\Ext\ExtDll.c myApp\Ext\ExtDLL.dll myApp\Ext\ExtDLL.dll.local myApp\Ext\Dep.c myApp\Ext\Dep.dll` [download] MyApp.exe uses myDll.dll uses (myApp\)Dep.dll. At runtime, MyApp.exe LoadLibs Ext\ExtDll.dll which uses (myApp\Ext\)Dep.dll. The presence of myApp\Ext\ExtDLL.dll.local should force the OS loader to look in MyApp\Ext for Dep.dll when fixing up ExtDll.dll's dependencies. It doesn't: depends64 -c -ot:MyApp.depends -pb myapp.exe ext\ExtDLL.dll [... snip lots ...] Started "c:\test\myapp\MYAPP.EXE" (process 0x1194) at address 0x000000 +0140000000. Successfully hooked module. Loaded "c:\windows\system32\NTDLL.DLL" at address 0x0000000077A50000. + Successfully hooked module. Loaded "c:\windows\system32\KERNEL32.DLL" at address 0x000000007792000 +0. Successfully hooked module. Injected "c:\perl64\bin\DEPENDS.DLL" at address 0x0000000075030000. Loaded "c:\test\myapp\MYDLL.DLL" at address 0x0000000180000000. Succe +ssfully hooked module. Loaded "c:\test\myapp\DEP.DLL" at address 0x0000000000140000. Success +fully hooked module. Entrypoint reached. All implicit modules have been loaded. LoadLibraryExA("ext\ExtDLL.dll", 0x0000000000000000, 0x00000000) calle +d from "c:\test\myapp\MYAPP.EXE" at address 0x000000014000103F. Loaded "c:\test\myapp\ext\EXTDLL.DLL" at address 0x0000000000160000. +Successfully hooked module. LoadLibraryExA("ext\ExtDLL.dll", 0x0000000000000000, 0x00000000) retur +ned 0x0000000000160000. GetProcAddress(0x0000000000160000 [c:\test\myapp\ext\EXTDLL.DLL], "boo +t") called from "c:\test\myapp\MYAPP.EXE" at address 0x00000001400010 +92 and returned 0x0000000000161000. Exited "c:\test\myapp\MYAPP.EXE" (process 0x1194) with code 0 (0x0). [download] The significant bit of that is that there is no `Loaded "c:\test\myapp\Ext\DEP.DLL" ...` line :( Which means that MS docs are either misleading or just outright wrong. It is also (theoretically) possible to attach a manifest to the XS dll that would force the OS loader to look for its dependencies in the same directory first. But the whole manifests thing just gives me a headache, so I haven't (re-)explored that this time. All of which means ... please forget I said anything and carry on as you were. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP Neil Armstrong	[reply] [d/l] [select]
Re^10: Module::Install hacking by bulk88 (Priest) on Sep 14, 2012 at 18:55 UTC
Redirection is a ".local" file in the directory with that DLL. Only works if perl.exe does NOT have a manifest. (VC/AP 5.10 good, VC/AP 5.12 no).	[reply]
Re^8: Module::Install hacking by bulk88 (Priest) on Sep 14, 2012 at 18:31 UTC
Maybe you should bring this up with the Mingw folks, both of them, since they have a ugly fork war right now. DLLs should always be backwards compatible if their names are the same. That is the expectation of Microsoft Windows. Another random idea, why not create a Perl module whose only purpose is to install that particular Mingw DLL? edit:grammer	[reply]
Re^9: Module::Install hacking by syphilis (Archbishop) on Sep 15, 2012 at 02:21 UTC
DLLs should always be backwards compatible if their names are the same Yes - I think this was a case of needing forwards compatibility. That is, build a module that uses the gcc-4.7.0 version of libgcc_s_sjlj-1.dll, then run it with the gcc-4.4.3 version of the same dll and we got a missing entry point in relation to __addtf3. I'm going to check this out again when I can get hold of strawberry perl 5.14.2.1 64bit. I currently have the 5.16.0 version of Strawberry (gcc-4.6.3), and there's apparently no such incompatibility between its libgcc_s_sjlj-1.dll and gcc-4.7.0's. Another random idea, why not create a Perl module dedicated whose purpose only purpose is install that particular Mingw DLL? Sounds good to me. It wouldn't help wrt the problem of perl/site/bin not being in the path, but it would ultimately avoid the need to use --force. Apart from writing and uploading the module, I would just need to alter the ppd files and package.xml (to specify this additional dependency) - which could be done programmatically. The actual binaries that are already on the repo don't need amending. The module should, I guess, be under the PPM namespace. Would PPM::Sisyphusion suffice as a name ? Is there something better ? I'll definitely give this some consideration. Cheers, Rob	[reply]
Re^10: Module::Install hacking by bulk88 (Priest) on Sep 15, 2012 at 05:13 UTC
Sounds good to me. It wouldn't help wrt the problem of perl/site/bin not being in the path, but it would ultimately avoid the need to use --force. Apart from writing and uploading the module, I would just need to alter the ppd files and package.xml (to specify this additional dependency) - which could be done programmatically. The actual binaries that are already on the repo don't need amending. The module should, I guess, be under the PPM namespace. Would PPM::Sisyphusion suffice as a name ? Is there something better ? I'll definitely give this some consideration. I've seen the Alien:: package used for C libraries, not sure if object files or executable files, of course ask module authors list if you are not sure. I'm not sure if there is a certain API that Alien:: modules have or not. My only experience with Alien::wxWidgets and I want to forget I ever had to use it (A::W demands to build Wx C++ libs from source, VC Wx only has a VS IDE Solution, no nmake makefile). I can't really help you much further the idea I proposed, I've never done with my eyes, and I dont know how feasible it is. It somehow sounds logical since the "Module" can get new CPAN releases to distribute bug fixes of that particular DLL. The DLL name is obviously versioned to a certain ABI unless you write your own libgcc.a that will LoadLibraryEx and GetProcAddress everything.	[reply]


Perl-Sensitive Sunglasses
	PerlMonks