UnBefuzzling XML::Parser: an adventure with EU::MM method overrides

[Intrepid]

First things first

The code is the major thing. The code is what the reader should try running* and try reading in order to understand the issues that I've addressed. The following links are current as of Wed Jan 16 11:08:42 UTC 2008 and point to different kinds of archives of the same source code kit, a tarball and a zipfile:

• beta 04, .tar.gz
• beta 04, .zip

Please check the website containing directory first for more recent updates before downloading the files above, would be my advice.

* Try running $ perl Makefile.PL verbose — use the verbose flag.

Required Reading

The CPAN module Request Tracker (RT) ticket connected to this coding effort is bug 32123.

Background on ExtUtils::MakeMaker

Documentation for the release of EUMM's MM_Unix.pm current as of this writing: ExtUtils/MM_Unix.pm

In all that follows, EUMM (a.k.a EU::MM) is used as a widely-known abbreviation for the ExtUtils::MakeMaker namespace and its sub-modules.

The notion at work in EUMM is code generation, where the type of "code" being emitted is the declarative syntax understood by `make'. The job of creating a Makefile was divided up into discreet methods, each of which would be responsible for a single section of the final Makefile. When "overrides" are mentioned, I am referring to the fact that about 60% of the methods for producing sections of Makefile were designed to be "overridden" so that special-case platforms (like ms windows) could be accommodated, or other special case scenarios could be achieved. The default methods live, appropriately, in the namespace ExtUtils::MM_Unix; this could be seen as "appropriate" since Unix is where Perl was born, and is such a (in many ways) standardized and widely-deployed family of OSes (as the famous footnote in the Camel puts it: "Assuming that you agree that Unix is both standard and fashionable" ;-).

In the ExtUtils/MM_Unix.pm module documentation mentioned above, a symbol (o) next to the name of the method is used to indicate that the method is one of the overridable ones. Methods lacking this mark do not have the automatic support for overriding; they were assumed to be doing parts of the Makefile text that would be invariant.

ExtUtils::MakeMaker Issues Addressed in this Work

The dlltool conundrum on mswin32.mingw

I was focused in this effort on the case wherein the platform OS is MS Windows and the "subplatform" is defined by the GNU toolset provided by the volunteer MinGW project, which includes the GNU Compiler Collection (GCC). Another example of a subplatform of MS Windows would be the VisualStudio 2008 from Microsoft. GCC, being free (in both senses) and standard across many types of systems, is a particularly attractive choice for those not employed at a workplace that provides and enforces the use of a proprietary commercial C compiler toolchain.

When the default EUMM -based Makefile.PL is run in the case of a C extension module like XML::Parser::Expat (the backend to XML::Parser), there is a section of Makefile code "emitted" by the method named ExtUtils::MM_Win32::dynamic_lib. This section of Makefile text includes a make rule (recipe) for building the dynamic lib (relocateable object code) or DLL which implements the C part of the extension. The rule will cause make to do the invocation of a program called dlltool which hasn't been required for most uses of mingw for many years now. It's obsolete code that targets a generation of mingw which is, in F/LOSS terms, practically ancient history.

One improvement I sought in writing the reworked build support for XML::Parser was the elimination of the baroque, little-understood, time consuming and unneeded invocation of dlltool in favor of a cleaner, up to date and more understandable direct creation of the DLL through invoking gcc. This way of calling gcc is much more congruent with the operation of gcc on Unix-like platforms that are familiar to many users with some background in compiling C code — like GNU/Linux.

The override that does the change to this method dynamic_lib() is located (at present) in the file hints/mswin32.pl in the source kit tree you'll get when you unpack one of the archives listed above. But please note that this way of overridding this method is not customary or best!. It was done in this way and in this file due to the particular unique process I went though of writing the overall refactoring, and may be moved out and done in a different (and better) way later.

Here's the excerpt from that file: the code that implements the overridden method (with comments as they appear in the source):

        package ExtUtils::MM_Win32;

    # clobber this one with extreme prejudice because of the gawd-awfu
+l bs
    # it is, for thinking that mingw gcc is still this clunky.

        sub dynamic_lib {
            my($self, %attribs) = @_;
            return '' unless $self->needs_linking(); #might be because
+ of a subdir
            return '' unless $self->has_link_code;

            my($otherldflags) = $attribs{OTHERLDFLAGS};
            my($inst_dynamic_dep) = $attribs{INST_DYNAMIC_DEP} || "";
            my($ldfrom) = '$(LDFROM)';
            my $tabchar = "\t";
            my(@m);

            my $dllname = $self->{BASEEXT} . "." . $self->{DLEXT};
            $dllname =~ /(....)(.{0,4})/;
            my $baseaddr = unpack("n", $1 ^ $2);
            $otherldflags .= sprintf("-Wl,--image-base,0x%x0000 ", $ba
+seaddr);

            push(@m,'
# This section creates the dynamically loadable $(INST_DYNAMIC)
# (ms windows DLL file) from $(OBJECT) and possibly $(MYEXTLIB).
OTHERLDFLAGS ='      .($otherldflags     ? " $otherldflags"  : "")    
+.'
INST_DYNAMIC_DEP ='  .($inst_dynamic_dep ? " $inst_dynamic_dep" : "") 
+.'

$(INST_DYNAMIC): $(OBJECT) $(MYEXTLIB) $(BOOTSTRAP) \
                 $(INST_ARCHAUTODIR)$(DFSEP).exists \
                 $(PERL_ARCHIVE) $(INST_DYNAMIC_DEP)
');
            push(@m, $tabchar,
                 q{$(CC) -o $@ $(LDDLFLAGS) }
               . $ldfrom ." \\\n"
               . q{ $(OTHERLDFLAGS) $(MYEXTLIB) $(PERL_ARCHIVE) $(LDLO
+ADLIBS)
}
            );
          # return our assembled Makefile text:
            join('',@m);

        }  #  END of sub

One thing of note is that the sub above is just Perl. Where there are strings that look like Makefile text, they are just Perl scalar strings. There is no "source filtering" or other eldritch magic going on here, merely using Perl to manipulated text, something that Perl is reputed to be rather good at ;-).

The override imposed on this section-builder method replaces an invocation that looks like this:

dlltool --def Expat.def --output-exp dll.exp
(a first invocation of dlltool, echo'd by make, followed by the commandline dlltool echos)
g++ -o ..\blib\arch\auto\XML\Parser\Expat\Expat.dll -Wl,--base-file -Wl,dll.base -mdll -s -LE:/perls/Strawberry_Perl_510_0/perl/lib/CORE -LE:/StPerl_Bundled_mingw/lib Expat.o -Wl,--image-base,0x31560000  C:\strawberry\perl\lib\CORE\libperl510.a E:\StPerl_Bundled_mingw\lib\libexpat.a E:\StPerl_Bundled_mingw\lib\libmsvcrt.a E:\StPerl_Bundled_mingw\lib\libmoldname.a E:\StPerl_Bundled_mingw\lib\libkernel32.a E:\StPerl_Bundled_mingw\lib\libuser32.a E:\StPerl_Bundled_mingw\lib\libgdi32.a E:\StPerl_Bundled_mingw\lib\libwinspool.a E:\StPerl_Bundled_mingw\lib\libcomdlg32.a E:\StPerl_Bundled_mingw\lib\libadvapi32.a E:\StPerl_Bundled_mingw\lib\libshell32.a E:\StPerl_Bundled_mingw\lib\libole32.a E:\StPerl_Bundled_mingw\lib\liboleaut32.a E:\StPerl_Bundled_mingw\lib\libnetapi32.a E:\StPerl_Bundled_mingw\lib\libuuid.a E:\StPerl_Bundled_mingw\lib\libws2_32.a E:\StPerl_Bundled_mingw\lib\libmpr.a E:\StPerl_Bundled_mingw\lib\libwinmm.a E:\StPerl_Bundled_mingw\lib\libversion.a E:\StPerl_Bundled_mingw\lib\libodbc32.a E:\StPerl_Bundled_mingw\lib\libodbccp32.a dll.exp dlltool --def Expat.def --base-file dll.base --output-exp dll.exp
(a second invocation of dlltool, echo'd by make, followed by the commandline dlltool echos)
g++ -o ..\blib\arch\auto\XML\Parser\Expat\Expat.dll -mdll -s -LE:/perls/Strawberry_Perl_510_0/perl/lib/CORE -LE:/StPerl_Bundled_mingw/lib Expat.o -Wl,--image-base,0x31560000  C:\strawberry\perl\lib\CORE\libperl510.a E:\StPerl_Bundled_mingw\lib\libexpat.a E:\StPerl_Bundled_mingw\lib\libmsvcrt.a E:\StPerl_Bundled_mingw\lib\libmoldname.a E:\StPerl_Bundled_mingw\lib\libkernel32.a E:\StPerl_Bundled_mingw\lib\libuser32.a E:\StPerl_Bundled_mingw\lib\libgdi32.a E:\StPerl_Bundled_mingw\lib\libwinspool.a E:\StPerl_Bundled_mingw\lib\libcomdlg32.a E:\StPerl_Bundled_mingw\lib\libadvapi32.a E:\StPerl_Bundled_mingw\lib\libshell32.a E:\StPerl_Bundled_mingw\lib\libole32.a E:\StPerl_Bundled_mingw\lib\liboleaut32.a E:\StPerl_Bundled_mingw\lib\libnetapi32.a E:\StPerl_Bundled_mingw\lib\libuuid.a E:\StPerl_Bundled_mingw\lib\libws2_32.a E:\StPerl_Bundled_mingw\lib\libmpr.a E:\StPerl_Bundled_mingw\lib\libwinmm.a E:\StPerl_Bundled_mingw\lib\libversion.a E:\StPerl_Bundled_mingw\lib\libodbc32.a E:\StPerl_Bundled_mingw\lib\libodbccp32.a dll.expwith one that looks like this:

gcc -o blib\arch\auto\XML\Parser\Expat\Expat.dll -mdll -s -LE:/perls/S
+trawberry_Perl_510_0/perl/lib/CORE -LE:/StPerl_Bundled_mingw/lib -O2 
+-mconsole -Wl,-export-all-symbols,-output-def,XML_Parser_Expat.def,-o
+ut-implib,XML_Parser_Expat.dll.a Expat.o \
 -Wl,--image-base,0x35130000  C:\strawberry\perl\lib\CORE/libperl510.a
+ e:/stperl_bundled_mingw/lib/libexpat.a e:/StPerl_Bundled_mingw/lib/l
+ibgdi32.a e:/StPerl_Bundled_mingw/lib/libwinspool.a e:/StPerl_Bundled
+_mingw/lib/libcomdlg32.a e:/StPerl_Bundled_mingw/lib/libole32.a e:/St
+Perl_Bundled_mingw/lib/liboleaut32.a e:/StPerl_Bundled_mingw/lib/libn
+etapi32.a e:/StPerl_Bundled_mingw/lib/libuuid.a e:/StPerl_Bundled_min
+gw/lib/libws2_32.a e:/StPerl_Bundled_mingw/lib/libmpr.a e:/StPerl_Bun
+dled_mingw/lib/libwinmm.a e:/StPerl_Bundled_mingw/lib/libversion.a e:
+/StPerl_Bundled_mingw/lib/libodbc32.a e:/StPerl_Bundled_mingw/lib/lib
+odbccp32.a
 
 Creating library file: XML_Parser_Expat.dll.a

I've inserted extra newlines in a couple of places in a futile attempt to make this console output more parseable to the human eye. But notice that the operations called for in the second instance involve less text and a single invocation of gcc.

---

Future elaboration

There might be more discussion of this patch later. At the time that I started writing this meditation I was not interested enough in teaching anyone anything due to the fatigue I'm experiencing after combining the coding effort with the effort to stay in touch with some of the other Perl authors involved. Such efforts included trying to contact a certain person at a known hangout (#p5p at irc - perl -org) and encountering the borderline psychosis so common among the current set of p5p personalities ... so please vote according to your displeasure at not finding more content here or at having the social club you identify with being insulted (or at neither of the above).

---

[moritz]

What do you want to achieve with this posting?

If it's a call for testing: I tested the tar ball on a Debian Etch i386 32 bit system, with libexpat-dev installed, and it built and "make test"ed like a charm.

If it's primarily a way to express your dissatisfaction with other members of the Perl community I don't think here's the right place. And remote psychological diagnosis doesn't improve the situation, independently of whether it's actually correct or not.

[syphilis]

The dlltool conundrum ...

Given that dlltool always runs flawlessly, I'm not sure that doing away with it should be highly prioritised. Then again, if dlltool is as anachronistic as you say (and I don't doubt you), then it's hard to argue against replacing it with something more contemporary.

I don't have the expat library in a place where it's found by default, so I ran your XML-Parser distro by starting with perl Makefile.PL INC="-IC:/_32/msys/1.0/local/include" LIBS="-LC:/_32/msys/1.0/local/lib -lexpat" and that all works fine on Strawberry Perl (5.10). But doing the same on my own MinGW-built perl 5.10, all of the tests fail (during 'dmake test') because Can't find 'boot_XML__Parser__Expat' symbol in C:\_32\comp\XML-Parser-2.36_04\blib\arch/auto/XML/Parser/Expat/Expat.dll .

I don't know why such a discrepancy arises. The CPAN version of XML-Parser-2.36 builds fine on *both* of those aforementioned builds of perl. I use different versions of dmake when building modules for those 2 builds of perl - but I doubt that's the reason. More likely some difference in EU::MM, I suspect. (It's a little intriguing ... but not nearly enough to make me go diving into the sources :-)

Cheers,
Rob

[Intrepid]

Great scott, another reply! ;-)

On Jan 17, 2008 our monk Rob told me (in #662812):

I don't have the expat library in a place where it's found by default, so I ran your XML-Parser distro by starting with perl Makefile.PL INC="-IC:/_32/msys/1.0/local/include" LIBS="-LC:/_32/msys/1.0/local/lib -lexpat" and that all works fine on Strawberry Perl (5.10). But doing the same on my own MinGW-built perl 5.10, all of the tests fail (during 'dmake test') because Can't find 'boot_XML__Parser__Expat' symbol in C:\_32\comp\XML-Parser-2.36_04\blib\arch/auto/XML/Parser/Expat/Expat.dll.

I don't know why such a discrepancy arises. The CPAN version of XML-Parser-2.36 builds fine on *both* of those aforementioned builds of perl. I use different versions of dmake when building modules for those 2 builds of perl - but I doubt that's the reason. More likely some difference in EU::MM, I suspect.

I suspect you're right about that.

I can say this much about the failure: it is due to the way the DLL was made, i.e. the way gcc was invoked. I know this because it was the last flaw I found when I was writing the build support changes; it is the lack of the flag --export-all-symbols that causes this. The DLL that gets built (blib/arch/auto/XML/Parser/Expat/Expat.dll) doesn't have any symbols from the object file built by the previous command because they are not marked for export in any way.

As to what could be interfering on your home-built Perl-5.10 installation, I don't know, but I would be happy as a barnacle by the bay if at some point we could figure it out, since I want this strategy to be as robust as possible.

It could help if you might be able to confirm the complete command line that is being run in the linking step that creates the DLL, so that we can confirm my diagnosis about --export-all-symbols, by running the build attempt again and capturing the console output.

Just other one point to get clarified right now (I'm going to separately mention your wise point about not removing the dlltool stuff if its working consistently in another note): you mentioned that "the CPAN release of XML-Parser-2.36 builds fine on both machines"; but it cannot be automatically built and installed using cpan, eh (the only way to build-test-install it is with the manual addition of commandline arguments)?

Gracias!

[syphilis]

but it cannot be automatically built and installed using cpan

That's a valid indictment of the way CPAN.pm works. It's not *just* XML::Parser that's subjected to this ... if I understand correctly, *any* module that needs to be built against a 3rd party library cannot be built using CPAN.pm iff the 3rd party library is in a location where perl is unable to find it (by default). On *nix, this is generally not a problem as the 3rd party libraries are normally installed in a location where perl *does* find them by default. But on Windows I, for one, build my 3rd party libraries using the MSYS shell, and install them into (what the MSYS shell regards as) /usr/local ... and perl does not find that "/usr/local" directory by default.

Actually, I think (based upon some recent private correspondence from adamk) that XML::Parser is a little special in that, even if the expat library is placed in a location where perl can find it by default, XML::Parser *still* won't build on Windows using CPAN.pm ... but complain about that to the XML::Parser author.

I'm sure CPAN.pm is very good, but I personally find it detestable (primarily because it ignores any instructions that have been given in the README) and don't use it. I tend therefore to also avoid modules such as Net::SSH::Perl which assume (via their copious use of other modules) that CPAN.pm is being deployed.

In short, since I don't use CPAN.pm (and since I'm not the author of XML::Parser) I don't really care about the caviats regarding the building of XML::Parser on Windows using CPAN.pm. I've personally found the building of XML::Parser to be quite straightforward (once I knew how :-)

However, I probably *will* get back to looking at why that disparity exists between Strawberry Perl and my own MinGW-built perl when building your patched XML-Parser-2.36. Curiosity is sure to get the better of me in the end :-)

Cheers,
Rob

[Anonymous Monk]

of course, I have no idea if this is the right place to ask this, but I'm desperate to ask it some where! just the usual storory... I am working in Mac OS10.5, and installing packages from cvs using fink. Can someone here tell me what is causing the following error? tdoak@129-79-233-163:~$ hawkeye (02-25 13:29) Can't find 'boot_XML__Parser__Expat' symbol in /usr/local/amos-2.0.8/lib/libExpa t.dylib at /usr/local/amos-2.0.8/lib/XML/Parser.pm line 14 Compilation failed in require at /usr/local/amos-2.0.8/lib/XML/Parser.pm line 14 . BEGIN failed--compilation aborted at /usr/local/amos-2.0.8/lib/XML/Parser.pm lin e 18. Let's just say that I've been banging my poor head on this for days now. Cheers, Tom

[Intrepid]

Update history on node 662656 (UnBefuzzling XML::Parser: an adventure with EU::MM method overrides):

1. Updated 03:52 PM Wednesday, January 16 2008 EST (US Eastern)
   • Added discussion of dlltool
2. Updated 04:54 PM Wednesday, January 16 2008 EST (US Eastern)
   • Added background, brief explanation of the EUMM module.
3. Updated 06:34 PM Wednesday, January 16 2008 EST (US Eastern)
   • Minor markup tidying.
   • Major addition: override on sub dynamic_lib() shown as code
   • Minor edits to text of last paragraph, to make my intention to express disagreement with downvoting based on social club hacker affiliation more clear. Essential inflammatory nature of original text not changed ("inflammatory" is in the eye of the beholder, dude).
4. Updated 06:46 PM Wednesday, January 16 2008 EST (US Eastern)
   • Minor text and markup changes to introductory paragraph that is at anchor #dllfool in the hypertext.
5. Updated 07:13 PM Wednesday, January 16 2008 EST (US Eastern)
   • Added <readmore> tags
6. Updated 02:33 AM Thursday, January 17 2008 EST (US Eastern)
   • Added a *note about using "verbose" when running Makefile.PL
   • Added expanded explanation of the effect of the dynamic_lib method override on the resulting operation when make runs, showing the commandlines output, before and after.
7. Updated 06:17 PM Saturday, Jan 19 2008 EST (US Eastern)
   • Minor changes to text and markup for section on "Background on ExtUtils::MakeMaker" and the section on "dlltool".

Update history on node 662868 (below: Re^4: UnBefuzzling XML::Parser: an adventure with EU::MM method overrides):

1. Updated 06:22 Fri, Jan 25 2008 EST (US Eastern)
   • Added discussion of loclibpth and locincpth.

Just a reminder:

As per the following on my home node:

---

I don't see posts by Anonymous Monk beginning at 6:30pm on 7 Dec 2005, by using CSS to override display settings. The texts of postings of any nature by any non-existing (Anonymous) Monk are invisible for me. This is a measure that I've taken until such time as the preferred solution comes to pass (the preferred solution being: the Perlmonks site no longer allows Anonymous posting at all).

---

[Anonymous Monk]

Hey. Nobody said you couldn't stick your head in the sand. Just because YOU don't see our posts doesn't mean that WE don't see our posts.

And it only proves that you are indeed THE biggest asshole at this site.