Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Using MinGW to build perl on windows forcing it to expand * into glob in cmd.exe

by Discipulus (Abbot)
on Feb 14, 2021 at 16:01 UTC ( #11128365=perlmeditation: print w/replies, xml ) Need Help??

Hello folks,

-Introduction-

A recent post evidentiated a bug in some raku builds for windows: if compiled using MinGW (instead of compiled with MS tools as rakubrew uses) an unexpected expansion of * into a glob happens in cmd.exe prompt. The thread born because Athanasius found impossible to escape this * in cmd.exe (more on this later on).

Contributions to the above threads highlighted a fact unkonwn to me: there is a tiny layer, the C runtime library, responsible to find argc and build argv to pass to main (see jcb's answer). The actual C runtime library depends on the tool used to build perl (or raku). At least this is what I understood.

While the above is interesting per se triggered another question in my mind: if a raku built using MinGW unintentionally expands * there must be a way to build a perl that expands too * into a glob. You will see later on why this is a futile question, but: futile, complex and hackish is exactly the kind of task I love.

-Prepare the ground-

To build this new frankestein perl we need: a recent strawberry portable edition: strawberry-perl-5.32.1.1-64bit-portable is what I used. Extract this wherever you want: in the following examples I will use C:\ulisse\perl5.32-64bit

Then we need to download perl-5.32.1.tar.gz.

Finally make an empty dir to use as playground. Open the folder where you extracted the strawberry portable edition and launch the portableshell.bat then in the prompt window execute (adjust to your needs):

---------------------------------------------- Welcome to Strawberry Perl Portable Edition! * URL - http://www.strawberryperl.com/ * see README.TXT for more info ---------------------------------------------- Perl executable: C:\ulisse\perl5.32-64bit\perl\bin\perl.exe Perl version : 5.32.1 / MSWin32-x64-multi-thread C:\ulisse\perl5.32-64bit>x: X:\>mkdir globperl X:\>cd globperl X:\globperl>

Then just to be sure nothing in PATH will interfere with our experiments clean up the PATH as much as possible, leaving only strawberry directories and windows system ones (adjust to your needs):

X:\globperl>set PATH=C:\ulisse\perl5.32-64bit\perl\site\bin;C:\ulisse\ +perl5.32-64bit\perl\bin;C:\ulisse\perl5.32-64bit\c\bin;C:\Windows\sys +tem32;C:\Windows;

Finally extract the source of perl downloaded from CPAN, perl-5.32.1.tar.gz to X:\globperl\perl-5.32.1

-Modify the source code-

Browse the folder X:\globperl\perl-5.32.1\win32 and edit the win32.c file, in my case around line 73 and change:

#ifdef __GNUC__ /* Mingw32 defaults to globing command line * So we turn it off like this: */ int _CRT_glob = 0; #endif

the relevant line into int _CRT_glob = 1; save and close the file.

In the same folder edit the runperl.c file around line 19 in my case putting again int _CRT_glob = 1; and so resulting in:

#ifndef PERLDLL int _CRT_glob = 1; #endif

Save and close.

-Compile it-

In the above portableshell.bat window move to the cd X:\globperl\perl-5.32.1\win32 directory where win32 perl source is. Now we have to instruct gmake ( which will use GNUmakefile inside win32 folder while Makefile and makefile.mk are used by other flavours of make) to install our new perl inside a custom dir. We can modify the GNUmakefile (around lines 49 and 50) to use different INST_DRV and INST_TOP or we can feed this parameter from command line.

So we are ready to build and install our new perl:

X:\globperl\perl-5.32.1\win32> gmake INST_DRV=x: INST_TOP=X:\globperl\ +my_perl # CCTYPE=GCC # GCCBIN=gcc # GCCVER=8.3.0 # GCCTARGET=x86_64-w64-mingw32 # GCCCROSS= # WIN64=define # ARCHITECTURE=x64 # ARCHNAME=MSWin32-x64-multi-thread # MAKE=gmake [...] X:\globperl\perl-5.32.1\win32> gmake install [...]

-First test and dll hell-

This shouldnt happen but as happened to me can also happens to you. Open a new command prompt ( not a portableshell.bat but a regular plain cmd.exe prompt). I erase completely PATH to be sure nothing will interefere, then I tried to invoke our brand new perl

C:\Users\io>set PATH= C:\Users\io>x:\globperl\my_perl\bin\perl.exe -v

A nasty error window pops up complaining libgcc_s_seh-1.dll is missing. It is present in the C:\ulisse\perl5.32-64bit\perl\bin strawberry folder and should be available during building, but we can copy in our brand new perl\bin directory to overcome the error, but then another error pops up about missing libwinpthread-1.dll and then again about libstdc++-6.dll so we copy these missing dlls and finally we got a sane perl.exe

C:\Users\io>copy c:\ulisse\perl5.32-64bit\perl\bin\libgcc_s_seh-1.dll +x:\globperl\my_perl\bin 1 file copied. C:\Users\io>copy c:\ulisse\perl5.32-64bit\perl\bin\libwinpthread-1.dll + x:\globperl\my_perl\bin 1 file copied. C:\Users\io>copy "c:\ulisse\perl5.32-64bit\perl\bin\libstdc++-6.dll" x +:\globperl\my_perl\bin 1 file copied. C:\Users\io>x:\globperl\my_perl\bin\perl.exe -v This is perl 5, version 32, subversion 1 (v5.32.1) built for MSWin32-x +64-multi-thread Copyright 1987-2021, Larry Wall

Hurrah!!

Sometimes the dll hell can be even worst and the sybilline The application was unable to start ... 0x000007b error appears: in this case you can profit of the wonderful program Dependencies to dipanate the dll hell.

-Fire proof-

Now let's look if our new perl is able to expand * into glob on cmd.exe as we wanted to prove:

X:\globperl\my_perl\bin\perl -e "print join ' ', @ARGV" * corelist.bat cpan.bat enc2xs.bat encguess.bat h2ph.bat h2xs.bat instmo +dsh.bat json_pp.bat libgcc_s_seh-1.dll libnetcfg.b at libstdc++-6.dll libwinpthread-1.dll perl.exe perl5.32.1.exe perl532 +.dll perlbug.bat perldoc.bat perlivp.bat perlthank s.bat piconv.bat pl2pm.bat pod2html.bat pod2man.bat pod2text.bat pod2u +sage.bat podchecker.bat prove.bat ptar.bat ptardif f.bat ptargrep.bat shasum.bat splain.bat xsubpp.bat zipdetails.bat

Tadąaaa! :)

-Why not?-

Now we have a perl expanding * into a glob, but... what if we need to pass a regex? Oh well we can escape an eventual * you'd say. No: this * is impossible to escape!

cd x:\globperl x:\globperl\my_perl\bin\perl -e "print join ' ', @ARGV" * my_perl perl-5.32.1 x:\globperl\my_perl\bin\perl -e "print join ' ', @ARGV" "*" my_perl perl-5.32.1 x:\globperl\my_perl\bin\perl -e "print join ' ', @ARGV" ^* my_perl perl-5.32.1 x:\globperl\my_perl\bin\perl -e "print join ' ', @ARGV" '*' '*' x:\globperl\my_perl\bin\perl -e "print join ' ', @ARGV" "^*" ^* # this is really fun! It globs the root of the current drive x: # Infact in windows each drive letter has its own root: \ x:\globperl\my_perl\bin\perl -e "print join ' ', @ARGV" \* \$RECYCLE.BIN \globperl x:\globperl\my_perl\bin\perl -e "print join ' ', @ARGV" \\* \\*

No way. It is because of this that Athanasius spotted a bug in the above mentioned post

-Acknowledgements and useful reads-

Thanks to Athanasius, jcb and sortiz (who supported me also in the #raku irc channel) for their contributions to the raku post.

I got an invaluable and patient help from a wise one in the #perl freenode irc channel who pointed me to the relevant source code modifications needed and helped me to escape from dll hell.

They also explained me that glob expansion is disabled by default because MinGW used to enable it by default, but now while mingw-w64 crt defaults to disabled globbing, mingw-builds scripts, which is what most people use to build mingw, instead of doing that manually, do enable glob expansion by default.

There are other discrepancies: while in mingw-w64 _CRT_glob is a boolean in mingw.org it's a bitfield.. so good luck.

See mingw-w64-headers/crt/_mingw.h.in and confront with _mingw.h.in (search for: "* Manifest definitions for flags to control globbing of the command line").

Also an interesting page about escaping in cmd.exe and a SO answer and dont miss everyone-quotes-command-line-arguments-the-wrong-way ( the last one added 2021-02-15, found here)

Nice, no?

L*

There are no rules, there are no thumbs..
Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.

Replies are listed 'Best First'.
Re: Using MinGW to build perl on windows forcing it to expand * into glob in cmd.exe
by syphilis (Bishop) on Feb 15, 2021 at 00:00 UTC
    /* Mingw32 defaults to globing command line * So we turn it off like this: */
    The comments in runperl.c provide some additional explanation:
    /* Mingw32 defaults to globing command line * This is inconsistent with other Win32 ports and * seems to cause trouble with passing -DXSVERSION=\"1.6\" * So we turn it off ....
    From that, we gather that one reason for setting _CRT_glob to 0 was to bring perl into line with perls built using a Microsoft compiler.
    IMO, this is no longer a valid reason. (I'm not even sure that it was ever a valid reason.)

    But then there's also a vague reference to a problem with passing --DXSVERSION=\"1.6\" when _CRT_glob is set to 1.
    Do we know what that's about ?
    Do we know that it's no longer an issue ?
    How does make test fare ?

    There are other discrepancies: while in mingw-w64 _CRT_glob is a boolean in mingw.org it's a bitfield.. so good luck.

    A couple of weeks ago I tried building perl with mingw.org's gcc-9.2.0, but got stumped by their recent decision to change sizeof(FD_SET) to "1". Both Microsoft (at least up to VS 2019) and mingw-w64 compilers have sizeof(FD_SET) == 260 (or 520 for mingw-w64 64-bit compilers). I don't know why mingw.org changed that spec, but these days they do seem to have quite a knack for making things difficult.
    To put it bluntly, I regard mingw.org as currently irrelevant - though that's perhaps unfair.

    Cheers,
    Rob
Re: Using MinGW to build perl on windows forcing it to expand * into glob in cmd.exe
by LanX (Cardinal) on Feb 14, 2021 at 16:45 UTC
    Thanks for the interesting read!

    > Why not?

    I hope this didn't suprise you much, otherwise I should have been clearer when I said

    avoid cmd.exe as a shell

    :)

    FWIW, some problems are mitigated when you start Perl from PowerShell.

    But in an ideal world we'd have a standard Perl CLI/shell/REPL which works consistently on all platforms.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

    Edit <5min

    Minor rewording and typo corrections

Re: Using MinGW to build perl on windows forcing it to expand * into glob in cmd.exe
by Anonymous Monk on Feb 14, 2021 at 22:04 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://11128365]
Approved by marto
Front-paged by marto
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (3)
As of 2021-03-07 17:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My favorite kind of desktop background is:











    Results (122 votes). Check out past polls.

    Notices?