Rationale for why successive Perl releases are not XS-compatible?

vrk has asked for the wisdom of the Perl Monks concerning the following question:

Dear monks,

at $work, we use SWIG to generate a Perl wrapper for a large C++ library. SWIG creates glue code in XS and pure Perl based on the C++ class definitions, and the XS code is then compiled against the header files of a particular version of Perl. This works great and you can use the same SWIG interface file to generate a Python, Java and C# wrapper as well.

The problem is, when the XS code is compiled against a particular version of Perl, say 5.18, the resulting compiled library is only compatible with the same (major) version of Perl. Our customers use a range of Perl versions, from 5.10 through to the latest one, so we have to compile and provide support for an ever increasing number of versions, even though the actual C++ library is exactly the same in each case. It was manageable for 5.6 and 5.8, which were long-lived versions, but it's become exasperating since the move to annual Perl releases.

Can you point me to any resources that describe or discuss the rationale for why XS code could not be made binary compatible between Perl versions? Or why there couldn't be some kind of reduced XS API that would stay binary compatible even if some parts change? Which authority should I petition to make it so?

Comment on Rationale for why successive Perl releases are not XS-compatible? Download Code

Replies are listed 'Best First'.
Re: Rationale for why successive Perl releases are not XS-compatible? by salva (Canon) on Mar 17, 2017 at 12:21 UTC
Mostly because perlapi is not a thoroughly and well designed API but just a subset of the Perl internals exposed, including functions, structures and macros. That means that whenever the representation of any structure used by perl internally changes, all the code must be recompiled and as you can imagine, evolving the language requires changing its internals...	[reply]
Re: Rationale for why successive Perl releases are not XS-compatible? by marto (Cardinal) on Mar 17, 2017 at 12:34 UTC
perl 5 and binary compatibility, Super search for further discussion.	[reply]
Re^2: Rationale for why successive Perl releases are not XS-compatible? by vrk (Chaplain) on Mar 17, 2017 at 15:57 UTC
chromatic's blog post is a good read, but as he points out, it would actually be beneficial to have an extra layer between perl's internals and user modules to prevent XS code from accessing the internals. I don't disagree with the necessity for changing the internals periodically. It would just be nice to have a binary compatible interface that shields you from those changes...	[reply]
Re^3: Rationale for why successive Perl releases are not XS-compatible? by Anonymous Monk on Mar 17, 2017 at 17:14 UTC
The perl api mostly consists of inline macros, for speed. It should be relatively simple to design a semistable binary api: just provide out-of-line-compiled functions for each of the inline macros. There would be a slight speed penalty, which is why nobody has done it.	[reply]
Re: Rationale for why successive Perl releases are not XS-compatible? by stevieb (Canon) on Mar 17, 2017 at 13:22 UTC
Perl 5 Porters are the authority you speak of. You can contact them through their mailing list. Perhaps it would be prudent to peruse the archives (in the same link) before reaching out in case it's been brought up before.	[reply]
Re^2: Rationale for why successive Perl releases are not XS-compatible? by vrk (Chaplain) on Mar 17, 2017 at 15:51 UTC
A quick search with a favourite search engine shows it has been discussed a couple of times in the past few years, so I'll need to spend some time reading through the archives.	[reply]
Re: Rationale for why successive Perl releases are not XS-compatible? by Anonymous Monk on Mar 17, 2017 at 16:27 UTC
Just spin up a server, install Perlbrew on it and install versions of Perl as per your customers' needs. Use Jenkins to build your software on the target Perl. It's easier than you might think.	[reply]
Re^2: Rationale for why successive Perl releases are not XS-compatible? by Anonymous Monk on Mar 17, 2017 at 21:25 UTC
easier when it works, if it works	[reply]
Re: Rationale for why successive Perl releases are not XS-compatible? by vrk (Chaplain) on Mar 27, 2017 at 07:55 UTC
Some blue sky thinking... FFI::Platypus is a libffi interface for Perl. You could modify SWIG to produce pure Perl glue code targeting FFI::Platypus instead of producing XS. Then you only need to worry about having FFI::Platypus installed on customer systems. But it would only solve the problem if 1) someone wrote a SWIG backend (some months of work) and 2) FFI::Platypus is easy to install (haven't checked).	[reply]


Problems? Is your data what you think it is?
	PerlMonks