|The stupid question is the question not asked|
I changed the two occurrences of return to XSRETURN, and the build succeeded
Whilst that will enable the build to succeed, it's unfortunately not the right solution.
XSRETURN() takes, as its argument, the number of elements to return from the stack. According to perldoc perlapi:
I'm not sure what will happen if your version of that block of code gets executed and an attempt is made to return gsl_nan() items off the stack ... but I fully expect that it won't be pretty.
If that block of code gets executed, then it needs to XSRETURN(1), and the one item that it needs to return from the stack is whatever-it-is-that-is-returned-by gsl_nan(). (Presumably that's either a NaN, or a Math::GSL object that holds the value NaN.)
So long as that block of code never gets executed I think your fix should, however, be fine.