I changed the two occurrences of return to XSRETURN, and the build succeeded
Whilst that will enable the build to succeed, it's unfortunately not the right solution. XSRETURN() takes, as its argument, the number of elements to return from the stack. According to perldoc perlapi:
Return from XSUB, indicating number of items on the stack. This is u
+sually handled by "xsubpp".
void XSRETURN(int nitems)
I'm not sure what will happen if your version of that block of code gets executed and an attempt is made to return gsl_nan() items off the stack ... but I fully expect that it won't be pretty. If that block of code gets executed, then it needs to XSRETURN(1), and the one item that it needs to return from the stack is whatever-it-is-that-is-returned-by gsl_nan(). (Presumably that's either a NaN, or a Math::GSL object that holds the value NaN.)
So long as that block of code never gets executed I think your fix should, however, be fine.