Is there a specific reason for it to be perl-only? Tight integration with perl internal structures?
I would recommend rolling this up as an independent library first, if possible. Or even just a separate translation unit. The XS is best used as a thin glue.
Are you sure that assembly is a requirement for this algorithm? Modern C compilers can autovectorize code; intrinsics are also available where needed. Furthermore, x86_64 CPUs span a number of feature-sets, beginning with just SSE3 (or is it SSE2?), going up to AVX2, and further, with miscellaneous extensions like POPCNT, BMI, FMA, and so on. Major speedups are often obtained by targeting the best feature-set.
So it comes down the particulars of your algorithm, and how it relates to perl.