Improving core performance with SIMD

[phizel]

After running across the StringZilla library, it got me thinking that Perl could greatly benefit from similar improvements. After a glance through perlfunc and Module::CoreList the following is a list I think could be targeted for optimization. Thoughts?

• reverse
https://github.com/Perl/perl5/issues/20692
• lc, uc, fc
http://0x80.pl/notesen/2025-02-02-utf32-change-case.html
• index, rindex
http://0x80.pl/notesen/2016-11-28-simd-strfind.html
• eq, ne
https://www.strchr.com/strcmp_and_strlen_using_sse_4.2
• tr, quotemeta
https://github.com/lemire/despacer
• split ' '
• join
• sort

core modules:

• Encode
https://github.com/simdutf/simdutf
• Digest::MD5
• Digest::SHA
https://github.com/minio/sha256-simd
https://github.com/qinwang/cryptopp/blob/master/sha-simd.cpp
• File::Compare
• MIME::Base64
http://0x80.pl/notesen/2016-09-17-avx512-foundation-base64.html
https://github.com/simdutf/simdutf?tab=readme-ov-file#base64
• Unicode::Normalize

non-trivial:

• pack/unpack
• regex, split //
https://github.com/intel/hyperscan
https://github.com/VectorCamp/vectorscan

[Corion]

Yes and no. Personally, I would look at Daniel Lemire's work on SIMD (https://github.com/lemire), but it's not trivial to employ the code.

Intel Hyperscan does not do capturing, and does not tell you where a match started. It is mostly aimed at network traffic scanners, where that is not necessary. This limitation would make it only suitable for very limited situations.

I think the best approach would be to find a library that does good detection of SIMD capabilities of a machine, and then port (or copy) selected parts into the appropriate Perl code guarded by the appropriate SIMD define. Finding a library that has both, (one or more) SIMD-variants and a non-SIMD plain C variant is important here, since otherwise, you will get feature disparity between SIMD and non-SIMD code.

[ysth]

There may be some things of help here, but given that perl strings may be utf8, not bytes, it isn't going to be easy.

[Fletch]

Were I to look for prior art I'd look at what ripgrep and the Rust regex engine do. I want to say its author had benchmarks showing that it was (one of)? the fast(est|er) UTF handling engines.

[cavac]

Yup, and a lot of processing power is required to correctly handle Unicode... no matter the programming language. That was always the compromise with Unicode, and it's a good one overall. Human languages and writing systems are messy, based on weird and complicated rules that evolved over centuries. At least we are not having to deal with codepages/"extended ASCII" anymore.

[Anonymous Monk]

Should somebody release a StringZilla interface on CPAN?