Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options

Re^3: RE on lines read from in-memory scalar is very slow

by tonyc (Friar)
on Feb 20, 2024 at 22:57 UTC ( [id://11157819]=note: print w/replies, xml ) Need Help??

in reply to Re^2: RE on lines read from in-memory scalar is very slow
in thread RE on lines read from in-memory scalar is very slow

The problem with what I think you're suggesting is readline/sv_gets() doesn't know about the SV behind a file handle created from a reference, it's just another file - one where the buffer isn't the normal size.

As to the modifications to the SV the file handle refers to, making such a file handle doesn't copy the SV - changes to the SV are reflected in the values returned by reads from the handle, and writes to the handle are (obviously) reflected when reading from the SV. (changes to the SV have led to some bugs).

Note that in the match case, on success perl makes a CoW copy of the whole matched SV, and the magic for $1, $& etc copies the appropriate text from from that CoW SV. That CoW copy is part of the problem here.

As to the multiple 16M allocations, that's the other part of the bug here, sv_gets() is preallocating the target SV to way too large a size.

For your performance issue, you could post here (on perlmonks, not necessarily this thread) with the code and performance numbers for discussion and/or make a ticket against perl itself on github. The main issue I can think of comparing readline() vs split is readline goes through more layers (pp_readline, sv_gets, PerlIO, PerlIO::scalar vs pp_split, regexp) which is going to slow things down. A test case can be investigated with a profiler (gprof, perf record, valgrind callgrind, cygwin profiler, visual studio profiler) to see if something is consuming more than the expected CPU time.

  • Comment on Re^3: RE on lines read from in-memory scalar is very slow

Replies are listed 'Best First'.
Re^4: RE on lines read from in-memory scalar is very slow
by NERDVANA (Curate) on Feb 20, 2024 at 23:43 UTC

    I had forgotten about reflecting writes to the string, that would certainly nix the idea of using copy-on-write substrings.

    Does the regex engine *always* make a full clone of the target? Can't that itself be a copy-on-write? Maybe only if the match target was already copy-on-write?

    Maybe what I'm suggesting here is an optimization for sv_gets that kicks in when

    1. The file handle is backed by a scalar
    2. The scalar already has the CoW flag set on it, or scalar itself has a refcount of 1 (meaning that the app no longer has access to modify it)
    3. The file handle is read-only,
    4. The file handle has only simple layers on it like the default or :raw (maybe extend to support :utf8)
    then short-circuit around the regular behavior and return a CoW substring? As I describe it, its sounding like too much effort, but I'm still curious if it would work.

      The regex engine currently always makes a CoW copy (ie doesn't copy the string itself) of the matched string.

      There's no CoW substrings - perl's PVs are stored with a trailing NUL so they're useful as a C-style string, and perl itself and XS code depends on that, so I don't think it can be implemented in any case.

        Ah-ha! Well that helps improve my understanding for future attempts at optimizing my scripts.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11157819]
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others pondering the Monastery: (3)
As of 2024-05-22 02:02 GMT
Find Nodes?
    Voting Booth?

    No recent polls found