Re: Does perl have a builtin limit to the size of shared memory segments I can write to?

[NERDVANA]

Yes, this looks like a bug in Perl.

Perl_do_shmio(pTHX_ I32 optype, SV **mark, SV **sp)
{
#ifdef HAS_SHM
    char *shm;
    struct shmid_ds shmds;
    const I32 id = SvIVx(*++mark);
    SV * const mstr = *++mark;
    const I32 mpos = SvIVx(*++mark);
    const I32 msize = SvIVx(*++mark);

The parameters are stored into variables declared as I32, rather than SSize_t.

The mem-read and mem-write are performed in Perl rather than a system call, so there's no reason to truncate the offset and length to 32-bit.

if (optype == OP_SHMREAD) {
    char *mbuf;
    /* suppress warning when reading into undef var (tchrist 3/Mar/00)
+ */
    SvGETMAGIC(mstr);
    SvUPGRADE(mstr, SVt_PV);
    if (! SvOK(mstr))
        SvPVCLEAR(mstr);
    SvPOK_only(mstr);
    mbuf = SvGROW(mstr, (STRLEN)msize+1);

    Copy(shm + mpos, mbuf, msize, char);

I'm afraid you won't be able to get a fix for this unless you recompile your own perl, but please consider filing a bug report so that perl 5.42 could have it fixed.

Meanwhile, there is File::Map which is basically the same thing, except you have to create your own files (in tmpfs if you want them to be pure RAM). You also have to be careful not to accidentally copy that mapped scalar, which would be expensive.

[Anonymous Monk]

Thanks for the prompt reply.

I opened an issue about that.

Recompiling my own perl is not an option, as I aim to share the code and attain some degree of portability (on Linux, no plans to extend this to other OS). I will carefully look into the File::Map option, or perhaps, I can find a simpler workaround by splitting my large index over a few shared memory segments, for now.

Best,

Valerio

[Corion]

Thank you very much for opening it!

Just for reference, the issue is GH #22895.