Calling perl systems from other systems, e.g. R

[swl]

I have a large Perl code base and I want to develop bindings of some description so it can be called from other systems on a local machine, in this case R.

What are good ways of approaching this problem? And is there prior art that I can adapt?

 

The reason for this is that a large chunk of the user base lives in R, and it is neither practical nor desirable to rewrite the system in R. I have a GUI system in place that hides all the coding but this obviously cannot be integrated with automated workflows for people who are comfortable coding.

My thinking at the moment is to wrap the Perl libraries in a local service via some front-end. The R end of the system can then send data and commands to it, and then receive results and metadata back. Ideally all the low-level stuff on the Perl side would be handled by an existing system, e.g., maybe something like Mojolicious. The aim is to have one Perl process per R session to avoid headaches with shared memory (unless there is a simple solution for that).

This needs to work on the main operating systems (Linux, Windows, Mac, BSD variants). The Perl side of things will be packed up using PAR::Packer.

And if you’re wondering about PDL, it is already in use for many of the analyses.

[aitap]

Depending on the complexity of the interaction between R and Perl that you desire, R packages can include Perl scripts (subject to availability of Perl on the target machine, of course, e.g. gtools::read.xml, gtools::installXLSXsupport), or R itself can work as a server accepting commands from clients (e.g. Rserve).

If you take the HTTP server way, please be careful about accepting arbitrary code to evaluate over the local socket. Far too many modern applications start HTTP servers on localhost, only for someone else to discover that there's an unprotected endpoint that can be trivially accessed by an evil website by sending a few thousand requests of the form http://localhost:${port}/eval?system("pwn_the_machine"). Ideally, I would suggest UNIX domain sockets and Windows named pipes (which are easily restricted to the current user and cannot be accessed by rogue JavaScript in a browser), but getting R to speak those (especially the latter) can be hard.

[swl]

Many thanks for this.

I need to look more deeply but gtools seems to use system calls. My fallback is to run scripts using system calls, but ideally there would be persistence so the perl side of things can run more analyses without having to restart scripts and reload data.

It also requires the user have a working perl. My system has a large dependency tree that includes many XS libs so can't be fatpacked. Compiling using RTools and installing using a project level local::lib is not impossible but does get complex and hard to control across user environments. ...Although now I read more I see it can compile such libs.

Thanks also for the sockets/pipes advice. I knew about arbitrary code but had never considered external websites hitting local services.

[Fletch]

Doing the obvious and checking CPAN, there's Statistics::R and R::YapRI (have had people at $work use the former with some degree of success, just saw the other so YMMV . . .). I believe those however are more intended for letting you call into R from perl which sounds maybe the converse of what you're trying to do.

As far as your suggestion of turning the perl bits into a REST/web service that's probably as good a solution as any (and Mojolicious as good a framework as any). That'll let you make a clear break with a defined API and has the added benefit you then could have clients in ANY language call things (say let the python using heathens in on the fun too).

[swl]

Thanks. I was aware of Statistics::R but had not seen R::YapRI before. However, as you note, they go in the wrong direction for my use case.

A generic API is a good target. One could even access such a service from different versions of Perl, which is a not entirely facetious statement as the dependency tree for the client will be much smaller than for the engine and maybe there are cases where the Perl cannot be updated but a packed exe can be installed that takes advantage of newer features. That case is somewhat tangential to this thread, though.

[Discipulus]

Hello swl,

this sounds like a big project!

> This needs to work on the main operating systems (Linux, Windows..

I wish you all the luck of the world :)

Now my 2 cents: perhaps something already done like Rex can be used for this? like Rex automation?

I read in the docs:

> On the systems you want to manage you need only two things: a Perl 5 interpreter, and a valid account for the SSH server running there.

and it seems it is possible on unfortunate OSs too.

L*

[swl]

Potentially big, but I have someone else handling the R side of things, and there is a limited number of analytical options to make available on the Perl end.

Thanks for the links.

[NERDVANA]

You haven't said enough detail about the nature of the libraries you want to call, but if your needs are simple enough, you could just write a main loop that reads JSON lines from stdin, calls a function by name, and writes a JSON line back to stdout. This works on all OSes and calling perl functions by package name is so simple you don't need a framework. If you need to manage persistent objects, you could extend it with a concept of a "stash", or even just eval lines of input as perl, or maybe a combination of the two so that you get the benefit of parsing JSON for the data exchange but can still tell it to eval some things. Maybe the first line of input is JSON and the second line of input is code to eval using the $data from the previous line? Lots of options here.

I've never used R, but I assume most languages have the ability to start a child process and read/write lines to it.

[swl]

JSON might well be simple enough in the first instance, and I see JSON::Server has just had a new release. Longer term it might be more efficient to use other formats for data transfer but it's best to get things working.

[perlfan]

Creating a webservice frontend makes a lot of sense. The reason is that it provides an opportunity for the R dev and you to collaborate over a language-neutral API specification. You guys agree on the API, then go back to your hidy-holes and do your thing; you only need to communicate about the API. That'll keep you sufficiently insulated from anything having to do with R.