Architecture advice, proxy or rebroadcast websocket

[Your Mother]

I got some great advice for putting together my websocket client here: Mixing asynchronous data feed with synchronous program flow control. And it’s working well.

10,000 foot view: I am consuming a stock price stream. This is a paid service that only allows one connection and it is somewhat fragile due to a short timeout period, a dearth of documentation, and no built-in caching/reconnecting. None of my plans breach the user agreement/contract; that’s not me.

I don’t want to write a cache/queue. I want to leave the stream alone and uncluttered with a bunch of synchronous calls that might throw exceptions or otherwise cause trouble; separation of concerns. I also want to test multiple versions of code against the single stream concurrently. I could record and play back data but that’s a pain, a disk hog, and it doesn’t address the fragility/complexity or any possible application of code against live data.

My ideas have been–

• proxy the websocket through nginx so multiple “muted” client connections still only share the single permissible connection to the real service (I’m not sure this is possible without a lot of code and it’s extra plumbing),
• write a websocket server, in Mojo still, around the websocket client that lets an arbitrary number of local clients receive an echo of the master client’s stream,
• write a simple streaming socket server in Perl that runs the websocket client and pushes out to any local socket clients,
• publish the data stream to a local DB (this would be slower than everything else if it’s relational but also perhaps the most reliable with a built-in “cache/queue” only in need of pruning),
• or come beg you guys for what is probably a better idea than I can summon.

So… I guess the last one. :P

[Fletch]

Maybe heavier weight than you're looking for but similar to your local db idea: perhaps mirror into kafka and let your consumers pull from that.

[Your Mother]

++ Thank you. I was unfamiliar with kafka and if nothing else, you’ve edified me and given something to mull should this grow or another project come up.

[haukex]

Personally, I'd implement your second suggestion (a multiplexer in Mojo), with the thirdfourth suggestion (DB) in second place, and the other two sound too complicated for my tastes ;-) Here's how I might have implemented the second option, with the sample server and client based on your code in 11110853:

Read more... code (6 kB)

Note how in the multiplexer, I'm using a Mojo::EventEmitter to decouple the server messages from the client connections. Of course the code has a few assumptions that you are free to adapt, like whether you want to ignore or forward messages coming from each client to the server (in the latter case, you could also relay them via the event emitter using a different event name, for example), or and more generally how transparent you want to multiplexer to be (or not).

[Your Mother]

You are a hero.