|Just another Perl shrine|
parallel programming and MCE - hubris and the frameby Discipulus (Monsignor)
|on Apr 10, 2017 at 11:00 UTC||Need Help??|
Discipulus will never be a master: he likes too many things to become a master in one of them. He likes Perl. Here at the monastery he looks at tchnologies shown by monks, often like Linus watching russian names reading Dostoyevsky..
He looks at Perl parallel programming with a big interest: he suspects parallel programming will be one of major battlefield for next generations of programmers. He'd like these programmer to be Perl ones.
He has his cell carpeted of good examples of parallel programming. He also knows it is important to choose right masters to learn and follows with a big attention all parallel programming examples by, among others, BrowserUk and zentara.
Then it happened that marioroy joined the monastery: his work, MCE - Many-Core Engine for Perl hit Discipulus's curiosity strongly: perhaps with such tool even he can, simply and safely produce some decent parrallel Perl program? Infact he's a bit lazy and many working examples are difficult to adapt without falling into horrible pitfalls.
(please note that I confuse parallel programming terminology, still. So when I say multithread you can read parallel: the dialogue is interesting beside my grossolane errors.
Discipulus:I follow your posts about parallel multithreading programming since ever
and I follow them as a pupil (discipulus in latin..) follow his masters. I say this withut any flattery intention. As you stated somewhere (iirc) multithreading in Perl is difficult to realize but is possible to do as you shown many many times.
BrowserUk: I mostly don't recommend MCE, (though I have suggested people consider it a few times), simply because I don't us it personally.
Discipulus:Now I know you do not need such a thing like MCE to produce good multithread perl programs. but what about us? If i have the need or the curiousity to start write a threaded application i will follow your footsteps: i go to library, i search one of miles of your working complete examples, the closest to my need and i will expand it as my needs. But MCE seems a safe a clean alternative to have a multithread program without writing it from scratch.
zentara:Another thing, Don't confuse forks and threads. MCE is based on forks, from what I can gather, not threads. It gets more confusing because on MS Windows, all processes are threads, not forks.
BrowserUk:I don't use it personally be a) for the most part I don't need it; b) I have a very strong aversion to frameworks. But please note, I saying that I do not generally actively recommend MCE to people, I am not saying that I recommend against it.
Discipulus:MCE is an alternative if you accept the overhead of using a framework: but this is very common in programming: was possible to write webpages without CGI some years ago, writing a server too but CGI was a cheap overhead in front of many pitfalls possible doing all by hand. The same for DBI..
BrowserUk:If you have an application for multi-tasking that MCE has a widget for, and especially if you can attract the attentions of MarioRoy to construct and refine some examples for your purpose (and perhaps tweak it internally to address any shortcomings your application throws up) then you could do a lot worse. From a cursory inspection, the code seems well designed, constructed and documented, and the author seems to have the time and energy to support you.
zentara:As far as the threads controversy goes, these are my thoughts.
0. Threads tend to blow up unexpectedly, so don't use them for anything where you demand certainty and stability. Also many modules are not thread-safe,i.e. Tk
1. Only use threads if you need to share data in realtime between threads in an easy manner, with threads::shared. Otherwise forking
a new process is more efficient because you do not have to copy the parent thread's code into the spawned thread, and the process
totally frees it's memory when finished.
2. If you are going to use workers, the MCE module looks alright, although it reminds me of Parallel Fork Manager. If I was to chose the one foolproof way of doing the task of many workers sharing data in realtime, I would use Parallel::Fork::Manager along with SYSV Shared Memory segments. (zentara unfold a perchment with a wonderful example.. )
Discipulus: the answer you gave me was somehow expected and somehow unwanted: I know you have no need for such framework because you can write down the right way a multithread perl program using perl and it's basic tools.
(saying the above I not mean MCE is the only way to do it right with Perl parallel programming. As always TIMTOWTDT and any author can choose among many.)
Note that I've ask both monks the permission to reproduce parts of our mails: if i've put something in wrong order, if something sounds not to be the real opinion of such monks it is my complete fault and i'll correct
There are no rules, there are no thumbs..
Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.