parallel programming and MCE - hubris and the frame

[Discipulus]

Introduction

Discipulus will never be a master: he likes too many things to become a master in one of them. He likes Perl. Here at the monastery he looks at tchnologies shown by monks, often like Linus watching russian names reading Dostoyevsky..

He looks at Perl parallel programming with a big interest: he suspects parallel programming will be one of major battlefield for next generations of programmers. He'd like these programmer to be Perl ones.

He has his cell carpeted of good examples of parallel programming. He also knows it is important to choose right masters to learn and follows with a big attention all parallel programming examples by, among others, BrowserUk and zentara.

Then it happened that marioroy joined the monastery: his work, MCE - Many-Core Engine for Perl hit Discipulus's curiosity strongly: perhaps with such tool even he can, simply and safely produce some decent parrallel Perl program? Infact he's a bit lazy and many working examples are difficult to adapt without falling into horrible pitfalls.

So Discipulus, taken his curage in his hands, has gone knocking at BrowserUk's and zentara's doors to ask their opinion...

(please note that I confuse parallel programming terminology, still. So when I say multithread you can read parallel: the dialogue is interesting beside my grossolane errors.

The Dialogue

Discipulus:I follow your posts about parallel multithreading programming since ever and I follow them as a pupil (discipulus in latin..) follow his masters. I say this withut any flattery intention. As you stated somewhere (iirc) multithreading in Perl is difficult to realize but is possible to do as you shown many many times.
What i will be glad to hear from you is an opinion about Perl MCE by marioroy available at http://search.cpan.org/~marioroy/MCE-1.827/ and visible in many posts by marioroy here in the monastery.

BrowserUk: I mostly don't recommend MCE, (though I have suggested people consider it a few times), simply because I don't us it personally.

zentara: I havn't the time to test MCE out. Besides I would rather do it manually, since I have total control of the code, as in Reusable threads demo.

Discipulus:Now I know you do not need such a thing like MCE to produce good multithread perl programs. but what about us? If i have the need or the curiousity to start write a threaded application i will follow your footsteps: i go to library, i search one of miles of your working complete examples, the closest to my need and i will expand it as my needs. But MCE seems a safe a clean alternative to have a multithread program without writing it from scratch.

zentara:Another thing, Don't confuse forks and threads. MCE is based on forks, from what I can gather, not threads. It gets more confusing because on MS Windows, all processes are threads, not forks.

BrowserUk:I don't use it personally be a) for the most part I don't need it; b) I have a very strong aversion to frameworks. But please note, I saying that I do not generally actively recommend MCE to people, I am not saying that I recommend against it.

Discipulus:MCE is an alternative if you accept the overhead of using a framework: but this is very common in programming: was possible to write webpages without CGI some years ago, writing a server too but CGI was a cheap overhead in front of many pitfalls possible doing all by hand. The same for DBI..
Is MCE something valuable in such perpsective? Is MCE a safer way to do parallel programming in perl, if you accept the minimal overhed and his learning curve?
Fast is fast, i rerun many bechmarks proposed at PM by marioroy and i can confirm. But is solid rock? this is behaind my capabilities to tell. It introduce other pitfalls?
I suspect MCE was highly undervaluated by the perl community and i think multithread is a key field for programmers of next generations: i'd like to see perl kicking competitors in the future.

BrowserUk:If you have an application for multi-tasking that MCE has a widget for, and especially if you can attract the attentions of MarioRoy to construct and refine some examples for your purpose (and perhaps tweak it internally to address any shortcomings your application throws up) then you could do a lot worse. From a cursory inspection, the code seems well designed, constructed and documented, and the author seems to have the time and energy to support you.
Beyond that I really don't have a lot to add.

zentara:As far as the threads controversy goes, these are my thoughts.

0. Threads tend to blow up unexpectedly, so don't use them for anything where you demand certainty and stability. Also many modules are not thread-safe,i.e. Tk

1. Only use threads if you need to share data in realtime between threads in an easy manner, with threads::shared. Otherwise forking a new process is more efficient because you do not have to copy the parent thread's code into the spawned thread, and the process totally frees it's memory when finished.

2. If you are going to use workers, the MCE module looks alright, although it reminds me of Parallel Fork Manager. If I was to chose the one foolproof way of doing the task of many workers sharing data in realtime, I would use Parallel::Fork::Manager along with SYSV Shared Memory segments. (zentara unfold a perchment with a wonderful example.. )

Discipulus: the answer you gave me was somehow expected and somehow unwanted: I know you have no need for such framework because you can write down the right way a multithread perl program using perl and it's basic tools.
But i've learn to not ask questions if i'm not prepared to receive back every answer :=)
I also understand your avversion to frameworks in general: why be constrained into a frame when my puissance can spread out free and untouched? this is a declination of the hubrys in it's positive, perlish sense.
But hubrys without solid rock foundations is foolishness and is where frameworks are valid options: I pay some of my freedom to the frame but i'm almost sure i'm not shooting on my own feets.
In this perspective i consider MCE a true gem: can MCE open the way to multithread Perl programming to many,many more peoples? I think this is it's virtue and this merit i suspect was underestimated.
I suspect and hope MCE can help perl programmers to afford such complex thing as multithread programming is.
If so we must spend some energy to push MCE forward and suggesting it as a way to start, safely, to do multithreaded perl programs.

(saying the above I not mean MCE is the only way to do it right with Perl parallel programming. As always TIMTOWTDT and any author can choose among many.)

Note that I've ask both monks the permission to reproduce parts of our mails: if i've put something in wrong order, if something sounds not to be the real opinion of such monks it is my complete fault and i'll correct

L*

[zentara]

Hi Discipulus, you give me more credit than I deserve, in as far as being an expert on threads and forks. I'm just a hacker. :-)

However, to continue sharing my observations, there is one thing you overlook in the whole discussion. It's the problem of i/o to the disk or network when writing programs for parallel processing. I/O is very slow compared to how fast the cpu runs, and unless you are running an 8-core Xeon machine, with a T1 internet line, most of the cpu time in the program is wasted in waiting for the i/o to complete. That fact makes parallel processing a tricky proposition because you may waste more resources trying to fork and spawn threads than you gain in overall program efficiency. Newer software designs like node.js and Perl6 are designed to address this issue. The general solution is to run single threaded "event-loop" programs which handle the i/o with "promises" and callbacks. This is essentially how gui code, like Tk or Gtk+ runs. Node.js is just a single threaded event-loop program which listens on a socket .

There is a simple explanation of this at asynchronous Perl6 and the author of Perl6's MoarVM, has an easy to watch video on youtube -> Worthington on Parallel programs in Perl6

Of course, frameworks like MCE, parallel forks, and spawning worker threads are still very useful in certain circumstances, but in general, the software of the future will be event-loop based solutions.

Finally, most of us, have single core or a few core computers. On machines like these, parallel processing is usually just an exercise in coding, and dosn't really speed anything up.

[1nickt]

Finally, most of us, have single core or a few core computers. On machines like these, parallel processing is usually just an exercise in coding, and dosn't really speed anything up.

Surely this depends on the task?

Parallel::ForkManager has long given the simple example of a script to download content from many websites. Here obviously the program is not bound by disk I/O but by network response, and parallelization results in many-fold speed increases. I am using MCE for similar tasks and "dosn't really speed anything up" could not be further from my experience.

I have switched recently from using Parallel::ForkManager to using MCE, mostly because of wanting the extra features MCE provides such as easy shared data structures with MCE::Shared, the fact that the author is very committed to supporting all major Oses, and the fact that the author is very quick to support and help.

Yes, I am a little mindful of the dangers of committing to a "framework", but as brother Discipulus says, we all use frameworks. Some have not been a good choice over my years in this work, but the vast majority of the ones I have chosen to use, after careful research and experimentation, have been reliable and powerful and useful. Perhaps Perl's prioritization of not breaking backwards compatibility is to thank for some of this, as well as the character of Perl developers and the Perl community.

Overall, MCE has been a great choice for me, your basic common-or-garden journeyman Perl developer (I'm not surprised you get different opinions from computer scientists like BrowserUk and zentara).

[zentara]

Hi, buried in my first reply was "Of course, frameworks like MCE, parallel forks, and spawning worker threads are still very useful in certain circumstances,", so I stand corrected and you are correct.

But I will have to agree with BrowserUk, that complex frameworks take alot of time to master, and often require the expert guidance of the module author, as shown in Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run. Without the author's guidance, I never would have been able to work that problem out. Also, I use Linux solely, and don't really care if stuff runs properly on Windows or not. I will use what is simple and easy for linux. I do applaud MCE's author for his effort though, it appears to be a great module.

[karlgoethebier]

"...parallel programming will be one of major battlefield for next generations of programmers"

And in the meantime (while we think about the magic one hand clapping koan) some other guys (using other programming languages) just fire up some more threads.

This dialogue made my day. Thanks.

[Discipulus]

an addendum: Your main event may be another's side-show. is a prequel of such discussion. There BrowserUk discuss about frame limitations even more in depth that in the current thread.

L*