Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re: to thread or fork or ?

by mbethke (Hermit)
on Oct 19, 2012 at 02:25 UTC ( #999847=note: print w/replies, xml ) Need Help??


in reply to to thread or fork or ?

Unless the work you do is A LOT more expensive than a simple frequency count, or your data set extremely large and your disks very fast, you're best off using a single process. Synchronization is a lot of work with threads so accessing say a shared hash can be orders of magnitude slower than a non-shared one; with processes you'd end up having to serialize the end result and somehow pipe it back to the master process---also very expensive.

If you're sure you want this (maybe just as a learning experience), I'd suggest just using fixed-size chunks from the input stream per process/thread, to minimize shared data. Say, read a couple of megabytes plus a line into a single string (so as to read at maximum speed, plus the line so you don't split your work in the middle of a word), then start a thread to process it (split into words, optionally normalize, count) into a local hash that then goes into a queue read by the master thread that checks for results from worker threads in-between blocks.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://999847]
help
Chatterbox?
[choroba]: and in fact, you don't push into the target repo, you push into your branch of your fork
[choroba]: the maintainer of the upstream repo than "merges" the pull request, i.e. they pull from your fork into the upstream
[Discipulus]: ' i.e. you asked them to pull from your repo' =~ I (subj) want to push
[Discipulus]: chorobayour words are reasonable
[choroba]: I'm just repeating some else's words as I remembered them after having asked the same question
[choroba]: s/some/someone/
Discipulus what a pity all people do not speak only in eatalian..
[choroba]: we'd need video calls more often :)

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (6)
As of 2017-05-27 20:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?