|
in reply to Re^2: ithreads, locks, shared data: is that OK?
in thread ithreads, locks, shared data: is that OK?
You don't need to lock the queue for enqueue and dequeue. You need it for peek, but why do you need peek? Enqueue and dequeue should be all you need :-)
($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord
}map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,
|
Re^4: ithreads, locks, shared data: is that OK?
by bliako (Monsignor) on Sep 20, 2018 at 10:08 UTC
|
got it thanks
well, I need to peek() because one of my queues is not a "work" queue: i.e. enqueue() any data for processing by the thread. Rather, it is a list of all data (=dictionary words) currently being processed by threads (call it CWq queue). And another one is a list of all the words that already have been processed and done with (call it REq). So, before a thread processes word W, it must peek() queue CWq and see if W is in there. In which case it will skip it. Also it will skip if the word is in the REq, so another peek() there.
Ideally CWq and REq should have been a hash but I find sharing a hash way too complicated than lock() and peek() a queue. A queue is definetely a weird hammer for that sort of nail. Any suggestions?
| [reply] |
|
|
I don't fully understand your workflow. If you want to process each word just once, create a thread that keeps a hash of the processed words and ask it before sending any word forward maybe?
($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord
}map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,
| [reply]
[d/l] |
|
|
Thanks Choroba for your time. I am bit apprehensive to start a thread to act as a data-query server. What about the communication overhead? Bottlenecks?
*Or* does a Thread::Queue or a threads::shared hash imply the exact same overheads and bottlenecks because of Perl's ithreads implementation?
summary: what you propose and my workflow:
Your suggestion: start a thread to handle enquiries about data (remember i need to do 2 checks, 1 if word is currently being processed and 2 if word has already been processed) and maybe also put the dictionary in there. So that eliminates all shared data and insane duplication of a readonly dictionary over all threads.
my workflow: I am keeping track of first 2 cases above using a queue which I basically (mis-)treat as a hash. I loop over its elements and peak() trying to find my word. Or I loop and peak() and if found, I dequeue() it. Ideally I should have used a shared hash for each of the two words. And a shared dictionary.
| [reply]
[d/l] |
|
|
|
|
|
|
