Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re^2: use threads for dir tree walking really hurts

by exilepanda (Pilgrim)
on Sep 01, 2016 at 13:55 UTC ( #1170969=note: print w/replies, xml ) Need Help??


in reply to Re: use threads for dir tree walking really hurts
in thread use threads for dir tree walking really hurts

Thank you very much for your vivid elaboration which is very inspiring. =D
Why are you doing that?
Because when an object fall into a thread scope, the object will be cloned, which is not the one I want. And since threads don't share object / complex data structure ( and I don't what to share them one by one ), this trick do share the object perfectly... until it's not.

Actually, I can do the job with simply: @dirToFetch : shared, but same issue Thread::Queue, I gotta leave it at a nested package scope , but create it inside an object become another mess to share around threads. Because I attempt to make it a module, so I hope to avoid if other script calling this module in threads, the data will mess up.

Though, I've update my OP's code, which will work and as fast as dir /s/b I create as many threads as how much in @dirToFetch

Replies are listed 'Best First'.
Re^3: use threads for dir tree walking really hurts
by Corion (Pope) on Sep 01, 2016 at 14:08 UTC
    Why are you doing that?
    Because when an object fall into a thread scope, the object will be cloned, which is not the one I want. And since threads don't share object / complex data structure ( and I don't what to share them one by one ), this trick do share the object perfectly... until it's not.

    Yes - due to Perls reference counting, accessing variables in another threads memory always means that your thread will also be writing at least to the refcount field of that variable. If the refcount happens to reach zero in another thread than where the piece of memory was originally allocated, the memory will be freed in the wrong thread context, which is not fun.

    I'm not aware of a way to make Perl skip its refcounting for variables, and I'm also not convinced that this could work except in the most trivial cases.

    Another idea to reduce the conceptual load of the appriach might be to simply shell out to cmd /c "dir /b /s $directory", but then you need to be aware of the codepage that cmd.exe uses for its output. Ideally you have set the codepage to Unicode / 65001:

    chcp 65001

    ... but then, you still have to live with the fun of Perl and the OS treating the octets for filenames differently unless you properly decode and encode them.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1170969]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (3)
As of 2020-02-29 04:48 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What numbers are you going to focus on primarily in 2020?










    Results (128 votes). Check out past polls.

    Notices?