|Syntactic Confectionery Delight|
Re: use threads for dir tree walking really hurts (MFT?)by Discipulus (Abbot)
|on Sep 01, 2016 at 08:26 UTC||Need Help??|
Very interesting topic, and ++Corion for the answer. Some sparse suggestion:
For the little i know the matter I suppose that beside the thread implementation you put on the field you must rely on the speed of the filesystem and of all underlying OS specific API call.
I doubt that more CPU can fetch a physical hard drive faster than a single one.
My experience fighting with windows is very long, so long to let me say that is by far better and faster to use as much as possible native tools offered (or well, concelead) by OS. An example is fetching file permissions: old Perl modules exhisted but wrapping around tools like icacls.exe is faster and less error prone and works for decades.
So going for native solutions I think the best would be to read the Master File Table directly: this intrigue me a lot but i suspect is a task by far beyond my hackery skills.
Also the Linux NTFS file system driver can be a reach source of information, if you are able to investigate a Linux driver.
For the thread part you can be interested in the marioroy's MCE that comes with many useful examples. At the monastery marioroy shown an exmple of dir walking using MCE in the thread Re: Perl threads - parallel run not working
Update: ATTENTION play your MFT tests on a test harddisk because the risk of corruption is always present!
Good luck and share your improvements!
There are no rules, there are no thumbs..
Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.