Beefy Boxes and Bandwidth Generously Provided by pair Networks Frank
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re^8: randomising file order returned by File::Find

by jeffa (Chancellor)
on Mar 01, 2011 at 23:29 UTC ( #890864=note: print w/ replies, xml ) Need Help??


in reply to Re^7: randomising file order returned by File::Find
in thread randomising file order returned by File::Find

"Only once you have turned over your cluster to single purpose, Java-based HDFS monoculture. And if all your processing needs can be force fitted into that monoculture's way of working."

That is simply not true. Look into Hadoop streaming.

"My needs, resources and pockets do not lend themselves to such."

That's too bad. I am currently working for a company and we are in the process of replacing our traditional means of processing the extremely massive volumes of data we consume with Hadoop. It is beyond awesome. Again, you should check it out instead of making assumptions. Cheers! :)

jeffa

L-LL-L--L-LL-L--L-LL-L--
-R--R-RR-R--R-RR-R--R-RR
B--B--B--B--B--B--B--B--
H---H---H---H---H---H---
(the triplet paradiddle with high-hat)


Comment on Re^8: randomising file order returned by File::Find
Select or Download Code
Re^9: randomising file order returned by File::Find
by BrowserUk (Pope) on Mar 01, 2011 at 23:36 UTC
    That is simply not true. Look into Hadoop streaming.

    You cannot process images by reading them line by line from STDIN.

    instead of making assumptions.

    Who's assuming here?


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      "Who's assuming here?"

      Surely we both are, but you seem to act like even though you have not studied or tried Hadoop, you know everything about it. You assumed that just because Hadoop is Java-based, you cannot use other languages within it. You also seem to assume that you know better than Hadoop, and, if true, that is just being vain. Cheers! :)

      Update: I guess my mistake was assuming you actually cared about new technology that Perl can be a part of.

      jeffa

      L-LL-L--L-LL-L--L-LL-L--
      -R--R-RR-R--R-RR-R--R-RR
      B--B--B--B--B--B--B--B--
      H---H---H---H---H---H---
      (the triplet paradiddle with high-hat)
      
        even though you have not studied or tried Hadoop

        I never said that. I said: "not so much".

        You assumed that just because Hadoop is Java-based, you cannot use other languages within it.

        I never said that, and certainly did not assume it. The monoculture--as I clearly said--is the filesystem, not the language it is written in.

        What I do know--from experience--is that problems of scaling the largely IO-bound processing of huge volumes of small text records, is entirely different to that of scaling the largely CPU-bound processing of relatively small numbers of large to huge binary datasets. Ie. The OPs 3D brain images.

        Your interpretation of my posts reflects your own assumptions more than mine.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://890864]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (5)
As of 2014-04-20 17:03 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (485 votes), past polls