Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Re: Out of Memory when generating large matrix

by Anonymous Monk
on Mar 06, 2018 at 03:07 UTC ( #1210386=note: print w/replies, xml ) Need Help??


in reply to Out of Memory when generating large matrix

Shamelessly hiding-out behind Anonymous Monk, now, just like everybody else does, you-know-who will simply take this opportunity to remind the entire lot of you that a technical requirement such as this one actually requires no(!) custom programming whatsoever. Any sort of statistics software ... or perhaps even a stats-plugin for whatever might be your favorite spreadsheet ... will scarcely blink at this requirement or this volume of data.
  • Comment on Re: Out of Memory when generating large matrix

Replies are listed 'Best First'.
Re^2: Out of Memory when generating large matrix
by Your Mother (Bishop) on Mar 06, 2018 at 03:11 UTC

    That's exciting. Since it's so easy, please whip up a working example like the rest of us regularly do.

    –AnonymousMonkey

      I'm not the Anon above, or you-know-who, but I'll gladly oblige!

      Tally up the counts of tokens that appear on field #2.

      $ cat datafiles/* | sort -k2 | uniq -c -f1

      Can we please see your code now? Or maybe yet another diatribe on how the approach is simply wrong, does not scale, requires expensive disk i/o, is inelegant, hazardous, broken, politically incorrect or in some other way unsuitable.

        Well, I was doing a bit since SunnyD cannot answer any question where a technical answer is required and the spec as stated is risible. It demands no "custom programming." Your command pipeline is customized for the problem. It violates the solution spec.

        As to "see your code now," was that for me or the anonymous monk formerly known as sundialsvc4? I made no assertions at all about any of the code or that I was one with an answer here. I merely inquired as to the ingredients of the snake oil being advertised. This problem domain is quite a ways from anything I'm good at. :P

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1210386]
help
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (5)
As of 2018-06-18 23:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?



    Results (111 votes). Check out past polls.

    Notices?