Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Re: Thinking out loud (was: Re^2: The Eternal "filter.pl")

by RichardK (Priest)
on Aug 30, 2011 at 11:18 UTC ( #923181=note: print w/ replies, xml ) Need Help??


in reply to Thinking out loud (was: Re^2: The Eternal "filter.pl")
in thread The Eternal "filter.pl"

Hi. I'm glad to see it's not just me that's been thinking along these lines :)

Not that I've come to any real conclusions, but I think that it's all just set theory. And the way to describe it may be those terms. i.e. sets, unions, intersections and complements.

If all the records fit in memory in a hash it's reasonably easy to describe a set that passes some test function

my @set_1 = grep { func($_) } keys %records_1;

then you can describe the relationship you're looking for in those terms.

So you might end up with something like this:-

set1 = set of all records_1 that pass func() set2 = set of all records_2 that fail func() results = intersection of set1 and set2; set3 = ... etc
and so you can describe any arbitrary combination of sets.

We will needs some support functions , but that's 'just a simple matter of programming' ;)

The main problem, as I see it , is how to deal with data sets that are too big to fit into memory. My only thought is to keep a hash of the (key, file offset) and re-parse each record on demand. Or maybe Tie::File could do the job ?

Arrgh! - you're making me want to try code this again :)

R.


Comment on Re: Thinking out loud (was: Re^2: The Eternal "filter.pl")
Select or Download Code
Re^2: Thinking out loud (was: Re^2: The Eternal "filter.pl")
by Voronich (Hermit) on Aug 30, 2011 at 15:52 UTC

    Well it won't fit in memory, but that's neither here nor there really. Plus I don't need separate pre-qualifiers for source record sets, which is nice.

    Hmm... closing in on an idea. Gonna go code some tests. I'll transmorgrify this into a CuFP yet!

    Me

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://923181]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (3)
As of 2014-09-21 03:18 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (166 votes), past polls