Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re^5: Comparing and getting information from two large files and appending it in a new file

by graff (Chancellor)
on Apr 01, 2012 at 00:13 UTC ( #962812=note: print w/ replies, xml ) Need Help??


in reply to Re^4: Comparing and getting information from two large files and appending it in a new file
in thread Comparing and getting information from two large files and appending it in a new file

What would be a good way to learn such tricks as indexing e.t.c.

in the context of setting up a table on a reasonably mature relational database engine (mysql, postgres, etc), creating an index on a given field is simply a matter of telling the database engine that you want that field to be indexed. The engine handles the rest for you -- that's one of the things that makes database engines so attractive. For example:

CREATE TABLE genome ( id int AUTO_INCREMENT PRIMARY KEY, start int, end int, strand varchar(6), ... -- include other fields as needed INDEX start, INDEX end ... -- index other fields that are often useful in queries )
By declaring that those two integer fields are to be indexed, mysql will take care of all the back-end work to make sure that queries like the following would execute with a minimum amount of time spent searching and comparing:
-- assume your current input data record has a "position" value of 876 +5: select * from genome where start <= 8765 and end >= 8765
As for your benchmark results that you reported, do you consider those to be good or bad? How much longer is that relative to the time it takes just to read the larger input file and do little else (e.g. how long does it take to run a simple one-liner, like:  perl -ne '$sz+=length(); END {print $sz,"\n"}' )?


Comment on Re^5: Comparing and getting information from two large files and appending it in a new file
Select or Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://962812]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (13)
As of 2014-12-17 20:01 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (31 votes), past polls