Re^5: Comparing and getting information from two large files and appending it in a new file

[graff]

What would be a good way to learn such tricks as indexing e.t.c.

in the context of setting up a table on a reasonably mature relational database engine (mysql, postgres, etc), creating an index on a given field is simply a matter of telling the database engine that you want that field to be indexed. The engine handles the rest for you -- that's one of the things that makes database engines so attractive. For example:

CREATE TABLE genome (
  id int AUTO_INCREMENT PRIMARY KEY,
  start int,
  end int,
  strand varchar(6),
  ...  -- include other fields as needed
  INDEX start,
  INDEX end
  ...  -- index other fields that are often useful in queries
)

By declaring that those two integer fields are to be indexed, mysql will take care of all the back-end work to make sure that queries like the following would execute with a minimum amount of time spent searching and comparing:

-- assume your current input data record has a "position" value of 876
+5:

select * from genome where start <= 8765 and end >= 8765

As for your benchmark results that you reported, do you consider those to be good or bad? How much longer is that relative to the time it takes just to read the larger input file and do little else (e.g. how long does it take to run a simple one-liner, like:  perl -ne '$sz+=length(); END {print $sz,"\n"}' )?