well, i don't know if you are familiar with SQLite (or MySQL), i often get stuck with processing large amount of data, at first i also tried to do it through hashes and arrays, but then i just gave up on it. my advise would be to use one of the freely available databases like SQLite or MySQL. just import data, sort it (order by), and when processing it retrieve line by line of data and do what ever you want to do with it.

this is memory efficient, but certainly slower way to do it

SQLite

example on how to use it through DBI and it would be usfull to check out DBD::SQLite

plus there are tons of concrete examples on perlmonks !!!

---

---

• Are you posting in the right place? Check out Where do I post X? to know for sure.
• Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:

  <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>

• Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
• Want more info? How to link or How to display code and escape characters are good places to start.