Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??

I have a feeling I am copying the data into memory several times, but I do not know how to avoid this.

Rewrite your program into subroutines, use lots of references, and watch the size of your program at each stage

There is a lot of copy/paste duplication in your program that should be rewritten

There are a lot of short variable names, way too many to keep track of, way too many to help you substantially rewrite what you posted

Also you say The total amount of rows is close to 1 million. and that is close to the MAXIMUM LIMIT for Excel of 1,048,576 rows by 16,384 columns

You might switch to DBD::SQLite, see Loading 283600 records (WordNet), Re: Is there a way(maybe tool?) to know the amount of memory an array takes?, dynamic bulk insert in sqlite


In reply to Re: Out of memory and While replacements with excel XLSX application by Anonymous Monk
in thread Out of memory and While replacements with excel XLSX application by Weinstar

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others avoiding work at the Monastery: (6)
    As of 2014-08-02 01:33 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      Who would be the most fun to work for?















      Results (53 votes), past polls