Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Out of memory and While replacements with excel XLSX application

by Anonymous Monk
on Oct 08, 2012 at 08:00 UTC ( #997762=note: print w/replies, xml ) Need Help??


in reply to Out of memory and While replacements with excel XLSX application

I have a feeling I am copying the data into memory several times, but I do not know how to avoid this.

Rewrite your program into subroutines, use lots of references, and watch the size of your program at each stage

There is a lot of copy/paste duplication in your program that should be rewritten

There are a lot of short variable names, way too many to keep track of, way too many to help you substantially rewrite what you posted

Also you say The total amount of rows is close to 1 million. and that is close to the MAXIMUM LIMIT for Excel of 1,048,576 rows by 16,384 columns

You might switch to DBD::SQLite, see Loading 283600 records (WordNet), Re: Is there a way(maybe tool?) to know the amount of memory an array takes?, dynamic bulk insert in sqlite

  • Comment on Re: Out of memory and While replacements with excel XLSX application

Replies are listed 'Best First'.
Re^2: Out of memory and While replacements with excel XLSX application
by Anonymous Monk on Oct 08, 2012 at 09:29 UTC
      ValueTargetCols should loop over rows first then cols, but it probably makes no difference in the results in this case

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://997762]
help
Chatterbox?
[choroba]: Dangers of WFH: a mailman rings, brings a parcel for my wife, I need to go outside, take my son with me, sign a paper *and* into the postman's mobile app, get back. Guess what the soup was doing meanwhiles?
[Discipulus]: soup without heat protections as CPUs are doomed to burned..
[1nickt]: The dangers of milk-based soup, you mean?
[Corion]: choroba: Ouch! But you don't need that many external events for that. I have on several occasions set up milk to cook and then programmed a bit, only to find the milk burned and congealed over the stove :-/
[choroba]: it's still edible, just need to scour the pot
[1nickt]: Corion I have a large site I need to check for broken links and absolute links. Making a scraper is easy of course; a spider that crawls a whole site is a little more involved ... I was planning a queue-based tool. Intersted to see what you do...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (12)
As of 2017-10-18 11:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My fridge is mostly full of:

















    Results (244 votes). Check out past polls.

    Notices?