Re: OUT OF MEMORY!
by dragonchild (Archbishop) on Aug 23, 2001 at 01:02 UTC
|
Dude! Further optimize! Let's see ...
elsif (/\*{4}stuff\*{4}(.*?)\*{4}endstuff\*{4}/sg;
) {
push(@stuff, $_);
}
}
That's one.
A style issue here:
for (my $i=0; $i < (@filenames); $i++) {
should be
for my $i (0 ... $#filenames) {
Now. If you're running out of memory, have you tried running it with a smaller dataset and figuring out where the memory threshold is? I can tell you it's not in loading your script into RAM ... it's just not big enough. It's when you're building these datastructures. Try just one filename's worth. If that breaks stuff, start making it smaller. Don't bite off more than you can chew! If it's not working, simplify until it does, then build it up again.
------ /me wants to be the brightest bulb in the chandelier!
Vote paco for President! | [reply] [Watch: Dir/Any] [d/l] [select] |
Re: OUT OF MEMORY!
by Monky Python (Scribe) on Aug 23, 2001 at 01:08 UTC
|
hmmmm.... I'm getting some errors like:
Global symbol "@temlate" requires explicit package name at test.pl line 67.
Global symbol "@title" requires explicit package name at test.pl line 67.
Global symbol "@heading" requires explicit package name at test.pl line 67.
after fixing compilation seems to work????
MP | [reply] [Watch: Dir/Any] |
Re: OUT OF MEMORY!
by reyjrar (Hermit) on Aug 23, 2001 at 01:52 UTC
|
I've experienced out of memory problems when code I've written
has gotten stuck in loops inifinitely or when I accidentally
pushed the contents of the same array onto itself multiple times.
A few issues I have with this code. You can't ultimately guarantee the size of
any of these files, so its best to not read them all in at the same time, you can process the file chunk by chunk
using the scalar <> or the read() or even sysopen,sysread if you want to get complicated.
instead of:
open(FILE, "<file.txt");
my @thing = <FILE>;
close FILE;
for(@thing) {
if(/this/) {
push @other,$_;
}
elsif(/that/) {
push @another,$_;
}
else {
push @crap,$_;
}
}
try:
open(FILE, "<file.txt");
while(local $_ = <FILE>) {
if(/this/) {
push @other,$_;
}
elsif(/that/) {
push @another,$_;
}
else {
push @stuff,$_;
}
}
close FILE;
Another thing that bothers me about this code is that you're not cleaning everything up
and you have multiple instances of those files in memory at any given time. This could be adding up..
in the @filenames lets say it takes up 128kb, then you join it together on $filename without
destoroying the array, you've used 256kb. if you want to operate on the scalar, then you might want to
undef the array @filenames=(); to save some memory. Also, if you're just gonna join things anyways, why not use
the str concatenation operator like so:
open(FILE,"<file.txt");
my $template = '';
while(my $line = <FILE>) {
$template .= $line;
}
close FILE;
which I believe can be shortened even more into:
my $template = '';
open(FILE, "<file.txt") && while($template .= <FILE>) {};
close FILE;
die "bad stuff happened" unless length $template;
anyways, play with it somemore. and use offline mode to do some debugging and
but 'print' statements all over your loops, Your data set may contain something you haven't thought of and it sounds to me
like the program is looping infinitely because of it.
anyways, good luck with it, and let us know how it turns out.
-brad.. | [reply] [Watch: Dir/Any] [d/l] [select] |
|
my $template;
{
local $/; # undef $/ => file slurp mode
open (FILE, '<', 'file.txt') or die "Couldn't open file.txt: $!";
$template = <FILE>;
close FILE;
}
-- Hofmator
| [reply] [Watch: Dir/Any] [d/l] |
Re: OUT OF MEMORY!
by runrig (Abbot) on Aug 23, 2001 at 01:04 UTC
|
What perl version? I get this on 5.6.1 and 5.7.2: Nested quantifiers in regex; marked by <-- HERE in m/\*{4}title\*{4}(.
+*?)\*{4}he
adings\*{4}+ <-- HERE / at tst7 line 60.
It seems like a bug to me, because it looks valid, I think. (Someone will surely correct me if I'm wrong).
Update: Ignore the above. Bad copy n paste job. Though I DO see many use strict related errors now. | [reply] [Watch: Dir/Any] [d/l] |
Re: OUT OF MEMORY!
by damian1301 (Curate) on Aug 23, 2001 at 01:41 UTC
|
Try replacing the foreach loop with a while loop and just loop through the file. Something like this:
open(PAGES, "<pages.txt") || die $!;
my @filenames;
my @titles;
my @headings;
my @xstuff;
while (<PAGES>){
if (m/^\*{4}filename.ext\*{4}/) {
s/^\*{4}filename.ext\*{4}//;
chomp $_;
push(@filenames, $_);
}elsif (m/^\*{4}title\*{4}/) {
s/^\*{4}title\*{4}//;
chomp $_;
push(@titles, $_);
}elsif (m/^\*{4}heading\*{4}/) {
s/^\*{4}heading\*{4}//;
chomp $_;
push(@headings, $_);
}else {
push(@xstuff, $_);
}
}
close PAGES;
$_.=($=+(6<<1));print(chr(my$a=$_));$^H=$_+$_;$_=$^H;
print chr($_-39); # Easy but its ok.
| [reply] [Watch: Dir/Any] [d/l] [select] |
Re: OUT OF MEMORY!
by Cine (Friar) on Aug 23, 2001 at 01:10 UTC
|
This can be of three reasons:
- Your pages.txt is HUGE
- Your template.html is HUGE
- You have VERY little memory ;)
T
I
M
T
O
W
T
D
I | [reply] [Watch: Dir/Any] |
|
have a small template.html and small pages.txt and should have plenty of memory,
it must be something wrong with the code
| [reply] [Watch: Dir/Any] |
|
I was trying to be funny ;)
Since I can see no wrong in the script.
T
I
M
T
O
W
T
D
I
| [reply] [Watch: Dir/Any] |
I got it
by maddfisherman (Sexton) on Aug 23, 2001 at 04:56 UTC
|
I downloaded a newer version of perl and it gave me a couple clear error messages. Thanks alot i should be able to handle it from here.
| [reply] [Watch: Dir/Any] |
perl version
by maddfisherman (Sexton) on Aug 23, 2001 at 01:10 UTC
|
im using perl 5.00502, should i get a new version
| [reply] [Watch: Dir/Any] |
|
Yes, you should upgrade either to 5.005_03 or 5.6.1. Which one depends on how many local dependencies you have and how much energy for testing. (If you have to bite the bullet, 5.6.1 would be better, but going to it is also inherently more risky.)
I have long ago blissfully forgotten the details, but 5.005_02 had at least one bug which lead to infinite loops. An unexpected infinite loop in which you are tying up memory would show up as an out of memory issue.
So I cannot guarantee you that an upgrade would solve your problem, but it sounds to me like a pretty good bet.
| [reply] [Watch: Dir/Any] |