Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

OUT OF MEMORY!

by maddfisherman (Sexton)
on Aug 23, 2001 at 00:50 UTC ( [id://107132]=perlquestion: print w/replies, xml ) Need Help??

maddfisherman has asked for the wisdom of the Perl Monks concerning the following question:

Not sure why its doing this but when i compile the program(perl -c create2.pl) it runs out of mem.heres the script:
#!/usr/bin/perl -w #use CGI::Carp('fatalsToBrowser'); #use Carp; #use diagnostics; use strict; #use warnings; open(PAGES, "<pages.txt") || die $!; my @pages; @pages = <PAGES>; close(PAGES); my @filenames; my @titles; my @headings; my @xstuff; foreach (@pages) { if (m/^\*{4}filename.ext\*{4}/) { s/^\*{4}filename.ext\*{4}//; chomp $_; push(@filenames, $_); } elsif (m/^\*{4}title\*{4}/) { s/^\*{4}title\*{4}//; chomp $_; push(@titles, $_); } elsif (m/^\*{4}heading\*{4}/) { s/^\*{4}heading\*{4}//; chomp $_; push(@headings, $_); } else { push(@xstuff, $_); } } my $xstuff=""; $xstuff = join("", @xstuff); my @stuff; @stuff = $xstuff =~ /\*{4}stuff\*{4}(.*?)\*{4}endstuff\*{4}/sg; open(TEMPLATE, "<template.html") || die $!; my @xtemplate; @xtemplate = <TEMPLATE>; close (TEMPLATE); my $xtemplate=""; $xtemplate = join("", @xtemplate); my @template; @template = $xtemplate =~ /^(.*?)\*{4}filename.ext\*{4}/sg; push (@template, $xtemplate =~ /\*{4}filename.ext\*{4}(.*?)\*{4}title\ +*{4}/sg); push (@template, $xtemplate =~ /\*{4}title\*{4}(.*?)\*{4}headings\*{4} +/sg); push (@template, $xtemplate =~ /\*{4}headings\*{4}(.*?)/sg); for (my $i=0; $i < (@filenames); $i++) { open(FILE, ">@filenames[$i]") || die $!; print FILE qq($temlate[0] $title[$i] $template[1] $heading[$i] $te +mplate[2] $stuff[$i] $template[3]); close(FILE); } print "Content-type:text/html\n\n"; print <<EndHTML; <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transisional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd"> <html> <head> <title>create</title> </head> <body> EndHTML foreach (@filenames) { print "<p>Name:$_</p>"; } print "<hr />"; foreach (@titles) { print "<p>Titles:$_</p>"; } print "<hr />"; foreach (@headings) { print "<p>Headings:$_</p>"; } print "<hr />"; foreach (@stuff) { print "<p>stuff:$_</p>"; } print <<EndHTML; </body> </html> EndHTML

Replies are listed 'Best First'.
Re: OUT OF MEMORY!
by dragonchild (Archbishop) on Aug 23, 2001 at 01:02 UTC
    Dude! Further optimize! Let's see ...
    elsif (/\*{4}stuff\*{4}(.*?)\*{4}endstuff\*{4}/sg; ) { push(@stuff, $_); } }
    That's one.

    A style issue here:

    for (my $i=0; $i < (@filenames); $i++) { should be for my $i (0 ... $#filenames) {
    Now. If you're running out of memory, have you tried running it with a smaller dataset and figuring out where the memory threshold is? I can tell you it's not in loading your script into RAM ... it's just not big enough. It's when you're building these datastructures. Try just one filename's worth. If that breaks stuff, start making it smaller. Don't bite off more than you can chew! If it's not working, simplify until it does, then build it up again.

    ------
    /me wants to be the brightest bulb in the chandelier!

    Vote paco for President!

Re: OUT OF MEMORY!
by Monky Python (Scribe) on Aug 23, 2001 at 01:08 UTC
    hmmmm.... I'm getting some errors like:

    Global symbol "@temlate" requires explicit package name at test.pl line 67.
    Global symbol "@title" requires explicit package name at test.pl line 67.
    Global symbol "@heading" requires explicit package name at test.pl line 67.

    after fixing compilation seems to work????

    MP

Re: OUT OF MEMORY!
by reyjrar (Hermit) on Aug 23, 2001 at 01:52 UTC
    I've experienced out of memory problems when code I've written has gotten stuck in loops inifinitely or when I accidentally pushed the contents of the same array onto itself multiple times.

    A few issues I have with this code. You can't ultimately guarantee the size of any of these files, so its best to not read them all in at the same time, you can process the file chunk by chunk using the scalar <> or the read() or even sysopen,sysread if you want to get complicated. instead of:
    open(FILE, "<file.txt"); my @thing = <FILE>; close FILE; for(@thing) { if(/this/) { push @other,$_; } elsif(/that/) { push @another,$_; } else { push @crap,$_; } }

    try:
    open(FILE, "<file.txt"); while(local $_ = <FILE>) { if(/this/) { push @other,$_; } elsif(/that/) { push @another,$_; } else { push @stuff,$_; } } close FILE;

    Another thing that bothers me about this code is that you're not cleaning everything up and you have multiple instances of those files in memory at any given time. This could be adding up.. in the @filenames lets say it takes up 128kb, then you join it together on $filename without destoroying the array, you've used 256kb. if you want to operate on the scalar, then you might want to undef the array @filenames=(); to save some memory. Also, if you're just gonna join things anyways, why not use the str concatenation operator like so:
    open(FILE,"<file.txt"); my $template = ''; while(my $line = <FILE>) { $template .= $line; } close FILE;

    which I believe can be shortened even more into:
    my $template = ''; open(FILE, "<file.txt") && while($template .= <FILE>) {}; close FILE; die "bad stuff happened" unless length $template;

    anyways, play with it somemore. and use offline mode to do some debugging and but 'print' statements all over your loops, Your data set may contain something you haven't thought of and it sounds to me like the program is looping infinitely because of it.

    anyways, good luck with it, and let us know how it turns out.


    -brad..

      Just a small comment on reading in a whole file at once. It's not efficient to read in linewise just to glue them together afterwards. Instead do it like this:

      my $template; { local $/; # undef $/ => file slurp mode open (FILE, '<', 'file.txt') or die "Couldn't open file.txt: $!"; $template = <FILE>; close FILE; }

      -- Hofmator

Re: OUT OF MEMORY!
by runrig (Abbot) on Aug 23, 2001 at 01:04 UTC
    What perl version? I get this on 5.6.1 and 5.7.2:
    Nested quantifiers in regex; marked by <-- HERE in m/\*{4}title\*{4}(. +*?)\*{4}he adings\*{4}+ <-- HERE / at tst7 line 60.

    It seems like a bug to me, because it looks valid, I think. (Someone will surely correct me if I'm wrong).

    Update: Ignore the above. Bad copy n paste job. Though I DO see many use strict related errors now.

Re: OUT OF MEMORY!
by damian1301 (Curate) on Aug 23, 2001 at 01:41 UTC
    Try replacing the foreach loop with a while loop and just loop through the file. Something like this:

    open(PAGES, "<pages.txt") || die $!; my @filenames; my @titles; my @headings; my @xstuff; while (<PAGES>){ if (m/^\*{4}filename.ext\*{4}/) { s/^\*{4}filename.ext\*{4}//; chomp $_; push(@filenames, $_); }elsif (m/^\*{4}title\*{4}/) { s/^\*{4}title\*{4}//; chomp $_; push(@titles, $_); }elsif (m/^\*{4}heading\*{4}/) { s/^\*{4}heading\*{4}//; chomp $_; push(@headings, $_); }else { push(@xstuff, $_); } } close PAGES;


    $_.=($=+(6<<1));print(chr(my$a=$_));$^H=$_+$_;$_=$^H; print chr($_-39); # Easy but its ok.
Re: OUT OF MEMORY!
by Cine (Friar) on Aug 23, 2001 at 01:10 UTC
    This can be of three reasons:
    • Your pages.txt is HUGE
    • Your template.html is HUGE
    • You have VERY little memory ;)


    T I M T O W T D I
      have a small template.html and small pages.txt and should have plenty of memory, it must be something wrong with the code
        I was trying to be funny ;)
        Since I can see no wrong in the script.

        T I M T O W T D I
I got it
by maddfisherman (Sexton) on Aug 23, 2001 at 04:56 UTC
    I downloaded a newer version of perl and it gave me a couple clear error messages. Thanks alot i should be able to handle it from here.
perl version
by maddfisherman (Sexton) on Aug 23, 2001 at 01:10 UTC
    im using perl 5.00502, should i get a new version
      Yes, you should upgrade either to 5.005_03 or 5.6.1. Which one depends on how many local dependencies you have and how much energy for testing. (If you have to bite the bullet, 5.6.1 would be better, but going to it is also inherently more risky.)

      I have long ago blissfully forgotten the details, but 5.005_02 had at least one bug which lead to infinite loops. An unexpected infinite loop in which you are tying up memory would show up as an out of memory issue.

      So I cannot guarantee you that an upgrade would solve your problem, but it sounds to me like a pretty good bet.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://107132]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others perusing the Monastery: (9)
As of 2024-03-28 18:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found