Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Out of memory error!!!

by sathya83aa (Acolyte)
on Mar 27, 2013 at 17:03 UTC ( #1025757=perlquestion: print w/ replies, xml ) Need Help??
sathya83aa has asked for the wisdom of the Perl Monks concerning the following question:

Hi Experts,

I'm working on a perl programming in which I'm just opening the file and using substr function splitting each line and getting some 50 variables including some arrays and printing it to the XML file. While executing this code I'm getting Out of memory error in the server. Can someone help me in resolving this issue?

Below is the code snippet of how my script looks like. I'm using a file with some 30 lines in it. Also I'm opening a log file and XML file in the same program.

#!/usr/bin/perl open (INFO, "<", "$File") or die("Cannot open the flat file"); open(my $LOG,">>LOG_file") or die("Cannot open"); open(XML,">xml_file") or die("Cannot open"); foreach $line2 (<INFO>) { $line2 =~ s/&/&amp;/g; print "$line2\n"; my $segment = substr($line2,0,2); if($segment eq "00") { my $Var1 = trim(substr($line2,40,8)); } if($segment eq "A0") { $Var2 = trim(substr($line2,2,9)); $Var3 = trim(substr($line2,14,3)); $Var4 = trim(substr($line2,20,42)); $Var5 = trim(substr($line2,62,30)); and so on } close LOG_file; close xml_file; close INFO; sub trim($) { my $string = shift; $string =~ s/^\s+//; $string =~ s/\s+$//; return $string; }

Thanks & Regards,

Sathya V.

Comment on Out of memory error!!!
Download Code
Re: Out of memory error!!!
by davido (Archbishop) on Mar 27, 2013 at 18:00 UTC

    Change "foreach $line2 ( <INFO> ) {..."

    to

    "while( my $line2 = <INFO> ) {..."

    Your current implementation slurps the flat file into memory because foreach is a ranged-based loop that evaluates <INFO> in list context, which gives you slurp behavior. while() is a conditional based loop, and thus evaluates <INFO> in scalar context, which gives you its iterator behavior.


    Dave

Re: Out of memory error!!!
by Anonymous Monk on Mar 27, 2013 at 18:17 UTC

    Davido's got your memory fault, but that's pretty inconsistent code formatting, and poor error reporting on the opens. (Remember, $! is your friend, and it never hurts to print the name of the file that failed to open in your error message)

      Didnt get what you were referring to regarding the code and open statement. Can you please explain me in detail as I'm new to Perl programming.

        use autodie; try a demo
        $ perl -e " use autodie; open my($fh), qw[< noexist ]; " Can't open 'noexist' for reading: 'No such file or directory' at -e li +ne 1
        the reason it can't open comes from $!
Re: Out of memory error!!!
by McA (Curate) on Mar 27, 2013 at 22:05 UTC

    Some hints:

    #!/usr/bin/perl
    Should be followed immediately with
    use strict; use warnings;

    You mix up several ways to open a file. You should take consistently the three parameter form. Also use scalars as file handles. So your

    open (INFO, "<", "$File") or die("Cannot open the flat file"); open(my $LOG,">>LOG_file") or die("Cannot open"); open(XML,">xml_file") or die("Cannot open");
    becomes
    open(my $INFO, '<', $File) or die "Cannot open the '$File': $!; open(my $LOG, '>>' 'LOG_file') or die "Cannot open 'LOG_file: $!"; open(my $XML, '>', 'xml_file') or die "Cannot open 'xml_file': $!";

    As stated by davido, you should iterate line by line over the file:

    while(defined(my $line2 = <$INFO>)) { }

    Every $line2 will have a LF at the end. Probably you want to get rid of it. In this case it's not necessary.

    chomp $line2;

    It seems you're working on fixed column format files (COBOL output?). The format of the row (line) is defined by the first two bytes (segment). So, instead of creating a big if-elsif-else-tree it would be more maintainable to create a dispatcher based on the segment. The usage of unpack may be your friend to extract the data easily. So I propose the following:

    my %config = ( '00' => { 'unpack' => "x40A8", 'fields' => [qw(Var1)], }, 'A0' => { 'unpack' => "x2A9x3A3x3A42A30", 'fields' => [qw(Var1 Var2 Var3 Var4)], }, ); while(defined(my $line = <$INFO>)) { my $segment = substr $line, 0, 2; if(exists $config{$segment}) { my @fields = unpack $config{$segment}->{'unpack'}, $line; my %record; $record{$config{$segment}->{'fields'}->[$_]} = $fields[$_] for +each (0..$#fields); print Dumper(\%record), "\n"; } else { die "ERROR: Unknown segment '$segment'. Config or data error?" +; } }

    At the end you have to close the filehandles and not the 'filenames':

    close $INFO or die "Can't close file '$File': $!"; close $LOG or die "Can't close file 'LOG_file': $!"; close $XML or die "Can't close file 'xml_file': $!";

    These proposals reflect only my personal opinion. Hopefully a little bit helpful.

    McA

      Hi McA,

      Thanks for those suggestions..Even after changing the foreach to while, I'm still facing the same Out of memory issue :-(

      Any suggestions?

      Thanks & Regards,

      sathya V.

        If you're still running out of memory, you'll have to find out where you are storing your data and maybe not store all the data in RAM. If you are storing the bulk of your data in a hash, an easy way of trading time for space is to use DB_File or some other key/value store that can be tied to a hash. Update: Also see Tie::DBI, which just had a new release out.

        Often it also helps to output progress through your file so that you get an idea where in your program you run out of memory.

        In your example you're opening a filehandle named XML, so I assume you want to convert your input stream to xml format. If this is right we would like to see how the xml output is produced. Probably you're using a memory intensive way.

        McA

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1025757]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (6)
As of 2014-08-21 02:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (127 votes), past polls