wrkrbeee has asked for the wisdom of the Perl Monks concerning the following question:
Hi PERL Monks, the code below generates the "out of memory" error. I suspect that the code is reading the entire file before writing/saving the results to disk instead of reading the file one line at a time. I am grateful for any insight you may have. Thank you! EDIT: I replaced the "for each" loop with the "while" loop (as suggested by choroba), but continue to receive the out of memory error. FYI, the crash occurs when downloading a 450MB text file, if that helps. Machine has 6gb of memory installed. I apologize for my ignorance, what I know about PERL fits easily in a doll house thimble.
#!/usr/bin/perl use LWP; use HTTP::Request; sub get_http { my $url = shift; my $request = HTTP::Request->new(GET => $url); my $response = $ua->request($request); if (!$response->is_success) { print STDERR "GET '%s' failed: %s\n", $url, $response->status_line; return undef; } return $response->content; } # user agent object for handling HTTP requests my $ua = LWP::UserAgent->new; # if you only want a portion of the filing, un-comment the next line #$ua->max_size(50000); # 50k byte limit ######################### write dir , use "\\" and not "\", for exampl +e: "C:\\temp" $write_dir = "C:\\Volumes\\EDGAR1\\Edgar\\Edgar2\\10K_10Q\\2014"; ######################### write dir ######################### filename with urls (put in same directory as + script) open dlthis, "Data2014.txt" or die $!; ######################### filename with urls (put in same directory as + script) ######################### log open LOG , ">download_log.txt" or die $!; ######################### log my @file = <dlthis>; foreach $line (@file) { #CIK, filename, blank is not used (included because it will capture t +he newline) ($CIK, $get_file, $blank) = split (",", $line); $get_file = "http://www.sec.gov/Archives/" . $get_file; $_ = $get_file; if ( /([0-9|-]+).txt/ ) { $filename = $write_dir . "/" . $CIK . ".txt"; open OUT, ">$filename" or die $!; print "file $CIK \n"; my $request = HTTP::Request->new(GET => $get_file); my $response =$ua->get($get_file ); $p = $response->content; if ($p) { print OUT $p; close OUT; } else { #error logging print LOG "error in $filename - $CIK \n" ; } } } close LOG;
|
---|
Replies are listed 'Best First'. | |
---|---|
Re: Out of memory
by choroba (Cardinal) on Jan 15, 2015 at 15:08 UTC | |
by wrkrbeee (Scribe) on Jan 15, 2015 at 15:13 UTC | |
by choroba (Cardinal) on Jan 15, 2015 at 15:22 UTC | |
by wrkrbeee (Scribe) on Jan 15, 2015 at 15:51 UTC | |
by wrkrbeee (Scribe) on Jan 16, 2015 at 17:32 UTC | |
by LanX (Saint) on Jan 15, 2015 at 15:22 UTC | |
Re: Out of memory
by Athanasius (Archbishop) on Jan 15, 2015 at 15:14 UTC | |
by Anonymous Monk on Jan 15, 2015 at 22:31 UTC | |
by wrkrbeee (Scribe) on Jan 15, 2015 at 15:21 UTC | |
by RonW (Parson) on Jan 15, 2015 at 23:58 UTC | |
Re: Out of memory
by fishmonger (Chaplain) on Jan 15, 2015 at 15:23 UTC | |
by wrkrbeee (Scribe) on Jan 15, 2015 at 15:28 UTC | |
by Corion (Patriarch) on Jan 15, 2015 at 15:44 UTC | |
by wrkrbeee (Scribe) on Jan 15, 2015 at 15:48 UTC |
Back to
Seekers of Perl Wisdom