So after a few days of intermittent network connectivity(data is on a networked drive) and testing I figured it out. I think.
push(@array, $data);
I went down the path of using only 1 thread for processing. I went down the path of parsing only a single file type. Once I started the behavior went away, and of course it wasn't until the final file type that the behavior came back. I looked through my code to see what it had that no other file type had and it was...
@array = sort { $a <=> $b } @array;
So I took out that code and tested again, the problem was still present. So I kind of commented things out piece-meal until I narrowed it down to the single line of code above. With that single line of code 3 additional MB of memory is used up as the thread leaves the parsing method for that particular file type.
So here is the basic rundown of this file.
sub parsefiletypeX
{
my $filename = shift;
#get the directory from the filename(w/ directory)
open(IN, $filename) or die...
open(OUT, $outfile) or die...
my $lineCount = 1;
my $nextline = <IN>;
my $headerlines;
my $samplesize = 1;
my @array;
$nextline = trim_whitespace($nextline);#my subroutine
++$LineCount; #Did this before I learned about $.
#read the header for the file(five lines)
#Do a bunch of regex checks on the header lines
#Read in the first record, which contains its own line of sub he
+ader data as well as the two lines of actual data in pascal float for
+mat I believe. Or fortran actually.
#Regex on the first line and push a certain piece of data. This
+line is NOT the bad line
push( @array, @fi[4]);#1st line has 5 values
#Regex checks on the next two lines
#Then read in the rest of the 3-lined records
until(eof(IN))
{
#get the same three lines and do the regex checks
#now the faulty call is made
push( @array, @fi[4]);
}
//Do stuff to the @array, like sorting and determining certain c
+hecks.
close IN;
close OUT;
#call function that uploads a record to the database.
, So after a few days of intermittent network connectivity(data is on a networked drive) and testing I figured it out. I think.
push(@array, $data);
I went down the path of using only 1 thread for processing. I went down the path of parsing only a single file type. Once I started the behavior went away, and of course it wasn't until the final file type that the behavior came back. I looked through my code to see what it had that no other file type had and it was...
@array = sort { $a <=> $b } @array;
So I took out that code and tested again, the problem was still present. So I kind of commented things out piece-meal until I narrowed it down to the single line of code above. With that single line of code 3 additional MB of memory is used up as the thread leaves the parsing method for that particular file type.
So here is the basic rundown of this file.
sub parsefiletypeX
{
my $filename = shift;
#get the directory from the filename(w/ directory)
open(IN, $filename) or die...
open(OUT, $outfile) or die...
my $lineCount = 1;
my $nextline = <IN>;
my $headerlines;
my $samplesize = 1;
my @array;
$nextline = trim_whitespace($nextline);#my subroutine
++$LineCount; #Did this before I learned about $.
#read the header for the file(five lines)
#Do a bunch of regex checks on the header lines
#Read in the first record, which contains its own line of sub he
+ader data as well as the two lines of actual data in pascal float for
+mat I believe. Or fortran actually.
#Regex on the first line and push a certain piece of data. This
+line is NOT the bad line
push( @array, @fi[4]);#1st line has 5 values
#Regex checks on the next two lines
#Then read in the rest of the 3-lined records
until(eof(IN))
{
#get the same three lines and do the regex checks
#now the faulty call is made
push( @array, @fi[4]);
}
//Do stuff to the @array, like sorting and determining certain c
+hecks.
close IN;
close OUT;
#call function that uploads a record to the database.
I even tried clearing out that array after I was done using it, such as...
@array = ();
undef @array;
But this has no effect!?!? So what is going on? | 