Re^2: Reading (the same) data in different ways & memory usage

[Neighbour]

I implemented your idea with a slight variation (moved the decimal-correction in the numeric data branch and put the check for empty fields in the non-numeric branch:

        if ($self->datatype->[$_] =~ /^(?:TINYINT|MEDIUMINT|SMALLINT|I
+NT|INTEGER|BIGINT|FLOAT|DOUBLE)$/) {
            $hr_returnvalue->{$CurrentColumnName} = 0 + substr ($texti
+nput, $self->flatfield_start->[$_], $self->flatfield_length->[$_]);# 
+create a numeric value.
            # Decimal-correction
            if ($self->decimals->[$_] > 0 and defined $hr_returnvalue-
+>{$CurrentColumnName}) {
                $hr_returnvalue->{$CurrentColumnName} /= 10**$self->de
+cimals->[$_];
            }
        } else {
            $hr_returnvalue->{$CurrentColumnName} = substr ($textinput
+, $self->flatfield_start->[$_], $self->flatfield_length->[$_]);
            $hr_returnvalue->{$CurrentColumnName} =~ s/^\s*(.*?)\s*$/$
+1/;    # Trim whitespace
            # Fill empty fields with that field's default value, if su
+ch a value is defined
            if ($hr_returnvalue->{$CurrentColumnName} eq "") { 
                if (defined $self->standadefaultard->[$_]) {
                    if ($self->datatype->[$_] =~ /^(?:CHAR|VARCHAR|DAT
+E|TIME|DATETIME)$/) {
                        $hr_returnvalue->{$CurrentColumnName} = sprint
+f ("%s", $self->default->[$_]);
                    } else {
                        $hr_returnvalue->{$CurrentColumnName} = $self-
+>default->[$_];
                    }
                } else {
                    # Remove empty field
                    delete $hr_returnvalue->{$CurrentColumnName};
                }
            } 
        }

but the idea is sound. Devel::Size now reports the returned data-structure to be 385251506 bytes, which, for some reason is smaller than the data-structure retrieved from the db...I'll have to look at things more closely to figure out why that is.

[BrowserUk]

the returned data-structure to be 385251506 bytes, which, for some reason is smaller than the data-structure retrieved from the db

Perhaps the DBI code doesn't trim leading/trailing spaces on string fields?

[Tux]

If the target fields in the database are of type CHAR (not VARCHAR), you can make DBI do so by:

$dbh->{ChopBlanks} = 1;

Some DBD's do extend this behavior to VARCHAR fields (when the database it too stupid to do so itself, as the ANSI standard tells it to) or when VARCHAR effectively is a CHAR internally (because the database doesn't support VARCHAR).

YMMV

[BrowserUk]

(when the database it too stupid to do so itself, as the ANSI standard tells it to)

Erm. If I store the string 'fred          ' into a DB, I'd expect to get that same string back. Does ANSI mandate otherwise?

[Neighbour]

Well, one thing I found is that all FLOAT (and probably DOUBLE) values are returned as string (for example "0.00" for a column defined as FLOAT(7,2)).
Using DBD::MySQL 4.0.18 (should that matter) with default connection options.

[BrowserUk]

one thing I found is that all FLOAT (and probably DOUBLE) values are returned as string

That does make a certain amount of sense.

If you converted floats to their binary numeric representation, then you could introduce typical floating point representation errors. If you are going to do math with them in your perl code, then that will happen anyway and you'd have to deal with it.

But if you are only going to display them, or store them into another table somewhere, then returning them as strings means that you don't have to deal with those errors.

Whether that is the reasoning behind it I cannot say, but it could be.