Re: Best way to read a file into memory and use normal operation on memory file?

[james28909]

Let me give yall a better explanation of what i am trying to do, i already have the byte reversing done so no need to read a file backwards, but i can open both files before the byte reversal like this:

use File::Slurp;

open my $in_file, '<', "file";
binmode($file);

open ( my $out_file, '>', "reversed" ); 
binmode($out_file);

my ( $buf, $data, $n, $bytes ); 

while (( $n = read $in_file, $data, 4096 ) != 0 ) {
    syswrite $out_file, pack( "v*", unpack("n*", $data ));   
    $bytes+=$n;
}

this above code will successfully read a file then byte reverse it to the other file, but you have to have a file on disk to do it.


now, what i want to be able to do, is open the $in_file, then byte reverse it in memory, and then write it to file after reversing it, or in other words, byte reverse the file in memory, then do whatever i want with it.

use File::Slurp;


open ( my $in_file, '<', "file" );
binmode( $file );

my $temp;
open  my $mem_file, '>', \$temp;
binmode( $mem_file );

my ( $buf, $data, $n, $bytes ); 

while (( $n = read $in_file, $data, 4096 ) != 0 ) {
    syswrite $mem_file, pack( "v*", unpack("n*", $data ));   
    $bytes+=$n;
}

open ( my $out_file, '>', "reversed" ); 
binmode( $out_file );

syswrite ( $out_file, $mem_file );

i have tried a few different things and nothing has worked tho i am able to get the program to actually write the file, but it writes SCALAR(XXXXXXXX) or GLOB(XXXXXXXX) ect to the file and not the actual reversed data.

[AppleFritter]

A few comments. First of all:

this above code will successfully read a file then byte reverse it to the other file, but you have to have a file on disk to do it.

Unfortunately, it won't. The code you posted will take a file, split it into chunks of 4096 byte, attempt to reverse each chunk, and then write the chunks to a new file in the same order. Worse, it fails to even reverse each chunk. Try generating a test file with several chunks, e.g. as follows:

$ for i in `seq 1 256`; do echo -n '0123456789ABCDEF' >>file; done
$ for i in `seq 1 256`; do echo "This won't work" >>file; done
$

Try running your script on the resulting file; you'll end up with something like the following:

1032547698BADCFE1032547698BADCFE1032547698BADCFE...
khTsiw not'w ro
khTsiw not'w ro
khTsiw not'w ro
...

Which is not "reversed" in even the loosest sense of the word.

Now, on to your second question. This

syswrite ( $out_file, $mem_file );

simply isn't going to work, as you've found out. syswrite expects to be passed data (in the form of a scalar), but what you're passing is a reference to a filehandle, which stringifies to something along the lines GLOB(0x8006c228).

Furthermore, I'm a little puzzled as to why you want to use an in-memory file to begin with. Your first script -- if you were to fix it -- would be useful since you could use it to reverse files too large to slurp into memory all at once; with the second one, that's obviously not true anymore.

You don't need to use an in-memory file: there's a much better, easier-to-use facility for holding data in memory, namely variables. So why not do what was suggested above, using e.g. File::Slurp to read all the file's contents into a variable, and then use reverse to reverse it before writing it out again?

Frankly, I'm puzzled, but perhaps I'm missing the obvious good reason for why you're doing it this way. If that is the case, please enlighten me; otherwise, please meditate on the advice the monks gave you.

[james28909]

the pack unpack function (on my system atleast) will take a file in 12 34 56 78 order and put it as 34 12 78 56 which is exactly what i need and works flawless everytime. ive even packed the function into an exe and added it as a right click context menu option because it did work so good :p

maybe it has to do with your machine maybe? im on windows pc ofcourse.

also i should add that the file i will always be working with is 16 MB's (6,777,216 bytes)

[sn1987a]

Is there any reason this has to be done as a file in memory, instead of a simple scalar? Just replace your

syswrite $mem_file, pack( "v*", unpack("n*", $data ));

with

$mem_file .= pack( "v*", unpack("n*", $data ));

And of course, simply use my $mem_file; instead of the open my $mem_file...;

[james28909]

yeah that def worked because i can byte reverse the file, then after the while loop i can write it to disk/hdd. i can also seek and read it ;) which is what i was after lol.

here is the final working code:

open ( my $in_file, '<', "file" );
binmode( $in_file );


my $mem_file;
binmode( $mem_file );

my ( $buf, $data, $n, $bytes ); 

while (( $n = read $in_file, $data, 4096 ) != 0){
     $mem_file .= pack( "v*", unpack("n*", $data ));   
    $bytes+=$n;
    
}


open ( my $otherMem_file, '<', \$mem_file ); 
binmode( $otherMem_file );

seek $otherMem_file, 0x00, 0;
read $otherMem_file, my $temp, 0x1200;

open my $out_file, '>', "reversed";
syswrite( $out_file, $temp);



and it give me all expected results byte for byte. :)
also i should add that i am parsing the file using its hexadecimal representaion of a binary file and i honestly dont think you would even need to binmode any of these files and it doesnt need file::slurp. completely independant code teehee