Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options

Re: Memory Management Problem

by Roger (Parson)
on Nov 21, 2003 at 00:07 UTC ( #308764=note: print w/replies, xml ) Need Help??

in reply to Memory Management Problem

If you want to minimize memory usage, you will have to suffer in speed somewhere, somehow. There is a trade off that you have to make.

I think one of the easiest method is to use a proper database to record your files, like MySQL, Postgres, Oracle, etc. Create a table with an *index* on the filename to speed up SQL query. Let the database do the search optimization for you.

#!/usr/local/bin/perl -w use strict; use DBI; use DBD::Sybase; use File::Find; my $dbh = ....; # connect to database my $sth = $dbh->prepare("select filename from bench where filename=?") +; # look for new files my @new_files; find( { follow => 1, no_chdir => 1, wanted => sub { if (! /\.$/) { # ignore unwanted . or .. $sth->execute($_); my $file_exists; while (my @res = $sth->fetchrow_array()) { $file_exists+ ++ } push @new_files if !$file_exits; } } }, '/'); $sth->finish; # insert new files into the database $sth = $dbh->prepare("insert into bench (filename) values (?)"); $sth->execute($_) for @new_files; $sth->finish; # do stuff with @new_files ....
I can think of another solution, although the performance in speed is not great, that uses resonable amount of memory.

#!/usr/local/bin/perl -w use strict; use File::Find; my $bench_file = 'bench.txt'; my @new_files; find( { follow => 1, no_chdir => 1, wanted => \&callback }, '/'); sub callback { if (! /\.$/) { # ignore unwanted . or .. if ( ! `grep '$_' $bench_file` ) { push @new_files, $_; # remember this file } } } # append unseen filenames to bench.txt file open BENCH, ">>bench.txt" or die "Can not append to bench.txt"; print BENCH "$_\n" foreach (@new_files); close BENCH;

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://308764]
[karlgoethebier]: good morning
[Corion]: Hi karlgoethebier!
[Discipulus]: good morning karlgoethebier

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (10)
As of 2017-07-25 08:58 GMT
Find Nodes?
    Voting Booth?
    I came, I saw, I ...

    Results (366 votes). Check out past polls.