Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re: Memory issue with large array comparison

by dave_the_m (Prior)
on May 24, 2012 at 21:29 UTC ( #972326=note: print w/replies, xml ) Need Help??


in reply to Memory issue with large array comparison

The following code, scaled up to 500,000 filenames and 100,000 IDs, takes 2 seconds to run and uses 90Mb approx.
use warnings; use strict; # create some sample data my (@pathnames, @safe_list); push @pathnames, sprintf "C:/abc/abc1/GS%06d", $_ for 1..500_000; push @safe_list, sprintf "GS%06d", $_ for 1..100_000; my %safe_hash; $safe_hash{$_} = 1 for @safe_list; my @list; for (@pathnames) { # (adjust regex to match whatever the filename/ID format is) /\/(\w+)$/ or die "bad pathname: $_"; push @list, $_ unless $safe_hash{$1}; }

Dave.

Replies are listed 'Best First'.
Re^2: Memory issue with large array comparison
by bholcomb86 (Novice) on May 25, 2012 at 00:00 UTC

    I decided to try Dave's solution and turns out it works perfectly. I just wanted to solve my memory problem which this does, but as a side benefit it is WAY faster than my previous solution. Consider this problem solved! I appreciate all input given on this matter.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://972326]
help
Chatterbox?
[Corion]: Discipulus: Notepad++, or Perl ;)
[marto]: xubuntu or lubuntu
[Corion]: :-D
[marto]: my desktop is a core2 duo, 120GB SSD, 4GB RAM, boots to (GUI) login in less than 10 seconds from cold start
[Discipulus]: mah, winbuntu 10...
[marto]: whent he boys start school I'll build myself as new machine, ryzen based
[Corion]: marto: That's cool! I think my (home) desktop spends 30 seconds in the BIOS...
[Corion]: marto: Heh - currently they seem promising, but I think I'll stay with nVidia+Intel for the time being, as I've been bitten too often by bad AMD drivers
Discipulus is (still?) not a big fan of notepad++

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (5)
As of 2017-07-27 08:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    I came, I saw, I ...
























    Results (407 votes). Check out past polls.