Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re: Re: Re: iteration through tied hash eat my memory

by iburrell (Chaplain)
on Dec 09, 2002 at 20:05 UTC ( [id://218644]=note: print w/replies, xml ) Need Help??


in reply to Re: Re: iteration through tied hash eat my memory
in thread iteration through tied hash eat my memory

You would need to modify the modules to be more efficient with memory usage with huge tables. You could either modify Tie::RDBM to use the Pg module for direct PostgreSQL access with cursors. Or modify DBD::Pg to use cursors. Otherwise, the SELECT statement to fetch the keys stores all the data on the client. You might want to investiage MySQL and see if DBD::Mysql uses cursors. It also looks like you will have to modify Tie::RDBM to not cache the values that it fetches.

The important question to ask is whether you really need a tied hash. If you don't need one, then you can do operations more efficiently with custom database accesses. Since PostgreSQL doesn't use cursors, you will have to avoid any queries that return all the keys.

For example, counting the number of keys is doable with a SELECT COUNT(*) FROM table statement. You might want to normalize the database to store each value in a row of a second table. Then you won't need to use Perl serialization to store the data structures. You can the total number of values with SELECT COUNT(*) FROM values. And individual counts for each key: SELECT COUNT(*) FROM values WHERE key = $key.

Replies are listed 'Best First'.
Re^4: iteration through tied hash eat my memory
by diotalevi (Canon) on Dec 09, 2002 at 20:20 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://218644]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others having an uproarious good time at the Monastery: (7)
As of 2024-03-28 09:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found