Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

RE: Schwartzian Transform

by perlcgi (Hermit)
on Apr 26, 2000 at 15:42 UTC ( #9216=note: print w/ replies, xml ) Need Help??


in reply to Schwartzian Transform

Fellow Seekers of Perl Wisdom, maybe I'm wrong, but I suspect the Schwartzian Transform be only good for datas that will fit in memory all at once. Given that Perl 5.6 can munge with files over 2Gb, (on systems that support files this big), can anyone suggest an elegant mod of the transform to accommodate big mother f.. files!


Comment on RE: Schwartzian Transform
RE: RE: Schwartzian Transform
by chromatic (Archbishop) on Apr 26, 2000 at 19:01 UTC
    Use a tied array (@input) that operates on a disk file?
      Thanks chromatic, Despite the tie of @input, should not the list of references to lists overflow with very large files. Also should @a and @b be tied and what about the anonymous array. Is it possible to tie anonymous arrays, and if so would it, and the all the others require serialization? TIA
      No.

      First of all mixing references and arrays tied to disk without thinking carefully about it is asking for serious problems.

      Secondly passing the array to Perl's sort function is asking for very serious trouble. That will pass it all into memory!

      What I would do for large data structures would be to use DB_File to tie a hash to a BTree, and use properly formatted strings as keys. Blech. But it will work up to the maximum file size for your OS. (OK, up to about half that - BTrees waste something like 40% of the space in the tree.) Or use a proper database.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://9216]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (3)
As of 2014-08-30 23:57 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (294 votes), past polls