Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"

Re^3: File caching for external storage

by tosh (Scribe)
on Nov 28, 2007 at 16:46 UTC ( [id://653574]=note: print w/replies, xml ) Need Help??

in reply to Re^2: File caching for external storage
in thread File caching for external storage

Exactly, server-side.

Squid is interesting I think, but I don't just want to proxy connections, I will need to do some programming.

One of the problems I have is that the PERL modules for S3 don't stream data, which means that if my application is sending a 2GB file to S3 then that 2GB file gets loaded into memory before it gets sent, that's bad. So I will store files over 500 MB locally and keep that info in a table.

But while I'm doing tables I might as well cache the most used files locally on my server instead of sending users to pick them up from S3 and incurring the bandwidth charge, say cache up to 100GB (the size of the local drive) before rotating out of the cache, and then how to determine which files should and shouldn't be in the cache.

This all seems to me to be something that somebody has to have done before, maybe not with S3, but just storing content and caching it. Maybe not...

  • Comment on Re^3: File caching for external storage

Replies are listed 'Best First'.
Re^4: File caching for external storage
by tuxz0r (Pilgrim) on Nov 28, 2007 at 17:02 UTC
    Cache::Cache looks like it has some of the functionality you need, specifically memory caching some of the smaller sizes or even caching on the local drive based on a size limit. I haven't used it before, so someone else can comment on its stability and their experiences with it.

    echo S 1 [ Y V U | perl -ane 'print reverse map { $_ = chr(ord($_)-1) } @F;'
    Warning: Any code posted by tuxz0r is untested, unless otherwise stated, and is used at your own risk.

      Cache::FileCache looks promising. Though, I have not used this myself.

      scratch that. I should have read more. The Module code may prove interesting in your coding efforts.
      "That which we persist in doing becomes easier, not that the task itself has become easier, but that our ability to perform it has improved."
        --Ralph Waldo Emerson
      Yeah, the Cache::Cache stuff is really cool for the caching part of things, but it doesn't really seem to be very good for determining what should go IN or OUT of the cache, because obviously I will want more requested files going IN and less requested files coming OUT, but certainly it might be part of my application, thanks!!!


      But Cache::SizeAwareCache might be exactly what I'm looking for, sweet!!!!

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://653574]
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others meditating upon the Monastery: (2)
As of 2024-06-12 18:10 GMT
Find Nodes?
    Voting Booth?

    No recent polls found

    erzuuli‥ 🛈The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.