Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re^2: File caching for external storage

by tcf03 (Deacon)
on Nov 28, 2007 at 14:48 UTC ( #653545=note: print w/replies, xml ) Need Help??


in reply to Re: File caching for external storage
in thread File caching for external storage

Sounds like hes looking for something server side though and my not have control over client ( browser ) proxies.
Ted
--
"That which we persist in doing becomes easier, not that the task itself has become easier, but that our ability to perform it has improved."
  --Ralph Waldo Emerson
  • Comment on Re^2: File caching for external storage

Replies are listed 'Best First'.
Re^3: File caching for external storage
by tosh (Scribe) on Nov 28, 2007 at 16:46 UTC
    Exactly, server-side.

    Squid is interesting I think, but I don't just want to proxy connections, I will need to do some programming.

    One of the problems I have is that the PERL modules for S3 don't stream data, which means that if my application is sending a 2GB file to S3 then that 2GB file gets loaded into memory before it gets sent, that's bad. So I will store files over 500 MB locally and keep that info in a table.

    But while I'm doing tables I might as well cache the most used files locally on my server instead of sending users to pick them up from S3 and incurring the bandwidth charge, say cache up to 100GB (the size of the local drive) before rotating out of the cache, and then how to determine which files should and shouldn't be in the cache.

    This all seems to me to be something that somebody has to have done before, maybe not with S3, but just storing content and caching it. Maybe not...

      Cache::Cache looks like it has some of the functionality you need, specifically memory caching some of the smaller sizes or even caching on the local drive based on a size limit. I haven't used it before, so someone else can comment on its stability and their experiences with it.

      ---
      echo S 1 [ Y V U | perl -ane 'print reverse map { $_ = chr(ord($_)-1) } @F;'
      Warning: Any code posted by tuxz0r is untested, unless otherwise stated, and is used at your own risk.

        Cache::FileCache looks promising. Though, I have not used this myself.

        UPDATE
        scratch that. I should have read more. The Module code may prove interesting in your coding efforts.
        Ted
        --
        "That which we persist in doing becomes easier, not that the task itself has become easier, but that our ability to perform it has improved."
          --Ralph Waldo Emerson
        Yeah, the Cache::Cache stuff is really cool for the caching part of things, but it doesn't really seem to be very good for determining what should go IN or OUT of the cache, because obviously I will want more requested files going IN and less requested files coming OUT, but certainly it might be part of my application, thanks!!!

        OH!!!!!

        But Cache::SizeAwareCache might be exactly what I'm looking for, sweet!!!!

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://653545]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (3)
As of 2022-05-28 08:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you prefer to work remotely?



    Results (99 votes). Check out past polls.

    Notices?