Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Re: File caching for external storage

by KurtSchwind (Chaplain)
on Nov 28, 2007 at 14:21 UTC ( [id://653536]=note: print w/replies, xml ) Need Help??


in reply to File caching for external storage

I'll second the squid love.

Install squid. Set all your clients to proxy to your squid server and run with it.

--
I used to drive a Heisenbergmobile, but every time I looked at the speedometer, I got lost.

Replies are listed 'Best First'.
Re^2: File caching for external storage
by tcf03 (Deacon) on Nov 28, 2007 at 14:48 UTC
    Sounds like hes looking for something server side though and my not have control over client ( browser ) proxies.
    Ted
    --
    "That which we persist in doing becomes easier, not that the task itself has become easier, but that our ability to perform it has improved."
      --Ralph Waldo Emerson
      Exactly, server-side.

      Squid is interesting I think, but I don't just want to proxy connections, I will need to do some programming.

      One of the problems I have is that the PERL modules for S3 don't stream data, which means that if my application is sending a 2GB file to S3 then that 2GB file gets loaded into memory before it gets sent, that's bad. So I will store files over 500 MB locally and keep that info in a table.

      But while I'm doing tables I might as well cache the most used files locally on my server instead of sending users to pick them up from S3 and incurring the bandwidth charge, say cache up to 100GB (the size of the local drive) before rotating out of the cache, and then how to determine which files should and shouldn't be in the cache.

      This all seems to me to be something that somebody has to have done before, maybe not with S3, but just storing content and caching it. Maybe not...

        Cache::Cache looks like it has some of the functionality you need, specifically memory caching some of the smaller sizes or even caching on the local drive based on a size limit. I haven't used it before, so someone else can comment on its stability and their experiences with it.

        ---
        echo S 1 [ Y V U | perl -ane 'print reverse map { $_ = chr(ord($_)-1) } @F;'
        Warning: Any code posted by tuxz0r is untested, unless otherwise stated, and is used at your own risk.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://653536]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others musing on the Monastery: (7)
As of 2024-06-12 17:36 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?
    erzuuli‥ 🛈The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.