Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Re^2: db content dump (tickers)

by daxim (Chaplain)
on Aug 29, 2013 at 13:08 UTC ( #1051430=note: print w/ replies, xml ) Need Help??


in reply to Re: db content dump (tickers)
in thread db content dump

At 1 request/s and ~1 million nodes that would take 12 days to crawl everything, also causing a huge traffic. Can you run this on the server and make a static dump/snapshot?


Comment on Re^2: db content dump (tickers)
Re^3: db content dump (tickers)
by tye (Cardinal) on Aug 29, 2013 at 14:47 UTC

    The web site is not static. One static dump just begets another static dump.

    Do you have a successful replacement web site all ready?

    I'm pretty sure that there is more than 12 days of work yet ahead of you and that the vast majority of that work does not require ~1 million nodes of sample data.

    Having tried to produce dumps for export before and having seen others try, it is not simple, is not fast, and is prone to important mistakes. That it may appear very simple from where you are sitting does not actually have the power to change any of that.

    - tye        

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1051430]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (12)
As of 2014-11-24 14:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (141 votes), past polls