http://www.perlmonks.org?node_id=1051430


in reply to Re: db content dump (tickers)
in thread db content dump

At 1 request/s and ~1 million nodes that would take 12 days to crawl everything, also causing a huge traffic. Can you run this on the server and make a static dump/snapshot?

Replies are listed 'Best First'.
Re^3: db content dump (tickers)
by tye (Sage) on Aug 29, 2013 at 14:47 UTC

    The web site is not static. One static dump just begets another static dump.

    Do you have a successful replacement web site all ready?

    I'm pretty sure that there is more than 12 days of work yet ahead of you and that the vast majority of that work does not require ~1 million nodes of sample data.

    Having tried to produce dumps for export before and having seen others try, it is not simple, is not fast, and is prone to important mistakes. That it may appear very simple from where you are sitting does not actually have the power to change any of that.

    - tye