Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"

Re^2: db content dump (tickers)

by daxim (Curate)
on Aug 29, 2013 at 13:08 UTC ( #1051430=note: print w/replies, xml ) Need Help??

in reply to Re: db content dump (tickers)
in thread db content dump

At 1 request/s and ~1 million nodes that would take 12 days to crawl everything, also causing a huge traffic. Can you run this on the server and make a static dump/snapshot?

Replies are listed 'Best First'.
Re^3: db content dump (tickers)
by tye (Sage) on Aug 29, 2013 at 14:47 UTC

    The web site is not static. One static dump just begets another static dump.

    Do you have a successful replacement web site all ready?

    I'm pretty sure that there is more than 12 days of work yet ahead of you and that the vast majority of that work does not require ~1 million nodes of sample data.

    Having tried to produce dumps for export before and having seen others try, it is not simple, is not fast, and is prone to important mistakes. That it may appear very simple from where you are sitting does not actually have the power to change any of that.

    - tye        

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1051430]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (6)
As of 2020-11-28 13:10 GMT
Find Nodes?
    Voting Booth?

    No recent polls found