Offline Copy of perlmonks

[LanX]

UPDATE

Never mind, I found a way to throttle wget and bend it to my needs :-)

---

Hi

I'd like to locally mirror this site for personal use. Need at least access to my thought processes of the last 2 decades.

Problem is that I don't want to put even more strain on the servers.

I looked at the wayback machine as alternative source and unsurprisingly it had problems accessing threads since "the AI" started its distributed denial of service attack.

I suppose a compressed tar ball would be in the range of 0.1 to 1.0 GB and could be offered as a torrent. (Unfortunately AI wouldn't be "I" enough to use that instead)

Hmm ... Alternatively to plain html maybe only the XML versions of the posts plus a mojolicious app running the search engine and rendering the posts.

Parts of this post rather belong to meditation, feel free to move it if appropriate.

[Corion]

Note that the XML is also mirrored at https://corion.net/perlmonks/nnnnn.xml, which likely has better performance than the site itself.

[LanX]

Thank you very much, even if they only start in 2007. And I suppose they don't reflect edit-updates by the users.

But it's a good start! :-)

Sorry for hammering your server with 700k requests.

It might be a good idea to tar -z bundles.tgz in the same number range to avoid excessive traffic.

[Fletch]

That's actually an interesting idea (a local mirror of the XML versions). Can any knowledgeable people comment with (ballpark even) how large the overall backup of the site is / might be?

[LanX]

About 1 GB ± 50% uncompressed text.

Rationale: ~ 1 million nodes at 1 kB average XML)

FWIW: Using the XML for alternative client is an idea as old as the site... It's mentioned somewhere in the doc catacombs.