Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Re: Re: Controlling file size when writing

by Vautrin (Hermit)
on Mar 02, 2004 at 22:45 UTC ( #333393=note: print w/replies, xml ) Need Help??

in reply to Re: Controlling file size when writing
in thread Controlling file size when writing

Well, the process is multithreaded. So even though over a day I might generate 100 MB at most at the top logging level, after 40 - 50 forks we're talking about 4GB - 5GB a day. This has compounded the problem because I'm trying to keep the logs sorted and rotated. And, even though I can turn down the detail, the bugs I am finding require a high level of detail for testing.

(The script is a web spider. Most of the bugs I encounter with it involve bizarre / broken HTML in web pages. Problem is that in order to figure out just what is going on I want to log lots of info if there are any anomalies. The problem becomes how to do that without being too processor intensive)

Want to support the EFF and FSF by buying cool stuff? Click here.
  • Comment on Re: Re: Controlling file size when writing

Replies are listed 'Best First'.
Re: Re: Re: Controlling file size when writing
by fokat (Deacon) on Mar 03, 2004 at 04:30 UTC

    And this is why you need production and dev boxes. In the production box, you only need enough logging so as to know when something breaks. In your dev box, you can re-harvest the offending pages and see the errors in all its glory, including the ability to add instrumentation to the code on the fly.

    If you cannot have the two boxes, perhaps you can run a second instance of your spider manually, when needed. I bet this is less resource-intensive that managing such huge logs.

    That being said, take a look at Logfile::Rotate.

    Best regards

    -lem, but some call me fokat

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://333393]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (2)
As of 2022-05-29 08:59 GMT
Find Nodes?
    Voting Booth?
    Do you prefer to work remotely?

    Results (101 votes). Check out past polls.