Well, the process is multithreaded. So even though over a day I might generate 100 MB at most at the top logging level, after 40 - 50 forks we're talking about 4GB - 5GB a day. This has compounded the problem because I'm trying to keep the logs sorted and rotated. And, even though I can turn down the detail, the bugs I am finding require a high level of detail for testing.
(The script is a web spider. Most of the bugs I encounter with it involve bizarre / broken HTML in web pages. Problem is that in order to figure out just what is going on I want to log lots of info if there are any anomalies. The problem becomes how to do that without being too processor intensive)
Want to support the EFF and FSF by buying cool stuff? Click here.
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
|
