Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re: Mail::Box fails miserably when trying to open 30_000 messages maildir

by BrowserUk (Pope)
on Jun 27, 2005 at 12:07 UTC ( #470230=note: print w/replies, xml ) Need Help??


in reply to Mail::Box fails miserably when trying to open 30_000 messages maildir

An off-the-wall guess that I have no way to verify. Could it be that you are running out of memory? A quick browse of Mail::Box and it's associated modules leads me to believe that they form a quite heavily nested hierarchy of modules with each level adding another hash or two for each item. Large volumes of nested hashes, even when each individual leaf hash is quite small, can rapidly consume large volumes of space.

On my system, Perl sometimes dies silently when it runs out of space.

Maybe you could monitor programs memory usage when running it on this large directory?


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.
  • Comment on Re: Mail::Box fails miserably when trying to open 30_000 messages maildir

Replies are listed 'Best First'.
Re^2: Mail::Box fails miserably when trying to open 30_000 messages maildir
by Courage (Parson) on Jun 27, 2005 at 19:23 UTC
    as of my personal experience, 5.6.1 version was unable to get more than 1Gb of memory, whereas 5.8.x versions behaved betted with respect to this (but was much slower on getting memory, BTW), so I would agree with your reasonable point.
Re^2: Mail::Box fails miserably when trying to open 30_000 messages maildir
by monsieur_champs (Curate) on Jun 28, 2005 at 01:09 UTC

    Good point. But my test script stills printing all planned output, even that below the "count messages point". I suppose that an out-of-memory Perl shouldn't be able to print anything...

    I'm awaiting for the test script results, so I can tell you more details. Please, if you think that the posted script is not enought to expose the problem, tell me, and I will try to write a more precise test. Patches welcome, too.

      Hmm. Probably not memory then.

      I took a quick scan of the code in Mail::Box::Manager and notice something that might be relevant. I the code for M::B::M::open(), I see this:

      return if $require_failed{$class};

      and scanning back to see where $require_failed is being set and see this:

      unless($folder_type) { # Try to autodetect foldertype. foreach (@{$self->{MBM_folder_types}}) { next unless $_; (my $abbrev, $class, @defaults) = @$_; next if $require_failed{$class}; eval "require $class"; if($@) { $require_failed{$class}++; next; } if($class->foundIn($name, @defaults, %args)) { $folder_type = $abbrev; last; } } }

      I may be misinterpreting the code, but it looks to me that if it attempts to auto detect the folder type and then fails to require the module for the folder type it detects, it sets the flag to indicate the failure and skips on without logging an error. Then later, it checks the flag and if it is set, fails silently returning undef.

      Could it be misdetecting the folder type and failing silently as a result?


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://470230]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (4)
As of 2019-07-18 07:02 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?