|Problems? Is your data what you think it is?|
Organising mbox into threads?by LoonyPandora (Novice)
|on May 22, 2006 at 16:26 UTC||Need Help??|
LoonyPandora has asked for the
wisdom of the Perl Monks concerning the following question:
Here is a question that has been plaguing me for a while... It's a bit long, so I apologise in advance. My eternal gratefulness if anyone can help :)
I have a fairly large mbox file of around 105,000 messages, and 350 MB in size. This is a backup of a Yahoo! groups message board that I created previously with my own code. It is well formed and otherwise in good condition - complete with message-id, and in-reply-to headers for 99.5% of the messages.
I am looking to get these messages into a Bulletin Board - The most important thing I need to do is organize the messages into threads, rather than the flat listing that Yahoo! groups provides.
In order to get these messages into my bulletin board, I need to get arrange the messages into 1 mbox file per thread, or one directory per thread. I've looked at the Mail::Box modules (which seems a bit complex for my puny mind...) and Mail::MboxParser (which seems to just parse mbox files, which I've already done successfully)
Any code that I have come up with is very slow - as basically it runs a nested loop - gets the message-id for the first message, and then loops through each message in the file again checking to see if there is a matching 'in-reply-to' header, then writes them both out to a new file. This means I am processing 105k*105k messages- which takes WAY too long to do it's work.
I've tried creating a hash of message-id's, and then adding any messages with a matching 'in-reply-to' header to the relevant key - faster, but has a flaw - if a message is in reply to the 2nd or 3rd message in a thread - there is no relevant key to add it to because it isn't the start of a thread. Meaning I can only process one level deep.
All other methods I have thought of have the same flaw as the hash method described above. I am unable to think of a way to do this without processing 105k*105k messages.
My question distils to this really: Is there any way to arrange this mbox file into threads organised by message-id WITHOUT looping through it 105k*105k times?
If I'm asking the impossible, I could post my exisiting code up and see if it is possible to optimise it and make it run any quicker. If anyone would be willing to help with that?Many Thanks,