Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery

(OT) Giving users what they want / My mail administration dilemma

by submersible_toaster (Chaplain)
on Feb 18, 2003 at 02:22 UTC ( #236173=perlmeditation: print w/replies, xml ) Need Help??

G'day Fellow Monks,
I've pre-fixed the title as OT because even for a meditation this is only vaguely perl related. I have been with my present employer for about six months and in this time I have used perl as glue, gaffer, hammer, etc to solve all manner of problems. From simple (yet sophisticated?!?) backup and archiving, various perl addons to squid, workflow enhancements, labor saving scripts, perl has allowed me to do some Great Things(TM). Those of you bored with that slightly sychophant rant can skip to the OT content of the next!

My present dilemma is email, its maintenance, administration and particularly its provision of service to our users. I should point out that this is provided internally, to employees only. Now mail 'security' is a noted issue for our users, not secure=~/uncrackable/ but secure=~/I-deleted-my-2000msg-inbox-why-didnt-YOU-back-it-up/. So to avoid making targeted backups of user's local mail, for most of them - we run IMAP and backup the maildirs on the server. Sadly the existing mail server is groaning under the weight of those 2000msg IMAP mailboxes, whilst I appreciate that.

  • Upgrading the server SHOULD alleviate the problem (IO and CPU bondage).
  • There is a user-education / attitude-adjustment issue here too.

what concerns me most is that we're expanding, purchasing other business, adding users. IMAP doesn't seem to scale well. More to the point, seeking through a file for a specific message header is un-necessary work IMHO. I metioned this in the CB, and asked if anyone had seen mbox or maildirs implemented as a DB of somekind. Lotus Notes for example (though I am speaking in ignorance) seems to store gobs of data in a DB, not just user-account information (which is what I find most links to when googling for 'mailbox + sql' or suchlike).

Now I feel like the bad guy, either we leave things as they are, allowing service to deteriorate until unusable (BAD). We could place some stricture on mail volume like quotas (BACKLASH?). Lotus Notes has been suggested, but I gather this would mean migrating everyone to a brand new system (BUT-IT-WORKED-THE-OLD-WAY-BACKLASH). I like my idea of sticking with IMAP, but changing the mailbox component to request DB objects rather than grind through a single file grepping for the unique message ID requested. I have no confidence in implementing this myself, even with perl as my spanner.

Phew, reading back I'm not sure about that stream of conciousness, I am meditating my approaches, I hope to invite debate/discussion... I am seeking wisdom but I didn't feel this was quite the ilk of SOPW.

regards , toaster.

I can't believe it's not psellchecked
  • Comment on (OT) Giving users what they want / My mail administration dilemma

Replies are listed 'Best First'.
Re: (OT) Giving users what they want / My mail administration dilemma
by theorbtwo (Prior) on Feb 18, 2003 at 03:27 UTC

    In general, my answer to questions like this is that freshmeat is your friend. A search for IMAP gives quite a number of results. Freshmeat has a nice filter-by-category function, though. Hitting the filter link next to "Environment :: No Input/Output (Daemon)" gives you 16 projects, #5 of which is called DBMAIL, and seems to be exactly what you're looking for.

    As far as the path sepperator thing goes...

    1. It shouldn't be a difficult change to make at all. I suspect you wouldn't even have to know C if your source-diving skills are in good shape.
    2. What client are your users using? Would it figure it out itself?

    Also, many mail folder formats have one file per message... it may not be neccessary to go with a non-standard IMAP server.

    Warning: Unless otherwise stated, code is untested. Do not use without understanding. Code is posted in the hopes it is useful, but without warranty. All copyrights are relinquished into the public domain unless otherwise stated. I am not an angel. I am capable of error, and err on a fairly regular basis. If I made a mistake, please let me know (such as by replying to this node).

      Very big ++theorbtwo , I guess neglecting freshmeat was my 1st mistake. I get v.v.disillusioned with sourceforge, in particular the number of projects that look like onetime posters, or are just dead in the water - unmaintained.

      Users are in the majority case using Outlook Express IMAP, some are using Outlook which is M$ Small Biz not enterprise so it don't have IMAP (pricks), but it supports Exchange (painful wart on the network). I don't have too many qualms with ppls using things like netscape mail , since they're so few and interestingly enough, rarely need assistance. OSX is a rather different beast, it's Mail client is featured if you're prepared to devise an LDAP database of'll notice LDAP and ActiveDirectory feature briefly on my homenode. Thankyou again.

      I can't believe it's not psellchecked
Re: (OT) Giving users what they want / My mail administration dilemma
by dws (Chancellor) on Feb 18, 2003 at 04:32 UTC
    Now I feel like the bad guy, either we leave things as they are, allowing service to deteriorate until unusable (BAD). We could place some stricture on mail volume like quotas (BACKLASH?).

    The problem you're facing is as much political as it is technical, and you're positioning yourself to be the bad guy. You can get out from under by shifting part of the burden onto your users, and enlisting their help. Make the case now that service is going to deteriorate as the organization scales unless people can limit the amount of email they keep on the server, and ask your users what they think a reasonable retention policy might be. You won't get answers from everyone, and some folks will complain, but if you get a couple of good answers, you can always say "we asked, and the answer was X" if people complain later.

    You'll still have technical problems to deal with, but setting reasonable limits now might give you some more breathing room.

Re: (OT) Giving users what they want / My mail administration dilemma
by blahblahblah (Priest) on Feb 18, 2003 at 04:25 UTC
    Don't forget that you can have multiple imap folders.

    I keep a lot of old emails just in case I need to refer to them later. I use pine, and I've noticed that it gets slower as my inbox gets bigger. It would be unusable if I kept everything in my inbox. To speed things up I create a new imap folder once a month and move my inbox to it, thus cleaning out my inbox.

    Maybe you should encourage your users to do something like that. Even if you go to a different imap implementation, having smaller folders might help keep things quick. You could automate the process with scripts that move the folders at scheduled dates, or with scripts that run periodically and warn you about users with exceptionally large folders.

      We certainly encourage users to organise their mail, most do it in folders by 'project of relevance' but I am sympathetic in many regards... my IMAP folder for squid mailing list is monstrous, sometimes the 'all-in-one-place' method has to be traded for clunky performance. I'm mid way through build-configure dance with dbmail1.0 released in december 2002 and if it does what it says , I will be thrilled... moreover, if their proposed features become reality I might even be able to junk an Exchange Server (oh baby!). I am setting aside some time tomorrow to do some logging on the IMAP server regarding how much CPU time is spent by which users, which should give me a shortlist to 'educate-and-enlighten' :) I like the idea of warning about exceptionally large folders, maybe the mail should go to the offending user.

      I can't believe it's not psellchecked
Re: (OT) Giving users what they want / My mail administration dilemma
by Tomte (Priest) on Feb 18, 2003 at 06:58 UTC

    What server are you using?
    The Cyrus IMAP server stores everything in a Berkeley-DB-Structure, is up to now fast and seems to be, claims to be, scalable and it normaly doesn't make a difference if I search/navigate in a small folder, like a project related one (10-100)*x, or the big one containing automated mail 1000*x.

    There is a (commercial) Plugin available for Outlook that improves the IMAP support to a noticable degree, /msg me if you're interested, and I ask our Windows-Client guy what it is.

    Concerning notes: If you can circumvent this, do it. It is satisfactory, if it works...yeah, well,... if, and generaly a PITA to work with as an admin (IMVHO).


      What server? legacy stuff I am afraid, freeware imapd v11.241 on a poor old SGI indy, 180Mhz MIPS R5000, which to its credit has survived remarkably well but my fear that it is nearing the end of its natural life, is shared by the whole team (3 of us!). I asked why it wasn't retired years ago, the answer I got struck warning bells, because when paraphrased it was, 'Security through Obscurity' , more people know how to crack a linux than a irix box. I personally believe this is rubbish, IRIX has not got a fantastic trackrecord in this area, and getting patches and ports in without a licensed cc is ... well don't ask.

      Will check out Cyrus, also I've found an exchange server replacment claimant that looks interesting.

      You don't know how many people have used PITA in the same sentence as Lotus, I laffed when I read that ++

      I can't believe it's not psellchecked
        I'm going to go even farther afield, and suggest the IMAP server we use at work. It's Windows-based, but it can run on Win2k Pro. Actually it can run on NT, or even 9x, but I wouldn't recommend that. - their 6.70 release just came out the other day. They've licensed Bynari's client plugin, so that Outlook users can use IMAP behind-the-scenes (even if your version of Outlook doesn't support IMAP natively) The product is inexpensive: They have POP or IMAP in the Pro version, they have an IMAP-based web-mail product that supports SSL, and has it's own calendar that will be integrated with Outlook in the next rev, they have a server-based IM system (looking to integrate with Trillian I think.) Inexpensive add-ons: AV product plugin based on Kaspersky, but you can use something else, the Bynari client (Groupware) will be an additional cost when it's release (you can start playing with the betas now if you'd like) The only PITA for me right now - they don't support IMAP SEARCH BODY, which I believe was one of your requirements. I've made a point to get that wish-listed. MDaemon keep messages as RFC-822 text files in server-side directories, one per IMAP folder. So, a server-side search-engine would be a worthwhile project; Perl seems like it would fit the bill. Email me ( if you think you would like to hear more from me on the topic.
        'Security through Obscurity' , more people know how to crack a linux than a irix box.

        As it stands today, the entire security industry is obscurity. From passwords to cryptography it is all reliant on obscurity. You just need the proper amount of obscurity to minimize the risks for the job at hand.

        Not the greatest example, but do a search for department of defense webstar and see what pops up.

Re: (OT) Giving users what they want / My mail administration dilemma
by l2kashe (Deacon) on Feb 18, 2003 at 16:45 UTC
    I had a slightly different suggestion, but I don't know your timeline prior to the failure of the present system, nor what type of resources you can throw at this situation. So with that being said.

    Why not implement an intranet, WWW mail gateway, with a DB backend. Or at least look along theses lines.

    1) Platform independance
    2) Can reallocate/repartition storage devices, with relatively little intrusion on the clients mail interface
    3) You can remove the actual large data files (movies I think you said in CB), out of the mail box, and onto an optomized partition for d/l.
    3a) Or maybe instead move the large data files into a partition which can then be automounted or mapped to via samba.
    4) All administration of the system should be seamless, and this model provides a nice upgrade path as you can segregate the WWW from the maildata, and even the attatchments. If any piece is under excessive load, you simply add another server, and add round-robin DNS entries.

    1) You now need to figure out a clean consistant and stable method for uploading content to be sent out via email.
    2) Your users now need to get acclimated to a new interface
    3) Your systems complexity has gone up an order of magnitude. You are dealing with NFS, HTTP and HTTPS, samba, some type of DB backend, and subsystem I/O monitoring
    4) You need to figure out a method for importation/exportation of "address book" type features. I've yet to see an outlook user who simply couldnt do without that feature.

    So just some random thoughts, but it could be interesting. And even if you dont use this setup, it may help in the decision making as to what components you might like to see in the future, which defines what pieces you implement now to work your way there.

    Happy Hacking

    /* And the Creator, against his better judgement, wrote man.c */

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://236173]
Approved by krujos
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (5)
As of 2021-03-07 19:13 GMT
Find Nodes?
    Voting Booth?
    My favorite kind of desktop background is:

    Results (122 votes). Check out past polls.