I second that. You would be increasing your performance by a great deal just by finding a clever way to keep the attachments directly on the filesystem. I would suppose that you have a user primary key and a message primary key, correct? If so, you could try doing a directory tree like:
/users/
/user1/
/attachmenta
/attachmentb
/attachmentc
/user2/
/attachmenta
/attachmentb
/attachmentc
Where userx would be the user pk and attachmentx would be the attachment pk (doh!). On the downside, it would probably be nice to develop a tool or cron job to keep track of lost attachments, old stuff and so forth...
#!/home/bbq/bin/perl
# Trust no1!