Speeding up a mailing list script

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi to all,
I have come here due to the high reputation in coding perl you seem to have developed among the people i ran into in the web.
Let me explain:
I've written a mailing script, the main part of whic is in PHP (sorry the rest of my site is in PHP so...) but the actual part which sends the newsletter to my list is in Perl. Now the script works fine but it needs to be speeded up.
Why?
Because my server (low cost hosting) times out after about one hour. Considering it sends approx 12000 30k messages in this time even shing off 0.1 second per mail sent would mean increasing production by one third.
So what I'm here for is your experience and your ability.
How can I cut useless process time?

Here is the script:

#!/usr/bin/perl

read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});
@pairs = split(/&/, $buffer);
foreach $pair (@pairs) {
    ($name, $value) = split(/=/, $pair);
    $value =~ tr/+/ /;
    $value =~ s/%0D%0A/|/g;
    $value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
        if ($in{$name}) {
         $in{$name} = $in{$name}.",".$value;
          }
        else {
         $in{$name} = $value;
          }
}

    $in{'txt_message'} =~ s/\|/%0D%0A/g;
    $in{'txt_message'} =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1
+))/eg;
    $text_message = $in{'txt_message'};
    $in{'html_message'} =~ s/\|/%0D%0A/g;
    $in{'html_message'} =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($
+1))/eg;
    $html_message = $in{'html_message'};
    $totalEmails = $in{'rightrange'} - $in{'leftrange'} + 1;
    $time_start = time() + $in{'time_diff'}*3600;
    $lock = "2";

$pid = fork();
print "Content-type: text/html \n\n fork failed: $!" unless defined $p
+id;
if ($pid) {

    if ($in{'archive'} eq "1") {
    &archive;
     #an archiving procedure at the end of this script
    }

    print "Content-type: text/html \n\n";
    print "<html><head><title>List Administration</title></head><body>
    <br><br><br><center>
    Congratulations!<br>The mailing has been started. You will receive
+ a confirmation e-mail when the mailing has been completed.
    <P><FORM><INPUT TYPE=BUTTON VALUE=\"Back\" onClick=\"history.go(-1
+)\"></form></center></body></html>";

    exit(0);
}
else {
    close (STDOUT);
    $count = 0;

    open(LIST,"$in{'list_dir'}/$in{'list'}/$in{'list'}.txt");
    if ($lock){flock(LIST, $lock);}
    @addresses=<LIST>;
    close(LIST);

    $text_message =~ s/(.{60}\s)/$1\n/g;
    
    my @boundaryv = (0..9, 'A'..'F');
    srand(time ^ $$);
    for (my $i = 0; $i++ < 24;) {
    $boundary .= $boundaryv[rand(@boundaryv)];
    }

for ($count = ($in{'leftrange'} - 1); $count < $in{'rightrange'}; $cou
+nt++) {
    
     $member = @addresses[$count];
    chomp($member);

open (MAIL, "|$in{'mailprog'} -t") || die "Can't open $in{'mailprog'}!
+\n";
    if ($in{'type'} eq "html") {
        print MAIL "Content-type:text/html\n";
    }     
    print MAIL "From: $in{'adminMail'}\n";
    print MAIL "To: $member\n";
    print MAIL "Subject: $in{'subject'}\n";
     if ($in{'type'} eq "multipart") {
        print MAIL "MIME-Version: 1.0\n";
        print MAIL "Content-Type: multipart/mixed; boundary=\"----=_Ne
+xt_Part_$boundary\"\n";
    }
    if ($in{'type'} eq "multipart") {
        print MAIL "------=_Next_Part_$boundary\n";
        print MAIL "Content-type: text/plain;\n";
        print MAIL "charset=us-ascii\n";
        print MAIL "Content-Transfer-Encoding: quoted-printable\n\n";
    }

     if ($in{'type'} eq "text" || $in{'type'} eq "multipart") {
        print MAIL "$text_message\n\n";
              if ($in{'remove_notice'} eq "1") {
                  print MAIL "----------------------------------------
+-----------------------------\n";
                  print MAIL "Removal notice";
                  print MAIL "$in{'subscribeUrl'}?$unsubscribe&$in{'li
+st'}&member\n";
                  print MAIL "----------------------------------------
+-----------------------------\n\n";
            }
     }
    if ($in{'type'} eq "multipart") {
        print MAIL "------=_Next_Part_$boundary\n";
        print MAIL "Content-type: text/html;\n";
        print MAIL "charset=\"base64\"\n";
        print MAIL "Content-Transfer-Encoding: quoted-printable\n\n";
    }
     if ($in{'type'} eq "multipart" || $in{'type'} eq "html") {
          print MAIL "$html_message\n\n";
            if ($in{'remove_notice'} eq "1") {
                   print MAIL "<BR><BR>";
                   print MAIL "---------------------------------------
+------------------------------<br>";
                   print MAIL "HTML removaql notice";
                   print MAIL "<a href=\"$in{'subscribeUrl'}?$unsubscr
+ibe&$in{'list'}&member\">$in{'subscribeUrl'}?$unsubscribe&$in{'list'}
+&member</a><br>";
                   print MAIL "---------------------------------------
+------------------------------<br><br>";
            }
    }
    close MAIL;

    if (int($count/100)*100 == $count) {
    $time_now = time() + $in{'time_diff'}*3600;
    open(LOG, ">$in{'list_dir'}/$in{'list'}/log_sent.txt");
    if ($lock){flock(LOG, $lock);}  
    $status = $count - $in{'leftrange'} + 1;
    print LOG "$time_start" . "::" . "$totalEmails" . "::" . "$status"
+ . "::" . "$time_now";
    close(LOG);
    }
}
}

    $time_now = time() + $in{'time_diff'}*3600;
    open(LOG, ">$in{'list_dir'}/$in{'list'}/log_sent.txt");
    if ($lock){flock(LOG, $lock);}
    print LOG "$time_start" . "::" . "$totalEmails" . "::" . "$count" 
+. "::" . "$time_now" . "::end";
    close(LOG);

open (MAIL, "|$in{'mailprog'} -t");
    print MAIL "From: $in{'adminMail'}\n";
    print MAIL "To: $in{'adminMail'}\n";
    print MAIL "Subject: Congratulations!\n\n";
    print MAIL "Congratulations!\nThe mailing was successfully sent to
+ $totalEmails people by $ENV{'REMOTE_ADDR'}.\nMailing started at $tim
+e_start and ended at $time_now.\nHere is what was sent:\n\n-----\n\n"
+;
    print MAIL "Type - $in{'type'}\n\n";
    print MAIL "Subject: $in{'subject'}\n\n$in{'message'}\n\n-----";
    close (MAIL);

exit;
[download]

Note -
All variables are passed through the form.

Questions -
1. Does including the archiving routine within the script slow it down?
2. What factors lower the number of emails which can be sent before time-out: message size, script size...
3. Any suggestions?

I would like to thank you all for your help and patience.

Adrien

Comment on Speeding up a mailing list script Download Code

Replies are listed 'Best First'.
Re: speeding up a script by atcroft (Abbot) on Dec 11, 2001 at 19:32 UTC
You might look at benchmarking the code to see what portions take the longest time. You might look at possibly using the NET::SMTP or another module for sending mail, and see if it would be feasible to use that module to send the message directly (or queue them up to your outgoing mailserver), rather than opening an external program to send each message-my guess would be that that is probably the most expensive portion of the program. (You might want to look at performing some form of benchmarking to verify this.) Also, if the same message is being sent to each recipient, you might consider building in memory a template of the message, make a copy then do a search/replace on the portions that are changing. I don't know if that would save much time-you'd need to look at benchmarking the change to be sure. I'm still learning when to benchmark code as well, but I think this would be an instance where it would prove quite helpful. I hope some of the more knowledgable monks will be able to provide more helpful information, and that this helps in some small way. Good luck with the project.	[reply]
Re: speeding up a script by count0 (Friar) on Dec 11, 2001 at 19:30 UTC
Well for starters, I'd like to point out that these are just some suggestions, and aren't entirely related to what you're asking. To make your life easier, as well as the lives of those who might have to maintain this, you should really check out some modules for the common tasks in your script. CGI (included with any perl distribution) would be perfect for getting the parameters sent to this script, as well as printing the HTTP headers. For sending email there are a whole slew of modules available, including Mail::Sendmail and Mail::Mailer. I'm not sure exactly what runs slow with your script, but if a problem lies in the amount of time it takes recepients to get the email, there is a solution. Most likely what would cause this the local mail transport agent queueing the messages to be sent, and periodically sending them all out. This can be avoided by using a module (such as Mail::Sendmail) that will send the mail right away.	[reply]
Re: speeding up a script by joealba (Hermit) on Dec 11, 2001 at 19:39 UTC
Consider using Parallel::ForkManager to fork off 10 or 20 (or more, depending on the machine and load) e-mailing processes. Check out the sample code in this node. Also, please use Super Search. You'll find that this question has already been answered. Keep searching to learn more about valuable modules like CGI.pm (to help pull in your parameters), and Email::Valid (so you don't send e-mails to invalid e-mail addresses).	[reply]
(Ovid) Re: speeding up a script by Ovid (Cardinal) on Dec 11, 2001 at 22:21 UTC
As has been noted before, you should be using CGI. I realize that the form-handling code that you are using is very common, but it is also very broken. I'm particularly curious about this line: `$value =~ s/%0D%0A/\|/g;` [download] That says "take every cntl-M, followed by a cntl-J and change that combination to a pipe." I have no idea what's going on with that. It should be noted that it's very easy to have whopping security holes with mail programs. You might want to check out STAMP (Secure, Template-Aware Mail Program) by btrott. Two quick speed tips which should help tremendously. One, you appear to be iterating over a list and then opening the mail program in every iteration. If you keep using the mail program, see if you can move the open statement outside of the iteration. Iterating over a huge list and re-opening that every time will slow things down tremendously. The other thing I would do is switch to HERE documents. You'll find them much faster. With a HERE doc, you specify a label and tell Perl to print until it encounters that label. Here's the basic syntax: `print <<"END_HERE_DOC"; This will be printed END_HERE_DOC` [download] A couple of things to note about HERE docs: The closing label must be on the line by itself with no semi-colon (I can't tell you how many times I've added the semi-colon by accident). There should also be a return after the HERE document or Perl can't tell where it ends (note: that's only going to happen if the label is at the end of the file). Here's a quick benchmark I whipped up to show the performance increase between multiple print statements and HERE documents. use strict; use Benchmark; open NUL, "> nul" or die "Cannot open nul: $!"; for ( 1 .. 4 ) { test_em(); } sub test_em { timethese (5000000, { 'print' => 'print NUL "This is one line.\n"; print NUL "This is another line\n"; print NUL "Still printing\n"; print NUL "And printing and printing\n"; print NUL "Like the stupid Battery Bunny...\n";', 'here docs' => 'print NUL <<END_HERE; This is one line. This is another line Still printing And printing and printing Like the stupid Battery Bunny... END_HERE ' } ); } [download] We're printing to NUL to avoid having this print to the screen (thanks for that tip, tye). Here's the output: `Benchmark: timing 5000000 iterations of here docs, print... here docs: 21 wallclock secs (19.04 usr + 0.95 sys = 19.99 CPU) @ 25 +0150.09/s (n=5000000) print: 44 wallclock secs (40.69 usr + 1.26 sys = 41.95 CPU) @ 11 +9189.51/s (n=5000000) Benchmark: timing 5000000 iterations of here docs, print...` [download] As you can see from this limited example, the HERE document is over twice as fast as the print statements. I imagine you'll experience even greater performance benefits. Cheers, Ovid Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.	[reply] [d/l] [select]
Re: speeding up a script by perrin (Chancellor) on Dec 11, 2001 at 21:17 UTC
use CGI! Also, use mod_perl if you possibly can. There are some suggestions about sending mail from mod_perl here. These apply for CGI scripts too. The key idea is to queue the mail and not make the user wait for it. You can do this easilly with qmail-inject if you run qmail, and there are similar ideas for other mail programs. Also, archiving while your user sites at the other end of the connection is definitely bad. Under mod_perl, you can put things like that in a post_connection handler. With CGI, it's trickier. You have to fork. This column by merlyn shows how to do it.	[reply]
Re: speeding up a script by belg4mit (Prior) on Dec 11, 2001 at 22:37 UTC
Well the first thing I'd do (just for maintainability and clarity) if ($in{'type'} eq "multipart") { print MAIL "MIME-Version: 1.0\n"; print MAIL "Content-Type: multipart/mixed; boundary=\ +"----=_Next_Part_$boundary\"\n"; } if ($in{'type'} eq "multipart") { print MAIL "------=_Next_Part_$boundary\n"; print MAIL "Content-type: text/plain;\n"; print MAIL "charset=us-ascii\n"; print MAIL "Content-Transfer-Encoding: quoted-printab +le\n\n"; } if ($in{'type'} eq "text" \|\| $in{'type'} eq "multipart") { print MAIL "$text_message\n\n"; if ($in{'remove_notice'} eq "1") { print MAIL "------------------------- +--------------------------------------------\n"; print MAIL "Removal notice"; print MAIL "$in{'subscribeUrl'}?$unsu +bscribe&$in{'list'}&member\n"; print MAIL "------------------------- +--------------------------------------------\n\n"; } } if ($in{'type'} eq "multipart") { print MAIL "------=_Next_Part_$boundary\n"; print MAIL "Content-type: text/html;\n"; print MAIL "charset=\"base64\"\n"; print MAIL "Content-Transfer-Encoding: quoted-printab +le\n\n"; } if ($in{'type'} eq "multipart" \|\| $in{'type'} eq "html") { print MAIL "$html_message\n\n"; if ($in{'remove_notice'} eq "1") { print MAIL "<BR><BR>"; print MAIL "------------------------- +--------------------------------------------<br>"; print MAIL "HTML removaql notice"; print MAIL "<a href=\"$in{'subscribeU +rl'}?$unsubscribe&$in{'list'}&member\">$in{'subscribeUrl'}?$unsubscri +be&$in{'list'}&member</a><br>"; print MAIL "------------------------- +--------------------------------------------<br><br>"; } } [download] becomes my $msg = ''; my $multihead =<<EOMULTIHEAD; MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Next_Part_$boundary" ------=_Next_Part_$boundary Content-type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable EOMULTIHEAD my $multibound =<<EOMULTIBOUND; ------=_Next_Part_$boundary Content-type: text/html; charset="base64" Content-Transfer-Encoding: quoted-printable EOMULTIBOUND my $removal =<<EOREMOVE; --------------------------------------------------------------------- Removal notice $in{'subscribeUrl'}?$unsubscribe&$in{'list'}&member --------------------------------------------------------------------- EOREMOVE my $htmlremove =<<EOHTMLREMOVE; <BR><BR> ---------------------------------------------------------------------< +br> HTML removal notice <a href="$in{'subscribeUrl'}?$unsubscribe&$in{'list'}&member">$in{'sub +scribeUrl'}?$unsubscribe&$in{'list'}&member</a><br> ---------------------------------------------------------------------< +br><br> EOHTMLREMOVE $msg .= $multihead if $in{'type'} eq "multipart"; unless($in{'type'} eq 'html') { $msg .= $text_message; #XXX include the newlines in the message body $msg .= $removal if $in{'remove_notice'} eq "1"; } $msg .= $multibound if $in{'type'} eq "multipart"; unless($in{'type'} eq 'text') { $msg .= $html_message; #XXX include the newlines in the message body $msg .= $htmlremove if $in{'remove_notice'} eq "1"; } print MAIL $msg; [download] This seperates the wheat(code) from the chaff(mail), and saves on print statements (which will speed it up). UPDATE: Damn, Ovid finished his reply first ;-) `-- perl -p -e "s/(?:\w);([st])/'\$1/mg"`	[reply] [d/l] [select]
Re: speeding up a script by Fastolfe (Vicar) on Dec 12, 2001 at 03:47 UTC
I think most of the other posts catch speed issues. I have a couple of things I wanted to suggest though: use CGI or die; use strict and warnings The code `print "Content-type: text/html \n\n fork failed: $!" unless defined $pid;` could be re-written using CGI::Carp simply as: `die "fork failed: $!" unless defined $pid;` The code `open(LIST,"$in{'list_dir'}/$in{'list'}/$in{'list'}.txt");` introduces some very serious security issues. Use taint-checking (-T), read perlsec and maybe skim through Sanitizing user-provided path/filenames. I could pass "\|/bin/rm -rf " as 'list_dir' and you would be hurting the next day. Similarly, this is just as alarming: `open (MAIL, "\|$in{'mailprog'} -t");`. Anyone can pass whatever command in 'mailprog' they want. You'd do better to specify this as a constant towards the top of your script. Never trust the browser. Hope this helps.	[reply] [d/l] [select]
Re: speeding up a script by Anonymous Monk on Dec 11, 2001 at 20:13 UTC
I would like to thank you all for your help. I originally thought of using Mail::Bulkmail as a module, but unluckily i do not have root access and the providers will not install it. Further I am not sucha good coder and would need step by step indications on how to work with these. Here is a list of the module my host provides me. http://www.resources.hostway.com/manual/323.htm Can somebody help me with this. Thanks adrien	[reply]
Re: Re: speeding up a script by Amoe (Friar) on Dec 12, 2001 at 02:50 UTC
You can still use Mail::Bulkmail. Just install it to your home directory, and then do `use lib '/home/ade';` on wherever you installed it to. -- my one true love	[reply] [d/l]
Re: speeding up a script by Anonymous Monk on Dec 11, 2001 at 20:36 UTC
for atcroft, how do i benchmark? thanks adrien	[reply]
Re: Re: speeding up a script by atcroft (Abbot) on Dec 11, 2001 at 21:44 UTC
I'm still learning to benchmark myself, and don't want to lead you astray, so the best I can do would be suggest the tutorial Benchmarking Your Code , and the sections related to benchmarking/the Benchmark module in Programming Perl and Mastering Algorithms with Perl (both published by O'Reilly) and David Cross's Data Munging with Perl (published by Manning). Can some of the more knowledgable monks please suggest additional sources for information?	[reply]
Re: speeding up a script by Anonymous Monk on Dec 11, 2001 at 21:40 UTC
for perrin i'm not sure i understood the links you provided me fully, but i'm tring to send a multipart message, while Net::SMTP allows only a simple text message. or am i wrong? adrien	[reply]
Re: speeding up a script by Anonymous Monk on Dec 11, 2001 at 23:25 UTC
to Ovid and belg4mit, thanks to both of you for the excellent suggestions, i'll try them both and post both the code and the results ciao adrien	[reply]
Re: Speeding up a mailing list script by Anonymous Monk on Dec 12, 2001 at 13:44 UTC
Hi to all again and thanks for the great help, Considering the amount of help i'm receiving i should clear things up. Firstly about me: 1. I can get a script to work (or apparently do so), but in no way will it be efficient or secure. Although I may not be a beginner I am far from being experienced. About the script: 1. This script is (toghether with others) going to be used for managing a newsletter: people will be able to subscribe and remove themselves and only I will be able to send them the newsletter. 2. The script is going to be in php with the exclusion of the bulk mail section which is going to be in perl. 3. I am on a low cost server. The script is going to be divided in two parts 4. An admin section which is going to be protected by .htaccess (of which this perl script is going to be part) 5. A public script which can be accessed by everybody With these premises let me update you with the situation. 1. I am now testing the script having removed as many print MAIL as possible. 2. Unluckily moving the open MAIL and close MAIL outside the iteration does not seem to work. Maybe I'm doing something wrong and somebody ccould suggest something. 3. I don't have a clue how to use CGI. 4. I am equally clueless about security issues. (maybe somebody could help me here) 5. For Ovid: $value =~ s/%0D%0A/\|/g; I am not sure what security implications this has, I've always used it in order not to loose line breaks in the textarea, maybe you now of better ways. I am now testing: I will be posting results and the "modified" script as soon as I've finished. Again thanks to all for your help. ciao, Adrien	[reply]
Re: Speeding up a mailing list script by Anonymous Monk on Dec 12, 2001 at 14:49 UTC
Ok, so here are the test results Test conditions - 50.000 emails 30k message type - multipart server time out time - average after approx 60mins Results 1. using the original script at 11.59am (nyc time) i managed to send 10.600 emails 2. using the NEW script (here below) at 4.00am (nyc time) i sent 9.800 emails 3. using the original script at 13.10am (nyc time)sending an empty mail (0k), by mistake, i sent 19.400 emails 4. using the NEW script at 3.00am (nyc time) BUT with the open MAIL and close MAIL tags positioned just outside the loop i managed to have the script process all the mail but i received NONE. Any suggestions? CODE #!/usr/bin/perl read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'}); @pairs = split(/&/, $buffer); foreach $pair (@pairs) { ($name, $value) = split(/=/, $pair); $value =~ tr/+/ /; $value =~ s/%0D%0A/\|/g; $value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg; if ($in{$name}) { $in{$name} = $in{$name}.",".$value; } else { $in{$name} = $value; } } $in{'txt_message'} =~ s/\\|/%0D%0A/g; $in{'txt_message'} =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1 +))/eg; $text_message = $in{'txt_message'}; $in{'html_message'} =~ s/\\|/%0D%0A/g; $in{'html_message'} =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($ +1))/eg; $html_message = $in{'html_message'}; $totalEmails = $in{'rightrange'} - $in{'leftrange'} + 1; $time_start = time() + $in{'time_diff'}3600; $lock = "2"; $pid = fork(); print "Content-type: text/html \n\n fork failed: $!" unless defined $p +id; if ($pid) { print "Content-type: text/html \n\n"; print "<html><head><title>List Administration</title></head><body> <br><br><br><center> Congratulations!<br>The mailing has been started. You will receive + a confirmation e-mail when the mailing has been completed. <P><FORM><INPUT TYPE=BUTTON VALUE=\"Back\" onClick=\"history.go(-1 +)\"></form></center></body></html>"; exit(0); } else { close (STDOUT); $count = 0; open(LIST,"$in{'list_dir'}/$in{'list'}/$in{'list'}.txt"); if ($lock){flock(LIST, $lock);} @addresses=<LIST>; close(LIST); $text_message =~ s/(.{60}\s)/$1\n/g; my @boundaryv = (0..9, 'A'..'F'); srand(time ^ $$); for (my $i = 0; $i++ < 24;) { $boundary .= $boundaryv[rand(@boundaryv)]; } ############# my $htmlhead =<<EOHTMLHEAD; Content-type:text/html\n EOHTMLHEAD my $multihead =<<EOMULTIHEAD; MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Next_Part_$boundary" ------=_Next_Part_$boundary Content-type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable EOMULTIHEAD my $multipart =<<EOMULTIPART; ------=_Next_Part_$boundary Content-type: text/html; charset="base64" Content-Transfer-Encoding: quoted-printable EOMULTIPART ############ for ($count = ($in{'leftrange'} - 1); $count < $in{'rightrange'}; $cou +nt++) { $member = @addresses[$count]; chomp($member); open (MAIL, "\|$in{'mailprog'} -t") \|\| die "Can't open $in{'mailprog'}! +\n"; my $textremove =<<EOTEXTREMOVE; --------------------------------------------------------------------- Text remove... --------------------------------------------------------------------- EOTEXTREMOVE my $htmlremove =<<EOHTMLREMOVE; <BR><BR> ---------------------------------------------------------------------< +br> HTML remove... ---------------------------------------------------------------------< +br><br> EOHTMLREMOVE print MAIL "From: $in{'adminMail'}\n"; print MAIL "To: $member\n"; print MAIL "Subject: $in{'subject'}\n"; my $msg = ''; $msg .= $htmlhead if $in{'type'} eq "html"; $msg .= $texthead if $in{'type'} eq "textpart"; $msg .= $multihead if $in{'type'} eq "multipart"; unless($in{'type'} eq 'html') { $msg .= "$text_message\n\n"; $msg .= $textremove if $in{'remove_notice'} eq "1"; } $msg .= $multipart if $in{'type'} eq "multipart"; unless($in{'type'} eq 'text') { $msg .= "$html_message\n"; $msg .= $htmlremove if $in{'remove_notice'} eq "1"; } print MAIL $msg; if (int($count/100)100 == $count) { $time_now = time() + $in{'time_diff'}3600; open(LOG, ">$in{'list_dir'}/$in{'list'}/log_sent.txt"); if ($lock){flock(LOG, $lock);} $status = $count - $in{'leftrange'} + 1; print LOG "$time_start" . "::" . "$totalEmails" . "::" . "$status" + . "::" . "$time_now"; close(LOG); } close MAIL; } } $time_now = time() + $in{'time_diff'}3600; open(LOG, ">$in{'list_dir'}/$in{'list'}/log_sent.txt"); if ($lock){flock(LOG, $lock);} print LOG "$time_start" . "::" . "$totalEmails" . "::" . "$count" +. "::" . "$time_now" . "::end"; close(LOG); open (MAIL, "\|$in{'mailprog'} -t"); print MAIL "From: $in{'adminMail'}\n"; print MAIL "To: $in{'adminMail'}\n"; print MAIL "Subject: Congratulations!\n\n"; print MAIL "Congratulations!\nThe mailing was successfully sent to + $totalEmails people by $ENV{'REMOTE_ADDR'}.\nMailing started at $tim +e_start and ended at $time_now.\nHere is what was sent:\n\n-----\n\n" +; print MAIL "Type - $in{'type'}\n\n"; print MAIL "Subject: $in{'subject'}\n\n"; if ($in{'type'} eq 'html') { print MAIL "$in{'html_message'}\n\n-----"; } else { print MAIL "$in{'text_message'}\n\n-----"; } close (MAIL); exit; [download] thanks Adrien	[reply] [d/l]
Re: Speeding up a mailing list script by grinder (Bishop) on Dec 12, 2001 at 18:32 UTC
A lot of people have given very sound advice so far; all of which should be heeded. But I am surprised that everyone appears to have missed the most obvious point of all... Who says you have to be online to run this script? So you have to process a bajillion messages? It's not like you have to press `[enter]` between each message. Put your list of subscribers in a text file that is held on the machine. Use your web form only for updated versions of the file. Then write a script (say "send-newsletter"), that sends out the message to each person. When you run it, run it as "send-newsletter &", i.e., run it in the background. Bingo! You script just took two seconds to run. You can log off, and if the silly thing takes six hours to complete, it doesn't matter, you don't have to hang around. There are a number of other things you can do. You can write a logfile to a password-protected URL on your site, to watch the script trundle along. You could also divide the script in two, if disk space is not a problem. Firstly, you write a script to generate all the emails. (Note, placing more than 1000 files in a directory is not a good idea, find some logical way to split them up among multiple directories). This script will probably be pretty fast, short enough for you to (want to) hang around and check the results. When everything is hunky-dory, you run a second script whose sole purpose is to take all those generated email messages and feed them to your local MTA, taking as long as it takes. That part you might want to run in the background and you go away and do something else. This approach is nice, because it gives you one more barrier from doing something tremendously stupid, like accidently mailing the subscribers a love-letter from your significant other. You can give a couple of the messages a final once-over, before committing to sending them out. -- `g r i n d e r just another bofh print@_{sort keys %_},$/if%_=split//,'= & *a?b:e\f/h^h!j+n,o@o;r$s-t%t#u';`	[reply] [d/l]

Speeding up a mailing list script

g r i n d e r just another bofh print@_{sort keys %_},$/if%_=split//,'= & *a?b:e\f/h^h!j+n,o@o;r$s-t%t#u';

`g r i n d e r just another bofh print@_{sort keys %_},$/if%_=split//,'= & *a?b:e\f/h^h!j+n,o@o;r$s-t%t#u';`