Beefy Boxes and Bandwidth Generously Provided by pair Networks DiBona
"be consistent"
 
PerlMonks  

Re: Web servers get hardware upgrade

by ehdonhon (Curate)
on Feb 13, 2004 at 21:36 UTC ( #328885=note: print w/ replies, xml ) Need Help??


in reply to Web servers get hardware upgrade

Upgrades are finished. Both webservers got upgraded to the present QS3 Specs.


Comment on Re: Web servers get hardware upgrade
Re: Re: Web servers get hardware upgrade
by hossman (Prior) on Feb 14, 2004 at 01:20 UTC

    Interesting.

    I've never really considered what hardwware PM runs on, but in one day I find out it's running on a FreeBSD box -- from the same person that posted a link in the chatterbox to info about the performance of MySQL on FreeBSD

    Makes me wonder if the MySQL setup provided by pair is running LinuxThreads.

    Clarification: I have no opinions or objections to the hardware/OS PM runs on. I was just geniunly curious how our MySQL server was compiled. Based on tye's reply, it looks like they'll be testing out hte LinuxThreads option soon.

      Actually, PerlMonks runs on three FreeBSD boxes. We have one DB server and two web servers.

      We had a lot of problems when the site was first moved from Linux to FreeBSD. It became clear (to me, anyway) that with our build of MySQL on FreeBSD (and this was back before the problem linked from MySQL on FreeBSD had been identified), a single long-running query could cause lots of other queries to block and that this was causing the site to completely lock up for several seconds or minutes (for example, whenever super search was used).

      This problem was worked on at several fronts. vroom rewrote simple title search and super search to use full-text indexing, then I rewrote them both again. I remember reworking the backups repeatedly to get them to not lock up the site and reworking other maintenance tasks and writing breathe to get them to not lock up the site.

      We all researched and eventually found the first reports of the problem and then possible fixes. ehdonhon rebuilt MySQL as suggested but the site just refused to stay up on it. We investigated switching back to Linux but never managed to get the resources needed for that.

      I patched and recompiled Term::ReadKey so that it no longer was a CPU hog on FreeBSD so that we could use 'mytop' more often to monitor the DB. Then I reworked it and renamed it as sqltop and taught it how to run as a daemon, logging long-running queries and even killing ones that ran *too* long.

      We finally had the site running pretty smoothly (though not exceedingly quickly).

      Then pair.com gave the DB server a huge upgrade (I think the CPU speed quadrupled) and the site was pretty happy for a while... though the traffic volume continued to grow...

      Last January the site was the busiest it had ever been. Traffic actually fell off a bit after that. This January we hit another peak and finally the web servers were having a hard time keeping up.

      Luckily I'd been working on finding problems and fixing them so I had several patches to apply that would eventually help fix the problem and really help quantify how well we were fixing things.

      Unfortunately, trying to deploy these was increasingly difficult because the web servers were often overloaded. I even ended up backing out a big set of behind-the-scenes changes even though my testing showed they increased web server load by less than 5%, because the site became nearly unusable for part of the next day.

      And we started getting the occassional web server needing to be rebooted. So I was trying to put together some data to back me up so I could ask Pair for a third web server without feeling like I was just whining... when Pair said the web servers were due hardware upgrades. Yeah!

      The web servers were certainly getting bogged down some of the time (you could easily notice this when logged into them). The load averages were higher than I expected (but usually not *really* high), but in any given 5-second period the web servers usually had some spare CPU cycles (usually quite a few). And they were only needing a fraction of their memory. My rather wild guess was that the web servers were alternating between everything waiting while MySQL got locked up for just a second or less and then lots of queries completing at once and being CPU starved to process them all.

      In any case, today's doubling of the web server's CPUs from 1GHz to 2GHz is very welcome.

      Fairly soon we should be testing MySQL v4 w/ LinuxThreads which should allow us to make better use of the DB server's power. And I'll be deploying efficiency improvements for quite a while. And demerphq just did a ton of work to make the best/worst nodes lists much more efficient (and other improvements). So the site might be quite happy for another year of growth in popularity.

      Anyway, I wanted fill in a bit about the site's performance history. I hope that was informative.

      - tye        

        Hi tye, thanks for this info. I wonder how much pair.com charges for all of this?

        ------
        This sentence is not true.
        I patched and recompiled Term::ReadKey so that it no longer was a CPU hog on FreeBSD so that we could use 'mytop' more often to monitor the DB. Then I reworked it and renamed it as sqltop and taught it how to run as a daemon, logging long-running queries and even killing ones that ran *too* long.

        Hi, tye -- Unfortunately we couldn't make head or tail of your chatbox tip, "I just disabled the "compatability test", tim (FreeBSD is certainly compatible even though it fails that test)".

        We're running 5.005_03 built for i386-freebsd on FreeBSD 4.9, and mytop-1.4 from freebsd ports. We tried ReadKey-2.21 and ReadKey-2.30, but runing mytop uses loads of system cpu:

        PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMAND 12587 diamondd 65 10 5660K 4608K CPU0 0 0:13 96.87% + 46.34% perl CPU states: 0.0% user, 3.5% nice, 46.5% system, 0.6% in +terrupt, 49.4% idle ktrace just shows 12587 perl CALL read(0,0x814e000,0x1000) 12587 perl RET read -1 errno 35 Resource temporarily unavailabl +e 12587 perl CALL gettimeofday(0xbfbff9d0,0) 12587 perl RET gettimeofday 0 12587 perl CALL read(0,0x814e000,0x1000) 12587 perl RET read -1 errno 35 Resource temporarily unavailable

        Google found this post, but hasn't given us any clues to the patch. If you'd be willing to share it, and allow us (on your behalf, of course) to add it to a patch submitted to update /usr/ports/devel/p5-Term-ReadKey from 2.21, we'd really appreciate it.

        I've you'd like to contact me via email I can be reached at tim.hunter atsign eircom dot net. Thanks!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://328885]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (5)
As of 2014-04-21 03:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (490 votes), past polls