Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re: Re: Web servers get hardware upgrade

by hossman (Prior)
on Feb 14, 2004 at 01:20 UTC ( [id://328911]=note: print w/replies, xml ) Need Help??


in reply to Re: Web servers get hardware upgrade
in thread Web servers get hardware upgrade

Interesting.

I've never really considered what hardwware PM runs on, but in one day I find out it's running on a FreeBSD box -- from the same person that posted a link in the chatterbox to info about the performance of MySQL on FreeBSD

Makes me wonder if the MySQL setup provided by pair is running LinuxThreads.

Clarification: I have no opinions or objections to the hardware/OS PM runs on. I was just geniunly curious how our MySQL server was compiled. Based on tye's reply, it looks like they'll be testing out hte LinuxThreads option soon.

Replies are listed 'Best First'.
Re^3: Web servers get hardware upgrade (PM hw)
by tye (Sage) on Feb 14, 2004 at 05:21 UTC

    Actually, PerlMonks runs on three FreeBSD boxes. We have one DB server and two web servers.

    We had a lot of problems when the site was first moved from Linux to FreeBSD. It became clear (to me, anyway) that with our build of MySQL on FreeBSD (and this was back before the problem linked from MySQL on FreeBSD had been identified), a single long-running query could cause lots of other queries to block and that this was causing the site to completely lock up for several seconds or minutes (for example, whenever super search was used).

    This problem was worked on at several fronts. vroom rewrote simple title search and super search to use full-text indexing, then I rewrote them both again. I remember reworking the backups repeatedly to get them to not lock up the site and reworking other maintenance tasks and writing breathe to get them to not lock up the site.

    We all researched and eventually found the first reports of the problem and then possible fixes. ehdonhon rebuilt MySQL as suggested but the site just refused to stay up on it. We investigated switching back to Linux but never managed to get the resources needed for that.

    I patched and recompiled Term::ReadKey so that it no longer was a CPU hog on FreeBSD so that we could use 'mytop' more often to monitor the DB. Then I reworked it and renamed it as sqltop and taught it how to run as a daemon, logging long-running queries and even killing ones that ran *too* long.

    We finally had the site running pretty smoothly (though not exceedingly quickly).

    Then pair.com gave the DB server a huge upgrade (I think the CPU speed quadrupled) and the site was pretty happy for a while... though the traffic volume continued to grow...

    Last January the site was the busiest it had ever been. Traffic actually fell off a bit after that. This January we hit another peak and finally the web servers were having a hard time keeping up.

    Luckily I'd been working on finding problems and fixing them so I had several patches to apply that would eventually help fix the problem and really help quantify how well we were fixing things.

    Unfortunately, trying to deploy these was increasingly difficult because the web servers were often overloaded. I even ended up backing out a big set of behind-the-scenes changes even though my testing showed they increased web server load by less than 5%, because the site became nearly unusable for part of the next day.

    And we started getting the occassional web server needing to be rebooted. So I was trying to put together some data to back me up so I could ask Pair for a third web server without feeling like I was just whining... when Pair said the web servers were due hardware upgrades. Yeah!

    The web servers were certainly getting bogged down some of the time (you could easily notice this when logged into them). The load averages were higher than I expected (but usually not *really* high), but in any given 5-second period the web servers usually had some spare CPU cycles (usually quite a few). And they were only needing a fraction of their memory. My rather wild guess was that the web servers were alternating between everything waiting while MySQL got locked up for just a second or less and then lots of queries completing at once and being CPU starved to process them all.

    In any case, today's doubling of the web server's CPUs from 1GHz to 2GHz is very welcome.

    Fairly soon we should be testing MySQL v4 w/ LinuxThreads which should allow us to make better use of the DB server's power. And I'll be deploying efficiency improvements for quite a while. And demerphq just did a ton of work to make the best/worst nodes lists much more efficient (and other improvements). So the site might be quite happy for another year of growth in popularity.

    Anyway, I wanted fill in a bit about the site's performance history. I hope that was informative.

    - tye        

      Hi tye, thanks for this info. I wonder how much pair.com charges for all of this?

      ------
      This sentence is not true.

        You've been a member for how long? And you've never read that pair Networks so generously provides the machines, bandwidth, etc for free?

        Four months, eh? Should have been plenty of time to see the text at the very bottom of the pages.

      I patched and recompiled Term::ReadKey so that it no longer was a CPU hog on FreeBSD so that we could use 'mytop' more often to monitor the DB. Then I reworked it and renamed it as sqltop and taught it how to run as a daemon, logging long-running queries and even killing ones that ran *too* long.

      Hi, tye -- Unfortunately we couldn't make head or tail of your chatbox tip, "I just disabled the "compatability test", tim (FreeBSD is certainly compatible even though it fails that test)".

      We're running 5.005_03 built for i386-freebsd on FreeBSD 4.9, and mytop-1.4 from freebsd ports. We tried ReadKey-2.21 and ReadKey-2.30, but runing mytop uses loads of system cpu:

      PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMAND 12587 diamondd 65 10 5660K 4608K CPU0 0 0:13 96.87% + 46.34% perl CPU states: 0.0% user, 3.5% nice, 46.5% system, 0.6% in +terrupt, 49.4% idle ktrace just shows 12587 perl CALL read(0,0x814e000,0x1000) 12587 perl RET read -1 errno 35 Resource temporarily unavailabl +e 12587 perl CALL gettimeofday(0xbfbff9d0,0) 12587 perl RET gettimeofday 0 12587 perl CALL read(0,0x814e000,0x1000) 12587 perl RET read -1 errno 35 Resource temporarily unavailable

      Google found this post, but hasn't given us any clues to the patch. If you'd be willing to share it, and allow us (on your behalf, of course) to add it to a patch submitted to update /usr/ports/devel/p5-Term-ReadKey from 2.21, we'd really appreciate it.

      I've you'd like to contact me via email I can be reached at tim.hunter atsign eircom dot net. Thanks!

        Grab Term::ReadKey, unpack it and apply the following patch to ReadKey.xs:

        --- TermReadKey-2.18/ReadKey.xs Sun Feb 10 08:13:35 2002 +++ TermReadKey-2.18tye/ReadKey.xs Tue Mar 19 12:12:36 2002 @@ -1418,14 +1418,14 @@ #endif /* This is for the best, I'm afraid. */ -#if !defined(FCOUNT) +/*#if !defined(FCOUNT) # ifdef Have_select # undef Have_select # endif # ifdef Have_poll # undef Have_poll # endif -#endif +#endif*/

        Then build the module and install as usual.

        - tye        

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://328911]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others scrutinizing the Monastery: (4)
As of 2024-03-19 07:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found