My work had an account with Brainbench, as we were required to take two tests. A rather large number of people at work failed the tests. Some of them even failed the same test three times or more.
I was bored, and as I had about 6 weeks before our contract with Brainbench ran out, I just started taking random tests. Now, I will admit, that I failed one test in taking it the first time (Software Business Analysis, which I passed the second time), but either I test rather well, I'm a whole lot smarter than I think I am, or the tests are way too easy. (eg, I passed the certification for CheckPoint FireWall 1, a program which I have never used, and I've spent at most 6 hrs in my whole life looking over someone else's shoulder who was configuring it.
Other fields, which I've done for years, had some absolutely horrible questions that were irrelevent, or had no good answers. For instance, here was the e-mail that I sent Brainbench after taking the Web Server Administrator test, and only getting a 4.21 on it: (yes, yes, there's a whole lot of typos, but it's accurate as what I sent, and Perl is mentioned twice)
Date: Wed, 7 Apr 2004 12:04:44 -0400 (EDT) From: Joseph Hourcle <oneiros@...> To: firstname.lastname@example.org Subject: The Web Server Administation Test I was recently required to take a few tests from your company, due to my job, and I thought I'd take a few other tests that were recommended, but not required. If you will check my records (transcript 4935509), you will see that I have taken a number of tests offered by your company, so I do have a grasp of the overall high quality of the tests being offered. I just wanted to let you know that I found the Web Server Administration Test to be possibly one of the most obnoxious tests I have ever taken in my life. I expected the questions to be on general concepts related to webserver administration, not those specific to particular programs. The questions given were rambling to the point of incoherency, and some of the questions were completely irrelevent to webserver administration. It doesn't come as a surpise to me that I scored as low as I did, and was still ranked in the 98th percentile. A large number of the questions seemed to have some parameters that were not mentioned, that would have significantly affected the answers. Eg, for security, what ports should I bind TCP/IP to? You didn't mention what the network topology was. How was I to know if TCP/IP was needed on the backend? I assumed this was a question for some Microsoft OS, as most UNIX servers would have used TCP/IP for their backend database communications, but you never stated if that was the case or not. There was a question about what was wrong with a perl script that was moved between systems, and why it didn't execute. Of course, you didn't mention what the error logs said. It might not have had permission to execute (it said the directory did, but not the file...or is this one of those MS specific things?). They might have different paths to the perl interpreter, and need the shebang line edited. It might be missing 'require'd or 'use'd perl modules. It might've been transfered from a DOS to UNIX system using binary FTP, and see 'perl^M' as the interpreter. You mentioned that the log files had a high 'percent redirector' or something like that. My log files are in common, or common+. I've never heard of 'Redirector', aside from squid, and none of the questions seemed relevent. I made a stab in the dark that it's an IIS thing, as I know they redirect if they get too much load, but I don't know what they use to determine 'too much'. You asked about the problem with giving users CGI access, but you didn't state if the processes were wrapped, or if the server were chrooted, which would have affected the answers. You asked something to the affect of 'what is the weakest part of a password' or some such, but you didn't mention to whom. I consider the users and transmission to both be problems. And I consider transmission to be weaker, as problems in communication can give up more than just one user's credentials. However, if I'm using SSL, that's not the problem, the user is. Many of the questions did not directly relate to webserver administration. There were two questions about SMTP, for god's sake. What's required before you send mail? What does (x) command mean? That's great if you're a mail administrator, or programming web services that might require them to send mail, but they have absolutely nothing to do with webserver administration. Am I going to see a question in mail administration asking how a caching proxy should handle POST, as compared to GET? Or what are some of the difference between HTTP/0.9 and HTTP/1.0 are? I know I didn't see any of them in this test, so I'm assuming they're elsewhere. If I were a network administrator, I might care what I needed to correctly bridge NETBUI, TCP/IP and IPX networks, but I'm not, and I don't claim to be. I have people I can fall back on for that, and we work well together, as I can keep my systems tuned, and they give me a place to plug into. Some of the questions had _no_ technically correct answer. You asked a question about a line that was a SONET B-ISBN connection with 23 channels, and was 30% used, and was expecting to double in a year... so what was the total bandwidth? Ignoring the fact that this is a networking question, and knowing what the bandwidth of 23B channel SONET connection is completely unrelated to web server administration, all of the answers were in KB. First off, I used to work for an ISP, so I infered that you're talking about a PRI, so it's 64Kb per channel, per second. Unfortunately, all of the answers were the wrong order of magnitude for KB, and they weren't KBps or Kbps. The way the question was worded, and given the unit for the answers, it would have to be the total information transfered in the year... but nothing was even close to the right scale for that. You asked some question about security thay mentioned 'ensure'ing something is safe. That goes against security. You can prevent, but you can never ensure, unless you're protecting against a fixed vulnerability. Oh...and 'PERL' is not a language. 'Perl' is. Many of the questions were overly specific. One of the questions about problems with disk usage gave a response that mentioned using RAID 5 because the disk usage was high. It did say that CPU was low, but why arbitrarily use 5? Actually, this probably falls to not enough information, as it didn't say that the problem was reads or writes, but none of the other answers fixed the disk problems, if I recall. If it's all read traffic, mirrored data, switching to SCSI, or using something with larger disk cache would have been useful. You might have also retuned the OS to better cache disk activity, better distributed your directories on the disks, seperated competing data repositories, or any number of things. For all I know, the issue was with the log files not being on an isolated disk. I was okay with the tuning questions. I was okay with the questions about ISAPI, DOM, and the like, even though I've never used them. I was okay with the questions about web-proxies. But asking questions specifically about a particular webserver is just lame. Asking questions that are networking, operating systems, or programming related are outside the scope of what I consider to be 'web server administration'. It's one thing to ask a question about traceroute or how to debug if the problem isn't yours, and it's someone elses (eg, if it's a DNS problem, and the IP works). When I took the security test, and the general UNIX administration, I thought there were maybe 2-3 questions in each one that were ambiguous. This time around, I started keeping count, and I found over half of the questions to be ambiguous, not have a correct answer listed, or were off topic. I was particulary annoyed by the overuse of the word 'ensure' in questions. Every system is unique. We can't define anythingin absolutely terms based on a sentence or two of description. We chould choose the answer that 'might help' or 'should best solve the problem', but it's just plain ignorant to assume that 5 seconds of diagnosis is going to fix a problem, or that there is one golden solution that will fix all problems. Please don't claim this test is all-encompasing for skills needed for webserver administrators. It barely scratches the surface is some areas, while completely missing in others. It has poorly worded and constructed scenarios. Personally, I thought it was an offense to even consider this to be a test for 'web server administrators'. ----- Joe Hourcle
Oh -- and for the record, I'm Brainbench certified in 29 job roles ... all because I got bored last year. (and I have no idea why 'systems architect' isn't marked as 'job role' like the others)
So, is there a point here? Well, I'll admit, that I'm for testing, as a form of accredidation, but I think it's a complex system that needs much more than just taking a test. I don't think it should be something that should be in the hands of a company whose main interest is people paying to take the test (like Brainbench, or Princeton Review). I'm interested mainly in licensing, or a union, or guild, or something similar where you can check to see if a person has any substantiated grievences filed against them. (In a way, it's a Better Business Bureau for people, but I'd prefer something more guild-like where they're interested in improving the overall quality of their members, not just tracking who has complaints)
Update: The Brainbench Perl test involved a whole lot of 'what would this script output' questions, which basically meant copy and pasting it into your shell, or for those times when they decided to present it as an image, it was a test of your typing speed. I'd also wonder how they're qualified to give a test on HTML and 'Web Design for Accessibility' when they don't put alt tags on all images.