A few (somewhat random) thoughts come to mind.
This site contains a number of articles from one point of view. Sometimes he makes sense, sometimes he doesn't. Still, he at least offers something to think about. (You'll want to scroll down to the section called "Growing a Team" to get to some of the initally relevant articles.)
Having worked for consultants, corporations, federal agencies, and companies in-between, I've seen a number of non-technical people try to judge my competence with a given tool or technology. Those that I've respected most have:
- had a clear idea of the type of contribution they're looking for, e.g. the roles the candidate will fill (and by that, I mean something other than "code monkey"
- had a clear idea of the type of person they want to hire (e.g. personality, style, and so on)
- understood the type of team they want this person to help develop, including its operating sytle and how it will help the organization materially
- been able to discuss the specific types of projects this person will help develop.
- tested programming skills using things they do understand. For example, one interviewer did not know the language I was being hired to use, so asked me to describe how I might solve a particular problem using terms from the language he used. (Which in itself was a challenge because I wasn't that familiar with his language)
understood that BS is a universal language and been able to recognize it
Keep in mind that just because four out of five users recommend a consultant only means that he may know what he's talking about, not that he really does. Consider, for example, the global popularity of a certain set of free CGI scripts versus their reputation in these halls.
Consultants do "get around," but they frequently don't get deep. Before you hire a consultant, ask for references and previous clients. Contact the IT/Support people of those clients and find out their take on the consultant.
I've seen (and worked for) too many consultants who were able to schmooze senior management quite well (and get endorsements) while the people who had to support their work were snickering up their sleeves (or openly kvetching) while quietly rewriting and/or replacing the consultant's code.
Along the same lines, you know your business the best. You also know the types of people that will work best with your organization. Trust those instincts as well.
If you have a colleague that works for another company that uses similar technology, see if they'd be willing to help you develop the interview questions and sit in on the first few rounds, just to help you get up to speed quickly.
Offer to take them to lunch, pay them a small stipend, or whatever.
Above all else, look for: Openness, Honesty, Listening, Learning, Critical Thought, Valid Evaluation of Weaknesses, Problem Solving Abilities, and the ability to say "I don't know, but I can find out." (Note on the latter: Follow through on that. Test it.)
Hollywood frequently hires people based on their "paper" qualities (looks, etc). It doesn't always lead to a good film, does it? Personally, I prefer organizations that look for natural talent and learning ability...not just the highest number of checked items on a skills list.
Finally, follow-through on the hiring process. If you see problems, communicate them as quickly as possible--well before the end of your probationary period. Problems are easily fixed when they're small. Misunderstandings are most easily corrected before they coagulate into Opinions and Assumptions.
Before you "involve" anyone technical, go out and buy an introductory book on the technology you're hiring. Work through the examples and give yourself enough of a background to be able to detect a flagrant liar from someone who really knows what they're doing.
In the end, remember that you're hiring this person to work with you and to collaborate on the success of your organization. Look for people looking for (and willing to give) honest contributions.
Skills can be taught. Initiative, talent, and passion can't.
--f
P.S: Post 500. Whee! (Thanks, y'all.)
