It has been eleven years since I was last on the market for an academic job, and about six years (if I'm remembering correctly) since I was last on a search committee working to fill a tenure-track position in my department. Among other things, this means that I can consider the recent discussion of "conference interviews" at The Philosophy Smoker with something approaching "distance".
However, as I'm well aware, distance is not the same as objectivity, and anyway objectivity is not the kind of thing you can achieve solo, so I'm going to do a little thinking
out loud on the screen in the hopes that you all may chime in.
The nub of the issue is how search committees in philosophy (and in at least some other academic disciplines) use preliminary interviews (typically 30 to 60 minutes in length) to winnow their "best" applicants for a position (as judged on the basis of writing samples, publication records, letters of recommendation, transcripts, teaching evaluations, and other written materials) down to the finalists, the number of which must be small enough that you can reasonably afford to bring them out for campus interviews.
The winnowing down is crucial. From more than a hundred applications, a search committee can usually reach some substantial agreement on maybe twenty candidates whose application materials suggest the right combination of skills (in teaching and research, and maybe also skills that will be helpful in "service" to the department, the institution, and the academic discipline) and "fit" with the needs of the department (as far as teaching, advising students, and also creating a vibrant community in which colleagues have the potential for fruitful collaborations close at hand).
But even if we could afford to fly out 15 or 20 candidates for campus interviews (which typically run a day or two, which means we'd also be paying for food and lodging for the candidates), it would literally break our semester to interview so many. These interviews, after all, include seminars in which the candidates make a research presentation, teaching demonstrations (hosted in one of our existing classes, with actual students in attendance as well as search committee members observing), meetings with individual faculty members, meetings with deans, and a long interview with the whole search committee. This is hard enough to squeeze into your semester with only five candidates.
So, the standard procedure has been to conduct preliminary interviews of shorter duration with the 20 or so candidates who make the first cut at the Eastern Division Meeting of the American Philosophical Association. For departments like mine, these interviews happen at a table in a ballroom designated for this purpose. Departments that have a bit more money will rent a suite at the conference hotel and conduct the interviews there, with a bit less background noise.
Job candidates pretty much hate this set up. The conference falls during winter holidays (December 26-30 or so), which means travel is more expensive than it might be some other time of year. Search committees sometimes don't decide who they want to interview at the convention until quite late in the game, which means candidates may not hear that a department would like to interview them until maybe a week before the conference starts (boosting the price of those plane tickets even more, or making you gamble by buying a plane ticket in advance of having any interviews scheduled). Even at conference rates, the hotel rooms are expensive. Occasionally, winter storms create problems for candidates and search committee members try to get to, or to flee from, the conference. Flu season piles on.
Search committee members are not wild about the logistics of traveling to the convention for the interviews, either. However, they feel like the conference interviews provide vital information in working out which of the top 20 or so candidates are the most likely to "fit" what the department wants and needs.
But this impression is precisely what is in question.
It has been pointed out (e.g., by Gilbert Harman, referencing research in social psychology) that interviews of the sort philosophy search committees use to winnow down the field add noise to the decision process rather than introducing reliable information beyond what is available in other application materials. This is not to say that search committees don't believe that their 30 or 60 minutes talking with candidates tells them something useful. But this belief, however strong, is unwarranted. The search committee might as well push itself to identify the top five candidates on the basis of the application materials alone, or, if that's not possible, randomly pick five of the top twenty for campus interviews.*
Of course, search committees seem not to be in a great hurry to abandon conference interviews, at least in philosophy. My (brief) experience on the scientific job market didn't include conference job interviews per se, but I did have preliminary interviews of very much the same nature and duration with some private sector companies and national labs -- which is to say, I don't think it's just philosophers who are making hiring decisions that are at least partially grounded on a type of information we have reason to believe could be misleading.
The question, of course, is what to do about all this.
Search committees could abandon these preliminary interviews altogether. That would surely put more pressure on the written components of the applications, some of which might themselves be misleading in interesting ways. I'm guessing search committees would resist this, since they believe (although mistakenly, if the research is right) that they really are learning something important from them. It's not obvious to me that job candidates would unanimously endorse this either (since some see the interview as a chance to make their case more vividly -- but again, maybe what they're making is pseudo-evidence for their case).
Search committees could work to structure preliminary interviews so that they provide more reliable information (as the research suggests properly structured interviews actually do).** This would require search committee members to learn how properly to conduct such interviews (and how properly to record them for later examination and evaluation). Moreover, it would require that search committee members do something like acknowledging that their instincts about how to conduct free-flowing, open-ended preliminary interviews that are also informative are probably just wrong. This is a task with a difficulty level that's probably right around what it takes to get science faculty to acknowledge that having learned a lot about their field might not be sufficient to be able to teach it effectively, and that science education research might be a useful source of empirically grounded pedagogical insight. In other words, I think it would be really hard.
Search committees could keep conducting preliminary interviews as they always have. Inertia can be powerful, as can the feeling that you really are learning something from the interviews. However, it seems like a search committee would have to take into account the claim that, empirically, interviews are misleading when drawing conclusions on the basis of preliminary interviews. (Of course this is a normative claim -- the search committees ought to take this worry into account -- rather than a claim that mere exposure to a research finding would be enough to remove the search committee's collective powers of self-delusion.)
Or ... search committees could do something else?
What else could they do here? How do those of you in scientific fields handle the role of interviewing in hiring? Specifically, do you take concrete measures to ensure that interviews don't introduce noise into hiring decisions? Or do you feel that the hiring decisions you need to make admit of sufficiently objective information that this just isn't a problem for you?
If you prefer to comment pseudonymously for this discussion, feel free, but one pseudonym to a customer please.
* For all I know, campus interviews may introduce some of the same kinds of noise to the decision-making process as conference interviews do. However, many include teaching demonstrations with a sample from the actual student population the candidate would be asked to teach if hired, a formal presentation of the candidate's research (including responding to questions about it), and ample opportunity for members of the hiring department to get a sense of whether the candidate is someone with whom one could interact productively or instead someone who might drive one up a wall.
** It is worth noting that some search committees, even in philosophy departments, actually do conduct structured interviews.