Wednesday, October 31, 2007

Evaluations.

I was thinking about the responses to the Dallas Cowboys series of posts, and it really does just come down to whether you believe anyone has the right or the tools to accurately guess what a stranger is capable of.

We can't all be professional profilers, after all.

Personally, I never believed in grades. I frequently got A's without trying, saw people cheat to get A's, and saw people freak out from insecurity like that described in this beautifully written post. It was clear to me from an early age that grades didn't mean much.

So if I have this fundamental belief that the way we assess students is wildly inaccurate except at the extremes of the curve, why the hell would I want to be a professor?

Maybe it's a religious discussion at the base of it, but to me it seems that teaching and learning need to be fluid things, tailored to different learning styles. This also means that assessment by just one method- a test, an essay, a presentation, a CV (!!!)- no one thing is ever enough by itself.

And yet, cuts are made based on one or two criteria. To get into college, you have to have a minimum SAT score... but there are exceptions. If you are an athlete, or a legacy.

You have to have a certain grade point average... but this varies among school districts.

It always drove me nuts that there isn't a better way to standardize grading, so that you don't get large intro courses evaluated based on who is the 'easy' prof and who is the 'hard' prof.

Even for courses where the criteria are ostensibly objective, e.g. anything involved calculations where the final answer is either the right number or it isn't, the questions on the exams can vary in difficulty depending on who is teaching the course.

The same thing drives me nuts in science, but it's even worse, because the perception of science is that it's totally objective, totally fair, all for the greater good, blah blah bullshit blah.

And it's even less true in science than it is in education. And the more cross-disciplinary departments become, the less they seem to grasp that some things are harder than others.

In my field, we almost never publish a result that was only arrived at by one method, and never by only one attempt. Everything is repeated, usually at least 3 independent times, 3 independent ways.

That means for every panel of every multipanel figure I've ever published, I had to do at least 9 experiments.

At least, that's how I do research. It means I don't have as many publications as some people, but it also means I'm pretty damn sure everything I've published is a real, reproducible result.

I know a lot of fields are not like this, for various reasons. Some experiments can only be done once, because of cost and/or time to do a single experiment. Some experiments can only be done one way, because the technology just isn't there (yet).

In some fields the standards are lower and nobody expects error bars at all, much less asks for a paper to be revised to include them.

So to me, to work so hard to get to the truth, and then be judged based on only a tiny sliver of what I really know and can do, is just plain insulting.

Worse than that, though, are the people who seem to think it's not even worth discussing whether the way we choose faculty right now could use some improvements.

There are tons of articles and forums devoted to the ongoing debate about grades, and everyone seems to agree that there are myriad ways of assessing and evaluating student progress. That it's a work in progress.

So why is it verboten to even try to start a discussion of new ways to assess and evaluate faculty candidates?

Amazing to learn what will trigger a knee-jerk, defensive reaction.

I must have found a button.

13 Comments:

At 7:03 AM, Blogger EcoGeoFemme said...

Do you have suggestions for better ways to hire faculty? It seems university admissions generally try to use many factors to assess potential students -- grades, test scores, essays, recommendations -- and that universities similarly try to use many factors to choose new faculty -- cv, research/teaching statements, letters, interview(s), dinners, seminar, chalk talk. I think that is much more thorough than for many other kinds of jobs, where 2-3 interviews of 2 hours or less would be all a candidate would get. Maybe the problem is really that there are too many people competing for jobs. The department I'm in recently had a search where they received 160 applications for one position. How do you think they should narrow it down?

It's not a perfect system and it's good to have discussion about it. My impression is that overall it's a "we know it's not great but we're doing our best" kind of situation. Also, my impression is that people want to do their best to get someone really good because they dont' want to add a bad colleague to their department.

 
At 10:30 AM, Blogger Ms.PhD said...

See recent posts on Dallas Cowboys tryouts and the comment from The Hirerer (and my reply).

 
At 11:03 AM, Anonymous Anonymous said...

So if I have this fundamental belief that the way we assess students is wildly inaccurate except at the extremes of the curve, why the hell would I want to be a professor?

I'm a professor and hate grades too (and hated them as a student, but not as much as now). I view them as a necessary evil. It is possible to teach students to think technically and critically despite the fact that some subset is really just interested in a grade. I find this far easier when teaching elective courses -- required ones are a bit of a nightmare since so many of the students do not want to be there.

 
At 12:30 PM, Anonymous Anonymous said...

Why do you assume that you are the only one who does nine experiments per panel?

 
At 12:36 PM, Blogger Vodalus said...

I think you need to increase the size of your "silly" label. So that maybe people will be more inclined to recognize when you're not entirely sincere.

 
At 6:32 PM, Anonymous Anonymous said...

It is also important to keep in mind that hiring a faculty is a far bigger risk for a university than hiring an analyst for morgan stanley. You are on a tenure track position and the start up funds for such a position at a major research university these days are around 1 million dollars. You also cannot be fired for the next six years or so therefore it is a high risk decision. Like someone above pointed out, nonacademic jobs ask for one page CV and nothing more. Do you think it is an objective way of evaluating a 30 year old person based on a one page information? it seems like you never looked for a nonacademic job. You need to wake up.

 
At 2:37 PM, Anonymous Anonymous said...

Wtf? Where have everyone's critical reading skills gone?

1) [i]Why do you assume that you are the only one who does nine experiments per panel?[/i]

She didn't. She said that she understands why sometimes that isn't a reasonable expectation.

2) Sylow, I have no idea where you're going with your comment. YFS is arguing that because of the high risk involved in hiring someone as a professor, there needs to be more intensive and productive examination. Which is why spending 10 min on each application and eliminating candidates based on titles on a CV is too cursory. So, yes, YSF thinks that the one page judgment is insufficient. And she's repeatedly stated that she's not interested in a non-academic job. So, it seems that your vitriol is unfounded.

 
At 6:12 AM, Anonymous Anonymous said...

I'm in the humanities, but lately I've been talking to a lot of science profs and reading blogs like yours. I have to say, I'm so glad I'm in the humanities! People in the sciences are so much more at the mercy of other people; it sounds like your PI has incredible power over your publication record. In the humanities, papers are single-authored, so if I have no publications, that's (usually) my own fault. Submitted papers are almost always double-blind reviewed, so the reviewers (in theory) don't know that I'm a woman, that I'm at a podunk institution, etc. For hiring, we do a first round of 30 minute interviews at a central conference. The search committee will interview as many people as possible, since the only cost is the time it takes to prepare for and conduct the interview. In the interview we discuss the person's writing sample and dissertation, and one can very quickly get a sense of the person's intellectual abilities. There's still tons of bias in the process (there's a decided preference for hiring people from top grad programs with letters of rec from big names), but it still seems to have advantages over what you've described in the sciences.

I have to disagree with a couple of your points, though: I don't think grading is as subjective as you portray it. When I've normed grades with other people, we almost always agree on a ranking of student essays from best to worst; the only disagreement is where to draw the cut-off lines for the actual letter grades.

Also, you complain about how subjective people's judgments about others' abilities/accomplishments are. But that's going to be a problem in any hiring process for any job. Unless you start your own business, you're going to have to be hired by somebody, and all of us have biases. Some people will really like a candidate, and some won't, for a variety of reasons. Luck plays a big role here, but there's no way to avoid that.

 
At 7:35 AM, Anonymous Anonymous said...

I thought to point out a WORSE model of hiring tenured faculty, from which I ended up benefiting, but I certainly recognize it's pervasive and omnipresent faults.

In France, to get a life-tenured position, you go straight from postdoc to applications at a national level for these positions (usually 2-3 per broad subject). A national peer committee is selected for hiring decisions and renewed every 4-5 years.

Judgment is made based exclusively on CV (where PhD thesis advisor reputation is very important, and impact factors as well) and ONE 15 minute presentation in front of the national peer review board. 15 minutes of questions, then you're out on your nose.

I did it four times and wouldn't have made it in at all if I hadn't chosen a good mentoring group in which to do my postdoc, who helped me get a leg-up Avenir position (sort of start-up funds) which then shoehorned me into one of these tenured positions. But it's not really tenure-track, and the institution bases all decision-making on CV and whether or not you had a cold on the day of your interview and the composition and personal interests of the committee.

Not to say we couldn't do better in North America, but you can do worse.

 
At 4:01 PM, Blogger Ms.PhD said...

Amy,

Thanks for the insightful comparison to humanities. I've been saying for years that our reviews should be double-blind, for instance. And I really do wish we could have PI-less papers.

Although, to make a pun, it's hard to imagine how PIs could do any less to help than they already do.

It's a funny point you bring up about grading. In science, in some ways, it can almost be more subjective.

In an essay you might look for certain qualities. Is it organized and persuasive, do they make their case. I for one never understood at the time why I got an A or a B on an essay in school, but that's because the professors I had didn't want to take the time to tell me how to improve, even when I went to see them specifically to ask.

On long-answer scientific tests, particularly at the graduate level, a question might be something like "What do you think the biggest question is in Field X right now, and how would you propose to solve it?" (I'm paraphrasing from an actual test question).

The problem is, your score has nothing to do with how thoughtfully you answer the question, although you'd think it should. It has more to do with picking the right answer than in how well you defend it.

You're expected to guess what the professor wants to hear, not say what you actually think.

And here's where, in a way, I guess it makes sense. If you're good at reading people and kissing ass, you'll do well on those tests. It doesn't mean you know how to think or that you'll ever be a good scientist, but it does seem to correlate with career success in some arenas.

I guess that never occurred to me when I was younger, that the tests actually do help select for certain critical professional abilities.

It just doesn't do anything good for science. There's a disconnect.

Alethea,

Damn, girl. Four times! But you got it in the end. Very inspiring. I can't quite imagine going through all that.

 
At 2:57 PM, Blogger Unbalanced Reaction said...

Wow, reading through the comments from some of your recent posts...jeez, on button, indeed. I'm currently applying for academic positions (and postdocs as well), so I have resigned myself to the fact that the application packets (which I spent hours preparing) might only receive a few minutes of a search committee member's time. Perhaps the hiring system will change eventually, but it certainly won't be in time for me!

 
At 9:26 AM, Anonymous Anonymous said...

Double-blind reviews!!! Now why DON'T we have that?! I mean, the author page is always separate from the rest. It should be straightforward for the editor to pick it off, or the submission programs to ask you to make a separate cover page.... this is a revelation to me. I'm going to plug it hard. What's the counter-argument?

 
At 11:40 AM, Blogger Ms.PhD said...

Alethea,

Wow, I'm surprised you never heard of that. But, now you know.

The counter-argument usually used is a ridiculous one: that everyone will know who it is anyway based on the topic, writing style & citations used. Which could be true, I guess, if the work is presented frequently at meetings and reviewed by people who attend those meetings. But it seems to me that this argument is dated. Once upon a time, when science was small, sure, everyone knew each other. But science is bigger now and we don't all know each other or attend all the meetings.

Also there is this argument, used especially for funding decisions, that the reputation of the scientist matters. That some people are more qualified than others to make certain statements.

To me this attitude is the opposite of science and needs to go away. But I think I've blogged about that in the past.

 

Post a Comment

<< Home