Saturday, July 14, 2007

Volume 1: Challenges of the NIH system of support

As per my last post, the series begins.

1. Challenges of NIH System of Research Support
Please describe any specific challenges presented by NIH’s support of biomedical and behavioral research such as the current array of grant mechanisms, number of grants awarded per investigator, and the duration of grants."


Let me just describe my perception of this, since I'm only an NIH employee in the most indirect sense. Perhaps it will be useful to some to get the perspective of how much a random postdoc presumes to know about how these things work. Hopefully anything I get horribly wrong will be corrected by alert readers.

Hmm. Specific challenges. Let's see.

The overall goals of the NIH aren't really clear, but I presume one needs to know what those are to know what challenges are inherent in reaching those goals. I guess I always assumed that NIH has two goals:

(1) find treatments for current patients
(2) research future treatments and diagnostics for things we can't catch early enough/don't know how to treat right now

So the challenges of that are to fund both:

a) practical, applications-oriented clinical trials
b) riskier, investigative studies

That's already a tall order.

Problems inherent in all of this:

- A wide variety of research areas requires lots of 'experts'
- Really expensive to fund all this (though only about as much as 1 day of the Iraq war)

I think my favorite image of NIH is the one of the grants room where your printed grant application goes when you mail it in. An RO1 (not that I've written one myself) is a very tall stack when all the copies are put together, so there is literally a room at the NIH stacked wall-to-wall, to the ceiling, with paper.

Of course I'm hearing lately that the new online system is actually more time consuming and disaster-prone than the paper version, so there are inherent challenges just in the basic execution of the steps:

Step 1: receive grants
Step 2: sort grants
Step 3: don't lose anybody's grant or pieces thereof (have heard plenty of horror stories about these things happening with alarming frequency).

Obviously my big complaints, at this point in my career, about the current array of grant mechanisms has largely to do with two things:

1. Why are we funding companies, who can charge money for their products, and foreign labs, who have their own taxpayers, with American taxpayer money when we can't even afford to fund all our American academic scientists?

2. Oh yeah, and while I'm asking, why do we have to pay taxes on our pittance government fellowships in grad school and during postdocs? Do any voting laypeople realize how stupid that is??

3. Why aren't there genuinely independent grants available for young scientists? Why do I have to get bullshit recommendation letters just to get my research proposal reviewed??

As for number of grants awarded per investigator, this is a major point of contention between the Haves and the Have-Nots.

If you Have a lot of grants, you don't see why you shouldn't be allowed to Get More.
If you Have Not a grant, you don't see why the same people get all of them.

From a totally obective, scientificky point of view, you'd think that if grant reviews were anonymous, that would be best.

Then quality would matter most, and if your new grants are good, you'd get them funded, regardless of whether you already have too many or if you've never had one before.

Okay, fine. But. In reality, I'm told, most sub-specialities are so small that grant review sections usually can tell whose grant is whose, even if the names were blacked out, they would know who they were from.

So I think the central challenge of not just the NIH, but science in general, is: CORRUPTION.

There, I said it.

We'd all like the think that science is this inherently noble profession, that we're all honest and just want to get the right answer.

But thanks to the competition for funding and jobs, the NIH suffers from a general mafioso atmosphere.

That is to say, you pay Tony Soprano, and he takes care of you.

I'm not saying there's literally cash changing hands, but there are fancy dinners with lots of wine, and there's definitely an "I'll scratch your back you scratch mine" attitude.

Also true for publishing, of course.

Most people I know have gotten a grant funded (paper published) at one time or another because they knew someone on the committee (one of the reviewers), and vice-versa. Sometimes the same grant (paper) gets a crappy score in one study section, but gets high marks at another one, just because of who is on the committee (reviewers).

That's clearly not fair or objective or scientific at all!

But what are you going to do, have a computer review the grants? That won't happen anytime soon. So far as I know, machine learning hasn't come far enough to parse language and decipher logic at that level.

I still argue, as I always have, that grants should depend on one thing, and one thing only: the work proposed, and how well thought out that is.

It shouldn't matter what you've done before.

You might have gotten a Nobel prize: I DON'T CARE.

You should still have to do the same work as everybody else to prove that your ideas are supported by preliminary data.

Oh, and inherent in the funding challenge is that chicken and egg problem: how do people get preliminary data, if they don't have funding yet?

Simple: they get someone else to pay for it (for example, senior postdocs trying to get preliminary data for when they start their own labs), or they pay for it off one of their other grants.

Nobody really cares how you spend your grant money, so this is totally unregulated as far as I can tell. I know one PI who never, ever works on what she's funded to do, and yet she always gets more grants.

Why is that okay?

I have a friend, a cancer survivor, who asks me things like this a lot. She's an educated taxpayer, so she wants to know things like this. Why, she says, are we funding these people?

I guess this funnels nicely into the question of how long grants should last. Is it better to have short grants and reapply every year or two? Or these long career grants for 5-10 years.

I can firmly say with some conviction that funding anyone for 10 years is BAD. NIH stopped doing this, so far as I know, maybe because they figured this out for themselves the hard way.

Five years is maybe a little too long. I think 3-4 years is probably enough to keep the momentum going. Right now most PIs I know tend to wait around, thinking they've got 5 years, and that's a long time, right?

...And then end up scrambling the last few months to scrounge up enough data- I mean, hound their grad students, techs and postdocs to do last minute one-off, potentially unreproducible experiments just for grant figures.

Do people really do that, some of you might be asking?

YES, a resounding YES.

One of my favorite anecdotes is about the guy who crystallized a protein and put the pictures of the crystals in the grant, and got it funded, only to figure out later that he had purified a contaminant in his protein preps.

But hey, he got the grant, right? And he's under no obligation to tell NIH about his mistake, and no one will check up on him.

So preliminary data doesn't have to be right, it just has to look good.

Which is lame, really, but I guess I would rather that people at least try to come up with preliminary data, than not have any... except for the problem of how to pay for it.

Let me close by saying something I think I've said before, but I'll say it again:

I think writing grants, in principle, is a really useful and- dare I say it?- fun
part of the scientific enterprise, if only because it really is one of the only times we're given license- nay, we're paid!- to really think about what we're going to do next and what our current stock of data really imply.

But the room with paper stacked to the ceiling? The broken online submission systems? They need to fix those ASAP.

Labels:

10 Comments:

At 10:11 AM, Anonymous Drugmonkey said...

Thanks for playing! (and make sure to make some comment at the NIH site)

I take a little bit of issue with the "only got a grant funded because they knew someone on the committee" thing. I know this is a common suspicion. And I often discuss the political / schmoozy aspects of NIH-grant careerism.

But let us be clear. A single reviewer cannot single handedly get a grant funded. Period. A single reviewer can likely torpedo a grant but cannot "fund one". The grant has to be at least decent and you have to get the other reviewers and ultimately the panel on board. Now, can a single reviewer make the difference in a very-good grant getting that last push over the line? sure. But unless you are there in that same round, reading the same grants you really do not know that there was some illegitimate back-scratching at the root of the decision. and how do you draw the line? I've had very clear support on grant review from people who know me and/or the lab pretty well. but also from people before they know who I am. how would one know what is a biased review?

scores go all over the place even in the same study section with the same reviewers, btw. Even with very little substantive revision. There are lots of reasons for this that are perfectly above-board. (Even if I criticize many of them, it is not on the basis of corruption, as you put it) In other words this is not evidence of back-scratching in and of itself.

 
At 3:06 PM, Anonymous Anonymous said...

A lot of otherwise very intelligent people seem confused about why graduate students and postdocs pay taxes - after all, it seems silly to hand us money and then take some of it back, right? And if there were no taxes, we'd have more money, right? But you haven't thought it through all the way.

You would be worse off if you didn't have to pay taxes
1) You would receive less money upfront. A lot of people think that if there were no taxes, we'd get more money - but stipends are supposed to offset your living expenses, no make you rich. The first thing that would happen is that stipends would be cut by 10-20%.
2) No deductions. So, the tax code is used as a policy instrument that promotes certain behavior. Thus, there are tax deductions for student loans, mortgages, certain types of cars, health care expenses, children, and providing housing for Katrina survivors. If you're not being taxed, you don't get to take deductions - where as now everyone gets more money upfront and gets taxed differently depending upon their personal situation. Tax credits (ie, a "refund") could theoretically be used instead - but the public doesn't like tax credits (see the constantly attacked EITC, as an example), but everyone likes deductions.
3) It's almost impossible to compare the earning potential of two seperate jobs if one is taxed and the other isn't, ie should you take the NIH postdoc at 30k/yr, or the HHMI postdoc (taxed) at 36k/yr? Does your answer change if you are married to a law student who is earning no income but is taking out student loans? (answer: probably yes)

No political support for an untaxed class
Broadly speaking, there are people who support taxes for societal good (A) and those who oppose all taxation (B). Type B people support taxing everyone, because everyone thus feels the pain of taxes, and will seek to keep taxes as low as possible - while having one class of people untaxed creates a voting block of those who will benefit from increased taxes at no personal cost. Type A people probably want a progressive tax, but want people to buy into the system early - people are more likely to accept taxes as their earnings increase if they always have paid something, even if it was small.

You would probably have to fill out taxes anyway
People hate filling out taxes! But untaxed stipends would almost certainly have to be reported. Married couples would still have to fill out taxes. People with bank interest income would have to fill out taxes. Any tax credits? Yup, got to fill out the taxes. In the end, it wouldn't save much paperwork or effort anyway.

 
At 4:30 AM, Anonymous Anonymous said...

I'm not sure I agree that it shouldn't matter what you've done - your track record is important in assessing whether or not you are likely to deliver the promised results. However, as you also point out, this relies on people checking up on previous grants and their output. It's nice to have good ideas but if you never deliver then you should be blacklisted, just as you would in a commercial environment. And there also has to be some mechanism for young scientists with great ideas, but not much of a track record, to get funded.

 
At 6:34 AM, Blogger Jenny F. Scientist said...

But most grad students defer loans: no deductions. Mortgages: that I should be so lucky to have a mortgage.

Besides, anon may misunderstand how deductions work. Say my stipend is 36K and I pay 7K in taxes but get 2K in deductions. I still have to actually pay the 5K left over. If you're not being taxed, you don't get to subtract money from what you owe because you don't owe.

Untaxed persons may still vote and petition their representatives.

I have no personal opinion on stipends being taxed, but I'd like to point out that most stipends- at least decent ones- cover much beyond the 'basic cost of living.' We in fact have IRAs and investments. Our cost of living is about half our stipends.

"It shouldn't matter what you've done before." This made me laugh- yesterday my advisor said of a paper "Well, but they know what they're doing [because they're famous]." And I thought: but it's STILL A BAD PAPER!!!

 
At 8:53 AM, Anonymous Anonymous said...

You. Know. Nothing.

Thanks, Anon, for giving YFS a lesson in tax policy. She seems to need it. From her other commentary, she is clearly a liberal (I am to, just to be clear), probably supports all kinds of taxes to fund all kinds of things (I do too, just to be clear), but doesn't want to pay them (I DO want to pay taxes, just to be clear. I want to pay MORE taxes, so the NIH budget increases, so elections can be publicly financed, so NO ONE is ever denied health care, so ports and chemical plants can be secured, and so NTSB can hire intelligent security personnel at air ports, not dumb f*cks who could care less about their job, going through old ladies underwear to see if there is some shampoo in a 4.2 oz bottle.)

YFS, by your own admission, you've never sat on Study Section. How can you POSSIBLY know what happens there?

Also, think of this. Someone has three RO1's. They continue to be productive and then apply for renewal. So they have three RO1's, which means they have three RO1's worth of employees. Students, techs, and postdocs, perhaps like yourself. Denying them a renewal, just because, well, they have three and someone else, who may actually suck, doesn't have any, well, then Prof. Three-RO1's would have to fire......her......postdoc.......

Not that easy, is it?

It's always someone else's fault. The system. Tony Soprano. The NIH. Peer review. Corruption.

Well, go ahead and persist in these ridiculous notions about how the system is stacked against YOU. That pretty much ensures that you will go nowhere.

On the other hand, if you figure out how it ACTUALLY works, you might have a chance.

 
At 8:41 AM, Blogger Ms.PhD said...

"YFS, by your own admission, you've never sat on Study Section. How can you POSSIBLY know what happens there? "

This is my point. Why DON'T we know? Why isn't it a transparent process? Why aren't we taught, during grad school and postdoc, how this ACTUALLY works? We're left to guess, from anecdotes, etc. It's not as though study section meetings are videotaped and made public. Or that even meeting minutes are available. Why NOT?

"Denying them a renewal, just because, well, they have three and someone else, who may actually suck, doesn't have any, well, then Prof. Three-RO1's would have to fire......her......postdoc......."

This is exactly my point also. Someone with 3 ROIs should have enough rudimentary management skills not to OVERSPEND or plan beyond what, and how many people, she can actually fund in the long-term. No one should have such a big lab. And having worked in big labs, I think they're a recipe for disaster.

Also having been in labs that lost funding, I did not blame the study sections. The grants my PIs wrote SUCKED. They did NOT deserve the funding, and nobody was doing me, or any other grad students and postdocs, any favors by keeping those labs in business. I would have been much better off working with people who could manage their resources, publish papers regularly, write good grants consistently, and PLAN AHEAD for contingencies.

I've said it before, and I'll say it again: right now, the system selects for fuckups. I'd like to think it's getting better, slowly, but we really need a sea change.

So yes, I think there is corruption. But I'm mostly talking about people getting grants when they shouldn't, so that really good grants aren't funded because there isn't enough money leftover.

I don't know of many examples where a grant was truly outstanding and just not funded because somebody had a vendetta against the author, although I'm sure they exist I suspect these things are not that common.

But it's very easy to not fund some grants by bumping someone else to the top of the list despite a crappy score. All these things factor into who gets funding and who does not.

 
At 1:31 PM, Anonymous Anonymous said...

"It's not as though study section meetings are videotaped and made public. Or that even meeting minutes are available. Why NOT?"

They are not videotaped because then everyone would see us take money from the PI's who want their grant funded, and sit around and make up bogus excuses for not funding grants from, say youngfemalescientists.

Alternatively, they are not videotaped because we are talking about people and their preliminary results and this information is generally, in polite society, considered confidential. Grant panel members generally treat access to this information as a privilege that carries with it great responsibility. Yes, yes, we all know there are exceptions, but those represent the minority of cases, by far.

It is actually a very transparent process, at least at NIH. One gets back detailed comments and a list of the reviewers on the study section. The SRAs even include their phone numbers. You can call them as ask what happened. They can tell you.

Provided you get Tony Soprano to rough them up or pay them off beforehand.

"So yes, I think there is corruption. But I'm mostly talking about people getting grants when they shouldn't, so that really good grants aren't funded because there isn't enough money leftover. "

And you would be the authority on "when they shouldn't"?

"But it's very easy to not fund some grants by bumping someone else to the top of the list despite a crappy score"

In WHAT universe? It's hard to get a great these days with a GOOD score. No one gets a grant with a crappy score.

"No one should have such a big lab. And having worked in big labs, I think they're a recipe for disaster. I've said it before, and I'll say it again: right now, the system selects for fuckups. I'd like to think it's getting better, slowly, but we really need a sea change. "

Right, right. It's all about you and your story. No one else's story matters.

It's not a perfect world, but we somehow manage to get by, discovering things, curing cancer, making new drugs, figuring out how cells work, what proteins might look like, and how plants grow and how animals develop.

All those "fuckups" that "the system selects for" (what the hell is the system, anyway?) must be doing something right.

 
At 3:13 PM, Blogger Ms.PhD said...

To the point about not sharing preliminary data in "polite company"?

Do you realize how ridiculous that is, as an explanation?

Imagine you're a layperson cancer patient, sitting at home wondering why you're sick and these brilliant scientists still don't have treatments that make any difference whatsoever, much less ways to diagnose the problem earlier before you start to feel like roadkill.

Do YOU, layperson taxpayer with cancer, care about whether it's "polite" to share preliminary data? Why should you? Because some self-centered idiot is afraid of getting scooped?

I don't think so. If I were a layperson taxpayer with cancer, I would say we need to get rid of this nonsense, because I don't want my taxes paying for a system that is clearly not working as well as it should.

"In WHAT universe? It's hard to get a great these days with a GOOD score. No one gets a grant with a crappy score."

Ok, point taken.

But right now, lots of grants get - and deserve- equivalent scores, all of them good.

So how do they decide who gets the money when that happens? POLITICS.

You're living in a dreamworld if you think the written comments from your study section tell the whole story of what went on in that room, or that your SRA is allowed to tell you, in so many words, what you should do differently next time.

"Right, right. It's all about you and your story. No one else's story matters. "

Well, it is my blog, after all.

"It's not a perfect world, but we somehow manage to get by, discovering things, curing cancer"

We've cured cancer?? Which cancer? All of them? Where have I been?

 
At 7:46 PM, Anonymous Anonymous said...

Sorry, but I make it my policy to only enter into discussions with reasonable people. Temporary lapse of judgement here. Good bye.

 
At 11:45 AM, Anonymous Anonymous said...

Boo hoo to the last Anonymous comment. BTW, no one is going to miss your great, anonymous intellectual contribution to the site, even after you decided to make a public announcement.

 

Post a Comment

Links to this post:

Create a Link

<< Home