Sunday, April 17, 2011

Trust but verify: side-effects of a life in science

I had an interesting experience about a week ago while hanging out with a new friend.

We were buying beer and pooling our pocket money to give as a present for the host of a going-away party, and didn't have anything to wrap it in. We wanted to be classy and get an envelope.

The scenario basically went like this. We walked into a small liquor store and bought beer. Then we went up to the counter to pay.

Friend: Do you have any envelopes? Or plain white paper?

Cashier: No.

Friend: Okay, we'll just take the beer.


Then we went and looked around the store again, to see if there were any office supplies we had somehow missed. We couldn't find anything.

Then my friend did the unthinkable.


Friend: Are you sure you don't have any plain white paper? Just one sheet back there behind the counter?

Cashier: Oh, yes. Here you go.

I was dumbfounded. Essentially the same question, asked twice, yielded opposite answers.


(If this happens in science, you go home and drink whiskey and come back the next day and try it 8 more times and take careful notes and try to figure out why you got one answer 5 times and the opposite answer 3 times, and then you modify your protocol and do it another 8 times and if you get the same answer 7 times, you publish it and say +/- 1. And write a long discussion section explaining what you think the variables are.)


In this case, I would never have thought to ask again. I spent the whole week thinking about it.



It reminded me of this time during my postdoc when I was struggling with setting up a new technique and I wasn't sure I really wanted to do it and it didn't work on the first try and it was a total pain the ass so I really didn't want to do it over again.

My advisor gave me two pieces of seemingly conflicting advice:

1. I'm sure you did it right. Don't second-guess yourself.

2. Just try it one more time. If it still doesn't work, move on. Get everything fresh and just try it again before you give up.

Now, statistical gurus will tell you that these two truths are not mutually exclusive. Yes, you could have done it perfectly the first time and just happened to grab a black sock instead of a red one. Yes, if you stick your hand back in the sock drawer and rummage around again, this time you might get a red sock.

But when you're trying to make the all-important decision WHAT DO I DO NOW, sometimes it's hard to feel like you're making the right choice. Are you wasting time by trying again? Are you being impossibly stubborn when you should really try a different approach? Or are you giving up too easily?



Which also reminds me of a time during grad school when I was struggling with a technique that worked initially and then stopped working for no apparent reason. I could not figure out what had changed. I hadn't changed anything.

One of my committee members said: You're near an edge.

And I said, What?

She explained that sometimes you get lucky and something works when it shouldn't. Then you go chasing after what are actually the wrong conditions for an experiment, when you'd be better off chucking the whole thing and starting fresh from first principles.

In this particular case, she was absolutely right. We redesigned the experiment and I got it working, consistently.

I never did get it to work the original way again.



It's a little heartbreaking and mind-fucking when you think you've seen something real, and potentially really interesting, and you can't reproduce it. If you're lucky, you're in a field where there are plenty of things to choose from, and you can pick any one of these to pursue or at least try a few times.... before you give up. And then you take a deep breath and try a different one.

In real life, you don't get all these do-overs.

Sometimes you're dealing with the most unpredictable thing in the universe: other people.



I've noticed that doing science for so long has made me really reluctant to trust my own decision making. Yes, in science I always went with my gut, and when I let people (my advisors) talk me out of doing that, I always regretted it. But it was also a fairly low-risk endeavor. I learned early on how to test the waters with cheap, quick pilot experiments. And I never got too attached to any one experiment (unlike my advisors and the ubiquitous asshole reviewer).

I think this lack of self-trust is actually one of the things that makes a great scientist great. The unwillingness to trust any result, no matter how appealing, until it has been thoroughly verified.

You can't say, "Well, it must be right, I did it! And I am awesome!"

No. You have to say, "This might be wrong."

You have to say it over and over and over. Even if you did it yourself with your own two hands, and saw the result with your own two eyeballs. Even if you think it's the coolest thing in the world and you desperately want it to be published in Cell so you can get a faculty position and a grant and a fleet of minions to do your experiments for you.

You always have to keep in mind that the journey isn't from Maybe Wrong -> Definitely Right.

It's from Uncertainty ---> Close Enough For Now.

On a long enough timeline, everything turns out to be just an approximation of the truth.

Sometimes you're pretty much right and you can be proud of that. But you can't really be sure until hindsight.

In real life, sometimes you have to go with your gut and hope you aren't completely fucking everything up. And sometimes when you ask for what seems like an impossible do-over, you can get one. Just because you asked nicely.

Labels: , , , ,

12 Comments:

At 1:02 PM, Blogger Kea said...

On a long enough timeline, everything turns out to be just an approximation of the truth.

And if the future turns out all right for humanity, albeit not for us, these words will be taught to every child in the context of a new cosmology, where the subjectiveness of truth is written into the mathematical laws of nature.

 
At 7:39 PM, Blogger CH said...

I was dumbfounded. Essentially the same question, asked twice, yielded opposite answers.

Sounds like you've never applied for a visa in an eastern European country :)

I've noticed that doing science for so long has made me really reluctant to trust my own decision making.

Non-science people have no idea about this level of brutal self doubt, and interpret it as wishy-washiness or worse.

 
At 10:45 AM, Anonymous Female Post-doc said...

I get SO nervous when my PI gets excited about preliminary data...even if I've seen the same thing twice. The third time, if I get the same results, then I'll sorta trust it.

CH, maybe that is why my sister does not understand my lack of satisfaction with my life and always tries to tell me that my life is great(!) and stable (!) and I have a job (!) and what's my problem (???). :-)

 
At 8:08 AM, Anonymous Anonymous said...

Interesting post, but your anecdote seems a bit of a stretch. It's possible that the cashier heard two different questions. In the first case, the cashier interpreted the question as "does your store carry envelope merchandise for me to purchase?", and the second time the cashier heard: "do you, personally, have a piece of paper to give to me, since I only need one piece right now?" Speaking as someone who's worked in retail, often times when people say "do you have...", they mean the royal you - stripping your identity and instead assigning you an identity as mouthpiece of the store...

 
At 11:15 AM, Blogger Ms.PhD said...

Female Post-doc,

This is why no one should EVER show preliminary results to a PI.

This is a very important piece of advice! Always repeat your experiments before showing anyone!!! ALWAYS!!!

I'm convinced that the majority of false and incorrect reports in science come from overenthusiastic PIs who assume work has been repeated (or are just in denial) because they like the result, in combination with the dangerous prevalence of trainees who are too afraid to admit that they can't reproduce their original results.

My point was that YOU, the person doing the experiments, has to be skeptical. You may have to be skeptical for your PI, as well, but if you don't have that rock-solid scientific cynicism at your core, you're going to be at risk of being pulled down the slippery slope.

Also, your sister can't possibly understand that your life is not stable as a postdoc and that it's not really a "job". It's not self-doubt, it's realistic. You're in a temporary appointment and there's no guarantee you'll get to move up or out to something better. Sure, yes you have a "job", for now, when many people don't. But it's perfectly understandable to want something better. A postdoc is not an end in itself.

Anon,

"your anecdote seems a bit of a stretch"

Wow, are you one of those asshole reviewers? Did this post actually annoy you, or did you just not realize how that came across?

Yes, of course the cashier must have interpreted the second question differently. In this case, I think perhaps she did not speak English completely fluently. However, almost identical language was used to make the request. THAT WAS THE POINT. I wouldn't have written about it otherwise.

 
At 6:27 AM, Anonymous Anonymous said...

Had money changed hands in between the first and second request? Perhaps the cashier was more inclined to help out a "paying customer" by reciprocating with a piece of paper the second time your friend asked.

 
At 10:04 AM, Blogger Ms.PhD said...

Anon,

Yes.

Maybe.

 
At 9:03 AM, Anonymous Anonymous said...

Are you kidding? Of course you ask again. You probably would be the kind of person walks into a market in Spain or Turkey or Thailand and just pay asking price for everything.

Where did all you guys get the idea that being hesitant and under-confident makes you a better scientist? A good scientist is always skeptical - but this does not mean you have to be so meek. I am excited by small bits of data, even if it turns out a fluke (and it's never happened to me so far that something I found early on and got excited about ended up being just nothing). If I did not enjoy those little rewards, science would be no fun!

 
At 11:57 AM, Blogger Ms.PhD said...

@Anon, you're right. I have never been to Spain, Turkey or Thailand, and I do not know how to haggle.

Does that make me meek? Maybe!

Where do you think people learn those skills? My family never taught me how to haggle. They don't know how. I didn't have any friends or mentors who taught me how to haggle.

I guess I would have to travel and learn the hard way! Too bad I can't afford to go globe-trotting right now.

I too am excited by small bits of data, even if it turns out to be irrelevant or unreproducible.

Forgive me if I find it VERY hard to believe that you have NEVER had it happen that something you thought you found ended up being nothing.

That would be my pesky scientific skepticism again. But I think anyone who really knows what they're doing has chased down enough possible results far enough to have found at least a few that did not turn out to be anything real.

 
At 1:00 PM, Anonymous Anonymous said...

@anon 9.57: "it's never happened to me so far that something I found early on and got excited about ended up being just nothing"

Really?? not even when debugging an experiment or code or test set up? then I'd say that either you've not been doing research for very long, or else you've only worked on 'safe' or well-established problems with few surprises....! (or maybe you're a genius)

 
At 4:38 PM, Anonymous Sasha said...

I had similar exchanges with people when I worked in retail. I would ask if they wanted the receipt with them or in the bag, and they would invariably answer "yes" which meant I would have to split the question into two separate questions (Do you want the receipt with you? Or would you like the receipt in the bag?) and ask again. I'm inclined to say it is a result of selective hearing and laziness, but I have no data to back that up.

I also wanted to let you know that I really enjoy your blog. [I didn't start reading it until 2010, so I've been catching up on all I've missed in hour long sessions.] I'm a female PhD candidate in agriculture, another male-dominated field, and have experienced my fair share of sexism. I can't believe professionals accept this inappropriate behavior!

 
At 4:16 AM, Anonymous Anonymous said...

in my field the dry lab and wet lab collaborate a lot and throughout the years i have learnt to appreciate the hard work on the other side of life sciences. however i am often appalled at the lack of statistical rigor in the wetlab experiments. (missing time points, thinking that sample at t=0 is adequate replacement for vehicle controls, etc)

i mean. it is hard work preparing all those pallets and all, so why do you not want to get it right in the first place so that all that work is not wasted and highly likely to be reproduced?

to us on the informatics side that would be akin to going further to analyze a set of genes obtained only from fold change differences across 10,000 genes in just 1 pair of experiments. When we are skeptical even when we get supposedly 'exciting' results from 6 pairs of data points!

 

Post a Comment

Links to this post:

Create a Link

<< Home