Sunday, August 20, 2006

Fact-checking and Freedom of Speech

Had an interesting chat yesterday with a friend who heard a recent NPR report on how- i.e. the mechanism of - journalism has degraded to the point of widespread inaccuracy.

This is third-hand, but apparently many media sources are now requiring 'story quotas' from journalists, which creates pressure to 'produce'. I'm not sure if there's any metric for quality of these stories- it would appear not.

This got me thinking about how public media is not that different from scientific media, and how some of the changes proposed recently would make scientific publishing more like public media sources.

We've discussed here on many occasions how arbitrary scientific peer-review can be. If you happen to get 3 reviewers who know your senior author, for example, and they all think he's a great guy, your paper- however stellar or crappy- is much more likely to get into Top Tier Journal than if your co-authors are all nobodies.

Similarly, it's much more difficult to get a High Impact paper if you're a nobody, or if everybody hates your boss, because the bias among all scientists seems to favor prior reputation over the current data at hand. Why reward someone you hate for a job well done?

It's probably a weakness inherently human and psychological that even scientists can't rise above in our supposed Supreme Objectiveness.

Having said that, what if we let everyone publish everything in One Big Journal, and then let everyone comment on everything, and posted all the comments for everyone to see? Kind of like One Big Science Blog (with the obligate filter for obscenities, a.k.a me, for this site).

If we assume that scientific discourse will follow a similar path to public media sources (and I'm including both mainstream and 'alternative' sources in that group), we can assume a few things will happen:

1. The amount of information available will skyrocket.
2. The amount of crappy information will also skyrocket.

Anybody want to come up with an equation to determine whether those two things are at all interdependent? That would make for an interesting math problem.

Anyway, the question on everyone's mind is, how are we going to sort through all of the crap to get to the good stuff?

I've seen the following three behavioral responses from scientists at all levels:

1. Scientist reads Almost Everything.
2. Scientist reads Almost Nothing.
3. Scientist reads only what's absolutely required- i.e. only what's directly related to their own area of research.

We've all been all three of these at various times, but I'm most concerned about #2. We really don't want very many - or any- scientists like that getting advanced degrees. Or, god help us all, their own labs.

But isn't that what the average American is doing these days? Most of us have stopped watching the news because it either

a) has no worthwhile content
b) is totally inaccurate
c) is too depressing.

And doesn't that also apply to most of the science publishing out there?

I've taken to watching Face the Nation and Meet the Press on Sundays, and listening to the occasional short burst of NPR or CNN when I'm lucky enough to flip by the station when they're actually reporting news. It's not enough, and if it were my job to be aware of wordly goings-ons I would be in big trouble.

But I'm also way behind on my reading for work. I'm too busy during the week and can't face doing it at night or on the weekend, the way I did in former years.

There are a variety of highly biased resources for keeping up with literature, like Faculty of 1000. Less biased tools have what I call a high 'clutter-factor' or CF, such as having journals email you the new table of contents (TOCs) every week or month when the new issue comes out. Another method with a high CF that hasn't worked well for me is having journals sent to my house ("I"ll read them on the toilet!") instead of to my lab.

The point of all this musing is the alarming conversations I've been having with people at all levels lately, wherein it becomes frighteningly obvious that most scientists don't read outside their immediate area at all. PIs are gifted at appearing to know all the latest research- because they go to meetings, but they don't know it in any depth.

The interesting side-effect of this is that most PIs don't know whether the correct controls have been done unless they're threatened enough to wait for the paper and examine it in detail. So PIs go around dropping names and results left and right, without ever bothering to verify the verity of their claims.

This feeds back into the media frenzy style of news communication- where word spreads like gossip, without any supporting evidence, and without any fact-checking.

So my question is, are we as scientists lacking the right to express our freedom to speechify? Shouldn't we be allowed to run off at the mouth like everyone else does?

Labels: , , ,


At 5:50 PM, Anonymous chris said...

You should really try the RSS feeds for searches either at PubMed or hubmed ( for your regular queries. I get my regular searches delivered (this includes specific author queries), and supplement those with eToC and random searches for stuff outside my field.

As to whether I actually *read* anything... I try to put one afternoon a week away to catch up on my reading, usually towards the end of the week when Nature and Science are hot off the press. Failing that, I force myself to at least skim things I'm unlikely to read in depth on the train, in waiting rooms, etc.

At 7:47 PM, Anonymous Anonymous said...

as to the reason why some don't read outside their research area, they are brainwashed slaves whose lives are completly robotic.

At 4:12 AM, Blogger dlamming said...

I'm not sure what the point of your post is...

Maybe I'm naive, but I don't think people will accept/reject your paper based on who you are. I do think that people will bend a bit to not be as harsh if you've done good work before, and will be more skeptical and demand more experiments if your work contradicts their own theories. Which is still bad, but not as bad as you make it out to be,

I disagree that most science is "a) has no worthwhile content b) is totally inaccurate c) is too depressing." It's often too specialized, but that's different. Which is why you don't read things outside your own field that don't show up in Nature/Science/Cell.

Anyway, if you care, you should find out what's going on in the world. CNN et al are worthless, because they report on such trival news - and you're a blogger! You should know where to go for real news.

At 4:23 PM, Anonymous Andre said...

You said:
"Having said that, what if we let everyone publish everything in One Big Journal, and then let everyone comment on everything, and posted all the comments for everyone to see? Kind of like One Big Science Blog (with the obligate filter for obscenities, a.k.a me, for this site)."

If you're interested in a more open process where referees don't assess impact (that's done by the community afterwards) you should really check out

For a related discussion, you can't beat Paul Ginsparg:

At 8:40 PM, Anonymous Anonymous said...

I actually did not particularly want to make this a comment, but I don't see any other way of contacting you. I just wanted to thank you for keeping up with, and originally creating this blog. I just came across it about a week ago in a rampant search of the internet for information from graduate students and/or post-docs. Some arcane search on google brought up your website.

I read all/most of the archives and wanted to thank you. Everything you have laid out has really helped me make my own decisions on whether I will/won't go to graduate school in a biological field. This first hand account is a great source of information, and has helped me firm my own life plans.

I really appreciate it and hope you continue the descimation of information. The blog world, though originally I found to be superficial and lacking, is helped greatly by these first hand accounts of life situations that others are interested in or considering. Its like test driving a car before buying it, except its years of your life.

Thank you for documenting.

At 12:25 AM, Anonymous Anonymous said...

I actually think there's a lot of truth to this post. Anyone who thinks peer review isn't seriously impacted by personality conflicts & big egos is naive indeed, as is anyone who thinks a scientist can work a full-time job and also keep up on the literature in a general field like "developmental biology," much less OTHER fields as well. The situation is even worse for educators, who have more demands on their time and very little practical justification to spend a day reading materials and methods or abstracts from meetings.

Obviously most people in academia and/or science WANT to read, we love to read, and we get quite frustrated when we can't keep up with the scientific (or world) news. Pretty much any way of sorting through information is biased - we may not be able to avoid bias, but at the very least we should recognize it. Good post.

At 1:10 AM, Anonymous Anonymous said...

Good journalism died the day the Daily Star was invented.

A tabloid full of tits, trash, tat, more tits and three-sentence long story scoops!


Post a Comment

Links to this post:

Create a Link

<< Home