Fact-checking and Freedom of Speech

Had an interesting chat yesterday with a friend who heard a recent NPR report on how- i.e. the mechanism of - journalism has degraded to the point of widespread inaccuracy.

This is third-hand, but apparently many media sources are now requiring 'story quotas' from journalists, which creates pressure to 'produce'. I'm not sure if there's any metric for quality of these stories- it would appear not.

This got me thinking about how public media is not that different from scientific media, and how some of the changes proposed recently would make scientific publishing more like public media sources.

We've discussed here on many occasions how arbitrary scientific peer-review can be. If you happen to get 3 reviewers who know your senior author, for example, and they all think he's a great guy, your paper- however stellar or crappy- is much more likely to get into Top Tier Journal than if your co-authors are all nobodies.

Similarly, it's much more difficult to get a High Impact paper if you're a nobody, or if everybody hates your boss, because the bias among all scientists seems to favor prior reputation over the current data at hand. Why reward someone you hate for a job well done?

It's probably a weakness inherently human and psychological that even scientists can't rise above in our supposed Supreme Objectiveness.

Having said that, what if we let everyone publish everything in One Big Journal, and then let everyone comment on everything, and posted all the comments for everyone to see? Kind of like One Big Science Blog (with the obligate filter for obscenities, a.k.a me, for this site).

If we assume that scientific discourse will follow a similar path to public media sources (and I'm including both mainstream and 'alternative' sources in that group), we can assume a few things will happen:

1. The amount of information available will skyrocket.
2. The amount of crappy information will also skyrocket.

Anybody want to come up with an equation to determine whether those two things are at all interdependent? That would make for an interesting math problem.

Anyway, the question on everyone's mind is, how are we going to sort through all of the crap to get to the good stuff?

I've seen the following three behavioral responses from scientists at all levels:

1. Scientist reads Almost Everything.
2. Scientist reads Almost Nothing.
3. Scientist reads only what's absolutely required- i.e. only what's directly related to their own area of research.

We've all been all three of these at various times, but I'm most concerned about #2. We really don't want very many - or any- scientists like that getting advanced degrees. Or, god help us all, their own labs.

But isn't that what the average American is doing these days? Most of us have stopped watching the news because it either

a) has no worthwhile content
b) is totally inaccurate
c) is too depressing.

And doesn't that also apply to most of the science publishing out there?

I've taken to watching Face the Nation and Meet the Press on Sundays, and listening to the occasional short burst of NPR or CNN when I'm lucky enough to flip by the station when they're actually reporting news. It's not enough, and if it were my job to be aware of wordly goings-ons I would be in big trouble.

But I'm also way behind on my reading for work. I'm too busy during the week and can't face doing it at night or on the weekend, the way I did in former years.

There are a variety of highly biased resources for keeping up with literature, like Faculty of 1000. Less biased tools have what I call a high 'clutter-factor' or CF, such as having journals email you the new table of contents (TOCs) every week or month when the new issue comes out. Another method with a high CF that hasn't worked well for me is having journals sent to my house ("I"ll read them on the toilet!") instead of to my lab.

The point of all this musing is the alarming conversations I've been having with people at all levels lately, wherein it becomes frighteningly obvious that most scientists don't read outside their immediate area at all. PIs are gifted at appearing to know all the latest research- because they go to meetings, but they don't know it in any depth.

The interesting side-effect of this is that most PIs don't know whether the correct controls have been done unless they're threatened enough to wait for the paper and examine it in detail. So PIs go around dropping names and results left and right, without ever bothering to verify the verity of their claims.

This feeds back into the media frenzy style of news communication- where word spreads like gossip, without any supporting evidence, and without any fact-checking.

So my question is, are we as scientists lacking the right to express our freedom to speechify? Shouldn't we be allowed to run off at the mouth like everyone else does?