Recently-acquired mancrush Jonah Lehrer had a cool article in the New Yorker recently: The Truth Wears Off (PDF of the otherwise-paywalled full article). It concerns something called “the decline effect”, which seems to afflict a broad range of scientific pursuits. In a nutshell, a new and exciting discovery is announced. Other scientists replicate it, and confirm that yes, the effect appears to be real. But then, over time, a curious thing happens: The magnitude of the measured effect begins to decline in subsequent studies. It’s almost as if, as Lehrer puts it in his title, the truth “wears off.”

Several of the scientists Lehrer spoke to for his article were uncomfortable being quoted. The decline effect calls into question some of the fundamental underpinnings of the objective, science-y stuff to which they have devoted their lives. The article apparently has been somewhat controversial; witness the cranky review given to it by my other recent mancrush, Steven Novella: The decline effect.

Novella quotes Lehrer’s conclusion (somewhat out of context), and rants on a bit:

And this is why the decline effect is so troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren’t surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.

This paragraph sums up what I was describing above – Lehrer is ultimately referring to aspects of science that skeptics have been pointing out for years (as a way of discerning science from pseudoscience), but Lehrer takes it to the nihilistic conclusion that it is difficult to prove anything, and that ultimately “we still have to choose what to believe.” Bollocks!

I find Novella’s relatively impassioned response to Lehrer really interesting. Rather than simply acknowledging that Lehrer, as a writer selling copies of the New Yorker, faces a different set of contingencies with respect to hyperbole and colorful language than does, say, a neuroscientist selling science-based medicine, Novella really goes for it, taking Lehrer’s statements way past the point where I, at least, think he intended them to be taken, erecting a strawman he can then knock down. Novella talks past much of Lehrer’s argument, ignoring those points where Lehrer offers explanations for the decline effect. For example, Lehrer talks about the too-human fallibility of scientists chasing significance, and the problem of a publication and review process that rewards the dramatic and “interesting” result, which at first is the result that demonstrates a new and hitherto unrecognized effect, but eventually becomes the result that undercuts the now-established wisdom.

Lehrer writes:

The problem of selective reporting is rooted in a fundamental cognitive flaw, which is that we like proving ourselves right and hate being wrong. “It feels good to validate a hypothesis,” Ioannidis said. “It feels even better when you’ve got a financial interest in the idea or your career depends on it.”

I thought about this dispute while listening to D.J. Grothe’s recent interview with paranormal investigator Joe Nickell. Nickell is a skeptic, as is Novella, but his approach impresses me more than Novella’s. Nickell recognizes that the things he’s researching are almost certainly bunk, but he doesn’t just rant (as Novella too-frequently does on the otherwise-awesome podcast he heads up, the Skeptic’s Guide to the Universe) about what delusional fools paranormal believers are. Instead, Nickell wants to know what’s really going on. It’s easy (and probably satisfying, on some level) to just cry “Bollocks!” when confronted by belief in something that flies in the face of the best evidence. But what Nickell emphasizes in his interview is that to stop at that point is to miss out on the opportunity to solve the real mystery. So Nickell investigates, in the sense that he actually goes to the site of the latest “haunting” (or whatever), listens to the witnesses with as open a mind as he can muster, and digs until the truth emerges.

As Doyle’s Holmes pointed out in “A Scandal in Bohemia”:

It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.

Novella is quick to dismiss Lehrer’s article — too quick, I believe. There’s something there, and it’s worth taking a look at. But someone who believes he has the answers already, and is driven by an apparent need to denigrate those who believe differently, is going to be unlikely to do much looking.

This entry was posted by jbc on Sunday, December 19th, 2010 at 6:32 pm and is filed under medical_science, Skepticism. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.