Should non-experts shut up? The skeptic's catch-22

As the whirlwind of TAM8 died down on Monday, I was left with warm fuzzies toward all the Rationally Speaking readers and listeners who came up to me throughout the conference. Thanks, everyone - it was delightful to meet you. I was impressed by how nice you are, and how articulate; and not least of all by what excellent taste you have in blogs and podcasts!

I was also left with lingering ruminations about a few of the TAM8 talks, including Massimo's. While I agreed with the main thrust of Massimo's argument, I also disagreed in a way that I felt was important and interesting enough to merit a blog post of its own. You can read the talk here, but in a nutshell, Massimo was admonishing skeptics who reject the scientific consensus in fields in which they have no technical expertise - the most notable recent example of this being anthropogenic climate change, about which venerable skeptics like James Randi and Michael Shermer have publicly expressed doubts (though Shermer has since changed his mind).

Now, I'm totally with Massimo that it seems quite likely that anthropogenic climate change is really happening. But I'm not sure I can get behind Massimo's broader argument that non-experts should defer to the expert consensus in a field. The main problem, as I see it, is that while we can usually count on the process of peer criticism to check the research within a field for validity, I'm not sure we can count on it to check the validity of the field itself. I'm talking here about the methodology and the philosophical assumptions that underlie a field, the usually-unstated pillars on which all its findings rest.

Why? First of all, while there are strong incentives for a researcher to find errors in other work in the field, there are strong disincentives for her to challenge the field's foundational assumptions. It will be extremely difficult for her to get other people to agree with her if she tries, and if she succeeds, she'll still be taking herself down along with the rest of the field. Second of all, fields naturally select for people who accept their foundational assumptions. People who don't accept those assumptions are likely not to have gone into that field in the first place, or to have left it already.

Sometimes those foundational assumptions are simple enough that an outsider can evaluate them - for instance, I may not be an expert in astrology or theology, but I can understand their starting premises (stars affect human fates; we should accept the Bible as the truth) well enough to confidently dismiss them, and the fields that rest on them. But when the foundational assumptions get more complex - like the assumption that we can reliably model future temperatures - it becomes much harder for an outsider to judge their soundness.

So we almost seem to be stuck in a Catch-22: The only people who are qualified to evaluate the validity of a complex field are the ones who have studied that field in depth - in other words, experts. Yet the experts are also the people who have the strongest incentives not to reject the foundational assumptions of the field, and the ones who have self-selected for believing those assumptions. So the closer you are to a field, the more biased you are, which makes you a poor judge of it; the farther away you are, the less relevant knowledge you have, which makes you a poor judge of it. What to do?

I've run into this problem before with, for example, postmodernism. No matter what else the experts in that field disagree on, they would at least agree that they are all more or less talking sense. I'm dubious of that consensus, and so are plenty of other people who are at least as smart as me. But when we outsiders object that postmodernism is bunk, postmodernists retort that we don't know what we're talking about because we're not experts in the field.

(Incidentally, it was in recognition of that Catch-22 that I began my tenure on this blog with a couple of posts trying to find some objective, external criterion to evaluate whether the postmodernists really are talking sense. But that was admittedly more of a mental exercise than a serious proposal, since, as many of you pointed out in comments, my criteria had some major flaws in them!)

Here's a trickier case study: economics. Plenty of smart non-economists have objected that the fundamental assumptions of economics - for example, that theoretical models can usefully represent the complex behavior of societies - are so flawed as to make the discipline worthless. Just one example, from philosopher Brian Leiter: "[Economics] parlays a set of implausible and utterly unrealistic assumptions into tidy, mathematically-expressible theories that have little or no connection to reality." Economists, on the other hand, will counter that economics is just too complicated for non-experts to have any hope of commenting intelligently on. As a case in point, this much-discussed op-ed by Virginia Fed economist Kartik Athreya, titled "Economics is Hard. Don’t Let Bloggers Tell You Otherwise." Athreya insists that the models are sound, they're just too complex to be understood except by people who have taken PhD-level economics, and that anyone who hasn't should refrain from judging the field.

So how should we deal with our Catch-22? Well, I don't pretend to have any definitive answers here, but I have a few preliminary thoughts. First, luckily, the Catch-22 isn't quite as stark as I made it sound. For example, you can often find people who are experts in the particular methodology used by a field without actually being a member of the field, so they can be much more unbiased judges of whether that field is applying the methodology soundly. So for example, a foundational principle underlying a lot of empirical social science research is that linear regression is a valid tool for modeling most phenomena. I strongly recommend asking a statistics professor about that. 

I also think there are some general criteria that outsiders can use to evaluate the validity of a technical field, even without “technical scientific expertise” in that field. For example, can the field make testable predictions, and does it have a good track record of predicting things correctly? This seems like a good criterion by which an outsider can judge the field of climate modeling (and "predictions" here includes using your model to predict past data accurately). I don't need to know how the insanely-complicated models work to know that successful prediction is a good sign. And there are other more field-specific criteria outsiders can often use. For example, I've barely studied postmodernism at all, but I don't have to know much about the field to recognize that the fact that they borrow concepts from complex disciplines which they themselves haven't studied is a red flag.

Again, let me be clear that my point is not that I’m skeptical of anthropogenic climate change. My point is that I don't think it's as simple as saying that skeptics should automatically accept the expert consensus in fields in which they have no technical expertise. I think we can, and indeed we should, pass judgment on fields we're not experts in. The tricky part is just figuring out what criteria to judge them by.