How to think about “think” tanks

It is sometimes said that think tanks are good for democracy; indeed the more of them, the better. If there are more ideas in the public arena battling it out for your approval, then it’s more likely that the best idea will win, and that we will all have better public policies. But intuitively many of us have trouble believing this, have trouble knowing who is being truthful, and don’t know who to trust.

This battle of ideas, studies, and statistics has the potential to make many of us cynical about the whole process, and less trusting of all research and numbers. If a knowledgeable journalist like the Canadian Kady O’Malley expresses a certain exasperation that think-tank studies always back up “the think-tank’s existing position,” what hope is there for the rest of us? A flourishing of think tanks just let’s politicians off the hook, always allowing them to pluck an idea that suits their purposes, and making it easier to justify what they wanted to do anyways.

Maybe we shouldn’t be so surprised that think tanks produce studies confirming their (sometimes hidden) biases. After all this is something we all do. We need to arm ourselves with this self-awareness. If we do, then we can also be more aware of the things in a think tank’s make-up that can help in judging its credibility, and also how public policy discussion should be structured to help promote a sincere exchange of facts and ideas.

1. Begin with the idea that “think” tanks don’t think

Well that’s overstating it, but what I mean is they are not guided, appearances to the contrary, by reason alone. But then again, none of us are. Our intuitions come first, and the reasoning side of our minds serve to justify those intuitions.

Many academic economists are very quick to dismiss think tank research because they see these studies as falling on the wrong side of the line between examining the economy as it is, and advocating for an economy as it should be: a distinction that Milton Friedman called “positive” versus “normative” economics. In a famous essay published in 1953 the late University of Chicago economist wrote:

Positive economics is in principle independent of any particular ethical position or normative judgments. … it deals with what is, not with what ought to be. Its task is to provide a system of generalizations that can be used to make correct predictions about the consequences of any change in circumstances. Its performance is to be judged by the precision, scope, and conformity with experience of the predictions it yields. In short, positive economics is, or can be, an objective science.  [ Milton Friedman (1953). “The Methodology of Positive Economics,” in Essays in Positive Economics. Chicago: University of Chicago Press, page 4. ]

This is an important touchstone for serious social scientists, and dressing up, inadvertently or not, normative research as positive science is crossing a line from professional to political. They would consider this a serious loss of credibility.

But we all know that this line is sometimes hard to draw, and in fact Friedman went on to write:

Of course, the fact that economics deals with the interrelations of human beings, and that the investigator is himself part of the subject matter being investigated in a more intimate sense than in the physical sciences, raises special difficulties in achieving objectivity … [ Friedman (1953, page 4) ]

The whole field of moral psychology has developed since he wrote this, and makes clear that our intuitions are automatic and instantaneous. The Righteous Mind: Why Good People Are Divided by Politics and Religion by Jonathan Haidt, who teaches at the New York University’s Stern School of Business, offers a masterful discussion of the challenges of engaging in what he calls “exploratory” thinking—of searching scientifically for the truth—in the face of a strong human tendency of falling into “confirmatory” thinking—of seeking evidence to support underlying intuitions.

His first rule of thumb is “Intuitions come first, strategic reasoning second.” We are hardwired to be that way, and Haidt’s book offers an explanation of the evolutionary advantage our ancestors developed by evolving so that reason serves intuition, and that now leads people to “bind themselves to political teams that share moral narratives. Once they accept a particular narrative, they become blind to alternative moral worlds.” [ Haidt (2012, page xvi) ]

As a starting point, remember, think tanks don’t think.

2. Be aware of the underlying intuition

So while we might all strive to attain Friedman’s ideal of objective, scientific inquiry, when it comes to think tanks it’s probably best to, in Haidt’s words, “Keep your eye on the intuitions, and don’t take … arguments at face value.”

But this is a big challenge, all think tanks seem to tell us they are “non-partisan.”

If we understand this to mean “objective,” or “scientific” (as opposed to not being formally affiliated with a political party), we should appreciate they may well be sincere in telling us that. But that should not be enough.

Consider this exchange between the noted journalist Steven Paiken, who appears to be somewhat surprised by a representative of what, he suggests, many recognize as the most right-wing think tank in Canada. The discussion with Mr. Jason Clemens of the Fraser Institute, part of a panel discussion on the role of think tanks in public policy broadcast on a popular current affairs program, begins after the other two panelists have placed their organizations on the political spectrum.

Then Mr. Paiken turns to Mr. Clemens (just after 8 minutes into the program):

“Jason I think everybody would put a label on your organization. It’s unabashedly, they would say, right-wing. Do you subscribe to that?”

“… Our approach is we are an economic think tank.”

“So where are you on the political spectrum?”

“We don’t worry about the political spectrum. … “

“We worry about picking issues that are important … and then using an accepted, quantitative approach to asking and answering questions. In fact, our motto here … is ‘if it matters, measure it.’ And so if you look at most of our research, it’s measurement approach because, as we learn as researchers, you can’t argue about 2 plus 2 equaling 4. … Our approach is to quantify things.”

The exchange continues all the way to 15:22, and there is nothing Mr. Paiken can say to get him to drop, even just a bit, the veil of objectivity with which he has cloaked his organization.

In some sense this is natural, we all do it. Haidt says “We are indeed selfish hypocrites so skilled on putting on a show of virtue that we fool even ourselves.” [Haidt (2012, page xv) ] We have no basis to question Mr. Clemens sincerity, it just means that we need an independent guidepost to the underlying intuitions because we can’t expect those to be explicit.

Stephen Tapp makes our challenge a bit easier by offering an excellent application of the “if it matters, measure it” principle to uncover the underlying intuitions of Canadian think tanks. His blog post called “What can a little birdie (Twitter) tell us about think tank ideology?” uses information on the followers of a long list of Canadian think tanks to rank them across the political spectrum.

The best way to explain his method is through example. If I were a think tank, Mr. Tapp would have gone to my Twitter account and noted that (as of November 22nd, 2014) I have 2,634 followers. Of these 153, or 5.8%, also follow the Fraser Institute—which Mr. Tapp takes as his touchstone for “right wing”—and at the same time do not follow the Canadian Centre for Policy Alternative, his touchstone for “left wing.” Similarly 327, or 12.4%, follow the Centre but not the Institute.

My score on the political spectrum is the difference between these percentages, -6.6 (or -0.066 if we were to express this a fraction of 1 rather than 100), the negative indicating that my followers are more likely to be following the paragon of left wing think tanks in the country than that of the right. In this sense I would be  a “left wing” think tank, a bit left of centre.

He does this for all the think tanks in Canada, and displays the results as follows. (You can visit his blog post to get the names of all the organizations if that interests you.)

A similar type of ranking for American think tanks is offered in the first column of the first table in this 2005 paper by Tim Groseclose and Jeffrey Milyo, published in the Quaterly Journal of Economics. It is based upon the number of times members of Congress cite the think tank, and ascribes the political leanings of the members—which can be measured pretty accurately—to the think tank.

You should have maps like this in your mind, but obviously they can’t offer the whole story, as Mr. Tapp, for example, is quick to acknowledge.

We should also appreciate that think tanks may be more likely to engage in exploratory thinking rather than confirmatory thinking depending upon how they are organized. We should also look at their organizational structure.

Here’s a paragraph from Jon Haidt’s book worth quoting:

We should not expect individuals to produce good, open-minded, truth-seeking reasoning, particularly when self-interest or reputational concerns are in play. But if you put individuals together in the right way, such that some individuals can use their reasoning powers to disconfirm the claims of others, and all individuals feel some common bond or shared fate that allows them to interact civilly, you can create a group that ends up producing good reasoning …. This is why it’s so important to have intellectual and ideological diversity within any group or institution whose goal is to find the truth (such as an intelligence agency or a community of scientist) or to produce good public policy (such as a legislature or advisory board). [ Haidt (2012, page 90) ]

If you see an organization tightly beholden to funders, so that “self-interest” is at play, or an organization dominated by like-minded individuals without diversity of ideological perspectives, then it will be harder for this organization to play the game on the “positive” side of Milton Friedman’s line.

3. Place more credence on conversation than on debate

It’s not just organizational structure that matters, but also how the interaction between think thanks is organized. Haidt points out that we are more likely to change some one’s mind if the interaction is “civil.” He suggests that “[i]ntuitions can be shaped by reasoning especially when reasons are embedded in a friendly conversation.”  [page 71.]

The metaphor of public policy discussion as taking place in some sort of arena with the gladiators fighting it out until only one is left standing is not helpful. You should wonder about the value of debate, since this type of forum puts reputations on the line, and hardens positions. Put more credence on interactions that can be characterized as conversation.

Here are two examples that touch me in a personal way, the policy discussion about the relationship between inequality and economic opportunity.

The first is conducted as a debate, one that took place in New York City on October 22nd, 2014 and organized by Intelligence Squared U.S. If you’ve never watched any of these, you should. The debaters are always articulate, the moderator is excellent, and the topics are provocative. But as you can appreciate the debaters must feel that their reputations are on the line, after all they are under pressure to change the opinions of the audience members, who are polled both before and after the debate. The side swaying opinion the most is declared the winner. Galidators in the arena, indeed!

In this case the motion is “Income inequality impairs the American dream of upward mobility“. For the motion are Elise Gould and Nick Hanauer; against Edward Conrad and Scott Winship. You don’t have to watch much of the debate or read much of the transcript (by clicking on the above link), to realize that there is nothing either side can say to the other that will lead them to change their mind.

This is a format, I might suggest, that in Haidt’s view will harden the participants’ reasoning in way that reinforces their underlying intuitions. It is terribly entertaining, and informative, but it puts a premium on performance. As audience members, we may well wonder about the veracity of the facts being thrown back and forth, and in the very least that some subtly in interpretation could well be lost.

Here’s one example. It deals with the descriptive relationship across countries between inequality and the degree to which individual adult income is related to parental income during childhood, the so-called Great Gatsby Curve about which I, among others, have written.

Scott Winship of the Manhattan Institute has been a persistent critic of this relationship. I bring this up, in part, because I also want to go on the record and note that when he says my “most recent paper highlights serious problems [with the Great Gatsby Curve and my] previous research,” it should be clear that this is Mr. Winship’s interpretation of my research, and not my understanding of my own research.

Grateful as I am to have a reader, I don’t feel my more recent work involving a three country comparison contradicts previous papers I have written. I certainly don’t have any reason to question Mr. Winship’s sincerity, he must believe what he says. But he’s never checked with me before he has written or spoken on the subject, we’ve never had a conversation about the issue, not even on one occasion when we sat across the very same dinner table together.

Here’s an excerpt from the transcript.

The real the point is that it must be hard for the audience to make a judgment on what’s right, what’s wrong, what’s more subtle and could be right, and where there’s scope for agreement. This is one instance in which you can just see “reason” going to battle for “intuition” in this kind of forum (and now I have to admit I’m also speaking for  myself!).

Contrast this format with a conversation on pretty well exactly the same subject between two individuals who, most informed people would reasonably suggest, lie on opposite ends of the political spectrum, Charles Murray of the American Enterprise Institute and Timothy Smeeding of the University of Wisconsin. At one point the moderator expresses surprise at the extent to which these two guys agree, even noting that they are wearing the same kind of shirt!

http://vimeo.com/100565488

After watching this I’m thinking that it would be interesting to have dinner with Charles Murray.

Conversions require leadership, goodwill, and civility. So look not just at who is speaking, but the structure in which they are placed. Conversations have more credence than debates.

4. Full disclosure

I think think tanks are valuable, and I have engaged with a number of them. Their great strength is that they offer academics what they so often lack, both a communication strategy for research that deserves a wider audience as well as a fountain of policy relevant ideas that deserve more research.

Organizations, like individuals, are more likely to be open-minded and engage in exploratory thinking when they are accountable to a broad audience, whose views are not fully known ahead of time, and who are well-informed. So when you think about think tanks, think about who they are speaking to, and recognize that quite possibly we get, in the end, the think tanks we deserve.

[ This post is published in an abbreviated form under the same title in the January/February 2015 issue of Policy Options by the Institute for Research on Public Policy. ]