Predicting “Null Results,” with Science

Chris Blattman and Daniel Nexon both link to a paper. “Oil and Conflict: What Does the Cross Country Evidence Really Show?,” published in AEJ: Macro (the article is gated, but an older version is available from SSRN).

The paper purports to show no correlation between the presence of oil and violence. This may or may not be true — the issue seems complicated and as long as North Dakota doesn’t erupt into another round of midwest violence, my life won’t change much one way or another.

What was interesting was Dan Nexon’s commentary on it. I once would have agreed with Nexon’s comment. That was before I learned the tools that allow to conduct research.. Anyway, Nexon’s comment:

Indeed, no one — and I mean no one — who has ever invoked “Popper” or “falsification” as a standard for scientific inquiry should be allowed into a proseminar while graduate students remain actively dissuaded from pursuing research with null results because everyone knows you can’t get null results published.

By “null result,” Dr. Nexon actually means a lack of significant correlation. Nexon’s comment only makes sense if you assume he has been exclusively exposed to fully-saturated models, of the r2 = .15, p = .04 sort beloved by first-year graduate students.

The rigorous way to test for a lack of correlation is in a structural model. For instance, take a structural equations model.

Each line with a number of it is analogous to an r2 finding in a fully saturated model. (You can read the whole thing if you want details 😉 ). But more important, every box-box, box-circle, or circle-circle pair with no line is assumed to be a “null result,” apart from the variation explained by following the lines of correlation between them. There are some easy introductions to structural equation modeling available. Structural equation modeling allows you not just to have science — but even normal science — while testing for lack for correlation between variables.

Karl Popper’s work on falsification is a hallmark of science. It is required for science to fulfill its objective of predicting, controlling, and improving behavior of the objects we study.