Tag Archives: Progress

How Academia Works After the Quantitative Revolution

In 2008, I noted the “quantitative revolution,” which replaced the romantic academia many dream about with a discovery factory:

Both the old Academy and the Leftists, however, are under even more heartless attack from the Quantitative Revolution, the measurement-and-control movement that subjects everything to test-and-reject, measure-and-fund, quantitative certainties.

The romantic academia that lives in our heart is dying or dead. Given a future between the Tyranny of Leftists and the Tyranny of the Quantitative Revolutions, my sympathies go to the quantitativists. They save what can be saved, submitting the universities to Research, Application, and funded Goals.

Last year, I formalized that description of how academia works:

Professors, like most people, respond to the incentives of power, influence, and money.

The institution of tenure reduces uncertainty regarding money, and focuses the incentives on power and influence.

Power in academia comes from the number of bodies a professor has under him. These bodies might be apprentices (graduate students he advises), journeymen (post-docs who have a PhD and work at the lab, or staff researchers), or simple workers (lab technicians, etc).

Influence in academia comes from the extent to which one is successful in influencing one’s peers. This is typically measured in terms of influence scores, which are a product of how often the academic is cited, weighted by how important of a publication he is cited in.

The best route to both power and influence is to earn grant money.

Daniel Allington must read my blog, as he writes the same thing:

Even among successful players of the funding game – and certain digital humanists have been very successful players, of late – one may find disquiet at the game itself, at the disproportionate importance now attached to it, and at the negative impact it is having on the careers of new researchers and (in the long term) on access to the profession as a whole by accelerating the casualization of both teaching and research. The underlying problem – regretted by practically everyone with a genuine love of scholarship – is the ongoing reconstruction of all disciplines on the social model of the natural sciences and the creeping abandonment of ‘autonomy’ (in the sense used by Bourdieu, 1993 [1987]) in the academic field through tacit acceptance of the principle – shared by university administrators, government ministers, and hiring committees alike – that knowledge can and should be valued primarily for its moneymaking potential. In

Allington is particular worried about the “digital humanities” which provides a road out of the ghetto for humanities students. Allington criticizes this as revealing “the corrupting agenda of our paymasters” — apparently he sees nothing wrong with the old boys network that progress-based research replaces.

Science is Real. Measurement is Real. Improvement Is Real

Bill Gates, the co-founder of the company I work for and a personal hero of mine, has an op-ed in the Wall Street Journal titled “My plan to fight the world’s biggest problems.” It’s an exciting piece because it ties together several of my recent posts very well.

Science allows us to predict, control, and improve variation in the world. In order to actually make progress to these goals, it’s important to establish exemplars of great work. This is enabled through operational definitions that allow concepts to be measured. The quest for progress in science collapses when measurement becomes too difficult tor too expensive.

But the reverse is also true: progress in science begins when measurement becomes accessible.

Bill Gates’ op-ed is so awesome because he brings us back to the real world. When someone says “science,” others thinks of some cartoon view of men in white coats in a laboratory. When someone says that goal of science is the prediction, improvement, and control of variation, someone else will say that such is a “very narrow definition of science, downgrading as it does understanding and explanation.”

But the person who writes you write like Bill Gates does — who never even bother with the word “science” and hammers in that improvements are real:

Such measuring tools, Mr. Rosen writes, allowed inventors to see if their incremental design changes led to the improvements—such as higher power and less coal consumption—needed to build better engines. There’s a larger lesson here: Without feedback from precise measurement, Mr. Rosen writes, invention is “doomed to be rare and erratic.” With it, invention becomes “commonplace.”

In the past year, I have been struck by how important measurement is to improving the human condition. You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal—in a feedback loop similar to the one Mr. Rosen describes.

This may seem basic, but it is amazing how often it is not done and how hard it is to get right. Historically, foreign aid has been measured in terms of the total amount of money invested—and during the Cold War, by whether a country stayed on our side—but not by how well it performed in actually helping people. Closer to home, despite innovation in measuring teacher performance world-wide, more than 90% of educators in the U.S. still get zero feedback on how to improve.

An innovation—whether it’s a new vaccine or an improved seed—can’t have an impact unless it reaches the people who will benefit from it. We need innovations in measurement to find new, effective ways to deliver those tools and services to the clinics, family farms and classrooms that need them.

… that’s the sort of person who can make a difference. The theory of science, measurement, and improvement are all left below the surface. What is left is a how-to guide to build a better world.

I write this blog for selfish reasons, I enjoy learning about the world. Bill Gates does what he’s doing to change the world.

Definitions and Progress

A couple days ago a post on Duck of Minerva linked to a working paper called “I can has IR theory?” [pdf]. The title was funny, but something about the contents bothered me.

I Can Has IR Theory Appears to have tow components
1. It is an extended hit peace against “neopositivism,” which appears to be a methodology (or something) disliked by the authors. It is difficult to know if this is true, however, because the authors do not bother to define their terms.
1. It includes a discussion of “scientific ontology,” which likewise is never defined.

Unlike “neopositivism” though (the only thing I can tell about which is that the authors — Patrick Jackson and Daniel Nexon — dislike it, and that it appears to be related to quantitative methods), the article includes numerous descriptions of “scientific ontology.” It is these descriptions that bothered me.

“Scientific ontology” appears to be synonymous for “nomological network,” an antiquated and simplistic form of modeling that is prone to error.

First, some passages from Jackson and Nexon’s working paper:

To be more precise, we think that international-relations theory is centrally involved with scientific ontology, which is to say, a catalog—or map—of the basic substances and processes that constitute world politics. International-relations theory as “scientific ontology” concerns:
• The actors that populate world politics, such as states, international organizations, individuals, and multinational corporations;
• Their relative significance to understanding and explaining international outcome
• How they fit together, such as parts of systems, autonomous entities, occupying locations in one or more social fields, nodes in a network, and so forth;
• What processes constitute the primary locus of scholarly analysis, e.g., decisions, actions, behaviors, relations, and practices; and
• The inter-relationship among elements of those processes, such as preferences, interests, identities, social ties, and so on.

(Note how they are measured is left out.)

And this passage (as mentioned above, “Neopositivism” is never defined and only loosely described, so focus on the passage related to “scientific ontology”)

The Dominance of Neopositivism
This line of argument suggests that neopositivist hegemony, particularly in prestige US journals, undermines international-relations theorization via a number of distinct mechanisms:
• It reduces the likelihood that international-relations theory pieces will be published in “leading” journals because neopositivism devalues debate over scientific ontology in favor of moving immediately to middle-range theoretic implications; • It reduces the quality of international-relations theorization by requiring it to be conjoined to middle-range theorizing and empirical adjudication; and
• It forces derivative middle-range theories to be evaluated through neopositivist standards.

(Note that scientific ontology thus excludes “middle-range theoretical implications.)

In an earlier work, I wrote that :

As a measure of construct validity, nomothetic span is more inclusive than Cronbach and Meehl’s (1955) concept of the nomological network, as nomothetic span includes not only how a construct relates to other construct, but also how measures of the same construct relate to each other (Messick, 1989).

Because the undefined concept of “scientific ontology” appears to be more or less identical to the idea of nomological network, which was described a half century ago. Without incorporating measurement into a model, it’s impossible to a functional definition, a method of falsifying the model, or even a way to make useful predictions. And without this ability, it’s impossible to make progress.

Operational definitions are absent from Jackson’s and Nexon’s piece, both from their primary terms, and their view of “scientific ontology.”

Escaping the Humanities Ghetto: Definitions and Paradigms

In both Political Science and the Humanities, the old boys network which prevented progress is collapsing, though it is hard for those who have lived in a field without progress to describe this.

ways_of_knowing_2

Earlier I criticized, Stephen Walt and John Mearsheimer for saying that International Relations work capable of progress is “non-paradigmatic”

Indeed, some senior IR scholars now rail against the field’s grand theories. In his 2010 ISA presidential address, for example, David Lake described the “isms” as “sects” and “pathologies” that divert attention away from “studying things that matter” (Lake 2011: 471). Thus, it is not surprising that “the percentage of non- paradigmatic research has steadily increased from 30% in 1980 to 50% in 2006” (Maliniak et al 2011: 439). Of course, one could advocate for middle range theories while disparaging grand theories, and indeed Lake does just that. The field is not moving in that direction, however. Nor is it paying more attention to formal or mathematically oriented theories (Bennett et al 2003: 373-74). Instead, it is paying less attention to theories of all kinds and moving toward simplistic hypothesis testing.

But if I am fair, I need to also mention that my friend Jason Heppler of Stanford University made a similar claim, about the Digital Humanities:

Those doing digital humanities tend to agonize over how we define the field (or is it a methodology?). Matt Gold’s edited volume Debates in Digital Humanities nicely sums up many of the ways people have tried to define digital humanities and what we mean when we say we do digital humanities. Certainly there are some common characteristics within the broad range of approaches, but the work itself is broad: it’s interpretation, coding, building, archives, theorizing.

Why define digital humanities? The enterprise is somewhat pointless. The promise and excitement of digital humanities lies with what we can do with it, not how it’s defined. But the queston is inescapable.

I don’t believe that the new International Relations work is apardigmatic, or that Digital Humanities is undefined. Rather, like other useful areas, these fields are defined by their tools and methods, not by ancient theoretical battles that can never be won. Chemistry is not the science of phlogiston, it is the science of titration and fission. Likewise, International Relations and Digital Humanities are increasingly defined by the tools they use to make progress, not connections in an old boys network.

This is a good change. Yesterday I mentioned the ghetto of the humanities, those fields whose graduates (a) can’t do math, (b) can’t conduct useful research, (c) can’t stay employed, and (d) can’t get paid.

wages_employment_majors_humanities_ghetto_md

The adoption of useful tools and progressive science is the best way to turn this around. The emerging paradigms of International Relation are embedded within statistics and modeling. The emerging definitions of Digital Humanities are embedded within text mining, semantic networks, and big data. What do these have in common? Useful tools designed to provide answers and enable progress.

Science, Paradigms, and the Old Boy Network

On Facebook, Daniel Nexon pointed me to this post by Steve Saideman, titled “Lamenting The Loss of the Light, The Ebbing of Grand Theory and The Decline of Old Boy Networks.” Saideman’s post itself is a commentary on Stephen Walt‘s and John Mearsheimer‘s ridiculous article, “Leaving Theory Behind: Why Hypothesis Testing Has Become Bad for IR,” which will soon appear in the European Journal of International Relations.

Walt and Mearsheimer’s article is absurd on many levels. But I mention it for how well it reflects my post, “Progress, Science, and Exemplars — or — When It Sucks to Be Young.”

In that post, I mention that it is horrible for your career to be young in a science with loose exemplars — that is, in a field that is “non-paradigmatic” or a “revolutionary science.” The more revolutionary the science, the looser the exemplars, the more “knowledge” and “experience” are both measured in years. The less things change — the less progress is made — the less youth matters relative to years of experience.

Or in diagrammatic form:

ways_of_knowing_3

What’s bizarre is that Walt and Mearsheimer agree with me! But this makes them sad. Walt and Mearsheimer would rather political science stay as anti-youth and revolutionary as possible, so that their power and influence could remain strong:

Over time, professions also tend to adopt simple and seemingly objective ways to evaluate members. Instead of relying on “old boy” networks, a professionalized field will use indicators of merit that appear to be impersonal and universal. In the academy, this tendency leads to the use of “objective” criteria—such as citation counts—when making hiring and promotion decisions. In extreme cases, department members and university administrators do not have to read a scholar’s work and form an independent opinion of its quality; they can simply calculate the “h-index” (Hirsch 2005) and make personnel decisions on that basis.22

The second part of the paragraph is literally incoherent, attacking the use of an h-index by arguing it’s a raw count of citation. Walt and Meirsheimer seem unable to do math, and so their inability to understand even basic fractions should not surprise you. What should be surprising is they are so openly defending the power aristocracy that comes from using subjective scores and the “old boys” network!

In fairness to Walt and Meirsheimer, the intellectual poverty they confess through their incoherent ramblings is not entirely their fault. Political science has been so revolutionary, so paradigmatic, so subjective for so long that few may know what a science actually is, or even understand the terms used to describe science.

Consider this earlier passage in Walt and Meirsheimer’s article, in which the “worse than wrong” passage is intended to be uncontroversial:

Indeed, some senior IR scholars now rail against the field’s grand theories. In his 2010 ISA presidential address, for example, David Lake described the “isms” as “sects” and “pathologies” that divert attention away from “studying things that matter” (Lake 2011: 471). Thus, it is not surprising that “the percentage of non- paradigmatic research has steadily increased from 30% in 1980 to 50% in 2006” (Maliniak et al 2011: 439). Of course, one could advocate for middle range theories while disparaging grand theories, and indeed Lake does just that. The field is not moving in that direction, however. Nor is it paying more attention to formal or mathematically oriented theories (Bennett et al 2003: 373-74). Instead, it is paying less attention to theories of all kinds and moving toward simplistic hypothesis testing.

The highlighted passage, originally by Daniel Maliniak simply means that empirical research is increasing, and that non-empirical research is declining, within political science. But Maliniak, and thus Walt and Mearsheimer, bizarrely use “paradigmatic” to refer to less paradigmatic (that is, less capable of progress) fields, and “non-paradigmatic” to more to more paradigmatic (that is, more capable of progress) fields.

ways_of_knowing_2

Political science has been in the fever swamp for so long that the notion of progress as an outcome of normal science has almost entirely been lost. If Walt and Mearsheimer had their way, it might be lost, and the field simply divided into a stationary oligarchy of old boys network.

At one point in their article, Walt and Meirsheimer say that “the creation and refinement of theory is the most important activity in [social science].” This is nonsense. The most important activity in science is the prediction, control, and improvement of behavior. Theory can help, diagram can help, interviews can help, process tracing can help. But the paen to old boys network, and the nonsense that Walt and Mearsheimer try to pass off as a scholarly article, certainly doesn’t.