Go-Up or Go-Next?

The idea that there are two cultures in academic life, a culture focused on the humanities and another on science, is not a new one. The famous “Two Cultures” lecture is more than fifty years old, and Brother Guy Consolmagno identifies instances of the two cultures in medieval Catholic Europe in his book of adventures.

Jason Lee Steorts, a writer for the National Review Online, defended NRO’s dismissal of John Derbyshire, demonstrates that by criticizing Derbyshire’s controversial article for being hypocritical. In 2012, Derbyshire writes in paragraph 4:

The default principle in everyday personal encounters is, that as a fellow citizen, with the same rights and obligations as yourself, any individual black is entitled to the same courtesies you would extend to a nonblack citizen. That is basic good manners and good citizenship. In some unusual circumstances, however—e.g., paragraph (10h) below—this default principle should be overridden by considerations of personal safety.

While two years previously, in a speech on race relations, he said

Group differences are statistical truths. They exist in an abstract realm quite far removed from our everyday personal experience. They tell you nothing about the person you just met.

This would be hypocracy, unless you believe the fundemental principles of statistics have undergone a revolution in the past two years. Which, of course, they have.

There are two ways of understanding statistics. The terms “Frequentist” and “Probabilistic” are thrown around here, but to me those words are more confusing than helpful. So I will call them the go-up and go-next views of statistics.

The Go-Up view of statistics is that statistics measures the population from which an observation comes from. The appropriate way to go-up is to wait until you have a sufficient number of observations. and then generalize about the population from our observations. This is the method that Derbyshire was describing in 2010. A large number of observations of academic performance show consistent gaps between black and white learners. Because we’re “going-up” from observations to populations, we can conclude some things about the population, and how outcomes in the population should work-out over all, but it makes no sense to try to predict any given student’s success based on this. We’re going-up, not going-next.

The Go-Next view of statistics is that statistics gives us the likelihood of something being true, based on what has come before. In Go-Next statistics, population-averages are besides the point. What matters is guessing what’s going to happen, next, based on what you’ve seen before. The whole point is to guess what’s going to work for individuals you know only a few things about, based on your experience with other individuals who shared some things with the new strangers.

Both the Go-Up and Go-Next interpretations of statistics are hundreds of years old. Go-Up statisitcs strikes many as more beautiful. Go-Next as, perhaps, more practical, more commercial, more technical. Astronomers use go-up statistics. Weathermen use go-next statistics.

The Internet changed everything.

Academics pay attention to reality. Professors, like most people, respond to the incentives of power, influence, and money. Companies like Google, Facebook, Apple, and my employer do not care much about abstract ideas like “What can we infer about internet users in general based on the observations we collect.” Instead, they care, very, very deeply, about making you delighted. Because people will spend money to be delighted.

When you log onto your Facebook screen, or type a search into Google, or click the genius buttons in iTunes, you want it to just work. You want the perfect update, the perfect site, the perfect song. Advertisers want the perfect ad for you.

In this context, the view of statistics that Derbyshire outlined in 2010:

Group differences are statistical truths. They exist in an abstract realm quite far removed from our everyday personal experience. They tell you nothing about the person you just met.

Is just stupid. Facebook doesn’t care about the group differences between men and women. It cares that when you log in, it can give you an update from your favorite sports team, or gossip from your favorite celebrity, or whatever. Never before in history has so much math been used to make you happy.

It’s all about you.

It’s all about guessing, based on what has come before, what’s best for you.

It’s all about guessing, based on prior observations, who you are, what you will do, and what you will like.

These major companies have been hiring those with statistical literacy very heavily for more than a decade. Professors, who seek, money, fame, and power, know what these large potential sources of money, fame, and power want, and teach their products — their students –accordingly.

The superstructure of science changes as the infrastructure of the economy changes. The Go-Next philosophy of statistics, once the peasant stepchild of the serene Go-Up interpretation, now reigns supreme.

The unfolding victory of Go-Next Statistics matters much, much more than, say, the Copernican Revolution. The number of people whose daily conversations were actually impacted by Copernicus may have been a few dozen, all involved in the Papal-Academic complex.

How many times a day does Facebook’s decision of which news to share impact you?

How many times a day does Google’s decision of what sites to show impact you?

How many times a day does your iPod’s decision of what music to play impact you?

Now, back to Derbyshire.

Mr. Derbyshire was born in 1945. His training is in Go-Up statistics. It took a complete revolution in statistics to change his view of it. That view clearly changed in the last 2 years.

We’ve all lived thru the revolution of Go-Next statistics. Derbyshire realizes it. Steorts, clearly, does not.

There are two cultures of knowledge, the humanities and the sciences. Part of Derbyshire’s intention of writing “The Talk – Nonblack Version” appears to have been to highlight this. If so, I think he succeeded.

5 thoughts on “Go-Up or Go-Next?”

  1. Have you seen Derb’s response?


    I thought this comment was interesting considering your post:

    “I don’t know why people have so much difficulty thinking statistically, as we behave statistically all the time. The sky is overcast; I have to go out to an event where I’ll be in the open; I take an umbrella. If, after all, it does not rain, do I feel like an idiot for having taken the umbrella? Of course not. I yielded to my inner statistician. I went with the percentages. We all do it a dozen times a day. It’s statistical common sense. The trouble-free black neighborhood is the rain-free overcast day: It happens a lot, but take that umbrella.”

    Compared to your:

    “Astronomers use go-up statistics. Weathermen use go-next statistics.”

  2. Wow! I hadn’t seen that… Talk about hitting the nail on the head!

    Jonh, you’ve done terrible things for my modesty! 🙂

    I read Noah’s criticism, that Derb references as being the best he’s read of the column, and I agree it’s quite good. If I hadn’t driven to Canada for a conference yesterday, I would have tried to form a collegial rebuttal, but I’m not sure one exists…

  3. Was Derbyshire being stupid, or being smarter than when he wrote that piece for Taki? Even when ‘customized’, advertising (to use your example) isn’t about the individual–it’s just a more sophisticated method of increasing the size of the subpopulation labeled ‘customer’. If you buy, great; if not, just as well so long as someone else does.

    When a person runs for office, however, or walks up to you in the street, you care a great deal about what that particular person is going to do. Knowing averages and standard deviations on different races doesn’t change the reality that candidate A is a better public speaker than B or that C is a jackass or D is quite possibly insane. Knowing the odds of a member of a given demographic group attacking you in the street is no comfort if he actually does–and acting on that knowledge can actually increase that probability if you act on it by treating him/her like an enemy.

  4. Hey Michael,

    Excellent comment, but I need to double-down on one point.

    Mass customization /is/ about the individual — the end-state is to create a returning customer, but at every step along the way, bayesian influence is used to create as customized an experience as possible, often with scant information.

    This isn’t stupid — this is a consequence of the bayesian interpretation of probability, which is the main theme of this post.

    Of course probability doesn’t change prior reality — but it allows you to understand the prior reality better.

    You’d be right this would be personally dangerous if Derbyshire suggested aggressiveness as a result of these conclusions. He suggests the opposite, however: warriness.

  5. I seem to be more alert with this second reading than the first–odd after several hours of moving and light janitorial work:P

    You make a valid point about customized advertising statistics. While the concerns feeding their efforts may not have changed (my earlier point), computers do allow them to give a bit of thought to the needs of each customer, which can be good business.

Leave a Reply

Your email address will not be published. Required fields are marked *