Stereotyping, and Rare but Important Events

Phil Arena has an interesting but problematic piece up at Duck of Minerva, entitled “Bayes, Stereotyping, and Rare Events.” The substantive topic of the post is a recent survey of Muslims that I’m not too interested in. But Phil uses statistics to mask a deeply flawed and irrelevant conclusion:

Put simply, the probability that you’d be mistaken to assume that someone who belongs to group Y is likely to commit or have committed act X simply because most such acts are committed by members of group Y grows exponentially higher as X becomes rarer. The reason you should not assume that a person is a terrorist just because they’re Muslim, then, is not just that this is politically incorrect and likely to offend delicate liberal sensibilities. It’s that it’s almost certainly incorrect, full stop.

The first and last sentences in that paragraph have almost nothing to do with each other. Phil’s conclusion is irrelevant, and the “full stop” leaves the most important part of the conclusion unsaid.

And Phil’s not alone in such a mistake. Take for example an recent statement on the NPR program “Tell Me More” by Fernando Vila. Fernando is responding to a statement that a disproportionate fraction of violent crimes in New York City are committed by African Americans:

VILA: Well, I mean, the notion of paranoia is a good one and Mario’s statistics actually sort of feed into that – into this culture of paranoia. I mean, the vast majority of black people are not committing crimes.

VILA: You know, it’s like to say, I don’t know – the vast majority of hosts on NPR are white males. That doesn’t mean that every time I encounter a white male on the street I assume he’s a host of NPR. You know, it’s just a backwards way of looking at it

Phil and Fernando make exactly the same mistake: false assuming the cost of a “false positive” (accidentally marking someone as suspicious) is the same as the cost of a “false negative” (accidentally marking someone as not suspicious). But the truth is all errors are not equal.

The cost of a mistake is a function of the severity of the mistake.

Is the cost to society of 1 false positive (falsely placing an individual under suspicion of terrorism) the same as the cost to society of 1 false negative (falsely removing suspicion from an actual terrorist)? No, of course not, but Phil’s post is based that on fallacy. Otherwise his conclusion makes no sense.

There is a serious question as to where we should become indifferent to the trade-off — 10:1? 100:1? 1:1000000? — but it is certainly not 1:1.

Likewise, Fernando’s statement on NPR is irrelevant. While the consequence of guessing an individual’s employment status at NPR might be 1:1 (few would care either way), the costs of falsely assuming someone would attack you is far less than the cost of falsely assuming an individual will not attack you. Again, there is a question of trade-offs — 1000:1, 10000:1, 1000000:1? — but the cost of all errors are not identical.

Now, obviously Phil and Fernando had different motives here. Phil’s obviously trying to popularize some basic statistics, while Fernando is doubtless ignorant of basic statistics. But in both cases an unwary audience will be led astray into thinking all errors are equally important.

10 thoughts on “Stereotyping, and Rare but Important Events”

  1. Hey Curtis,

    Thanks for the comment.

    Could you flesh out your example a bit more? TSA screenings are a net-killer in society, for while there is no evidence of a reduction in terrorist attacks because of them, there is evidence of diversion away from flying to the (way more dangerous) driving.

    Granted, I think you could make an argument that 1,000 (or whatever) road deaths is preferable to 1 (rounding up) terror death, but is that the argument you are making?

  2. Dan, blaming any single traffic death, or multiple traffic deaths, on the actions of TSA screeners is rather ludicrous, an example of counterfactual thinking. You would first need to show that any increase in traffic fatalities was directly correlated with individuals who had chosen to drive because they felt that the TSA’s screening methods were too intrusive—i.e., that these individuals specifically were responsible for the net gain in traffic fatalities. (You would also have to rule out other factors, e.g. increased population, economic drivers that have increased road traffic, and so forth.)

    And regardless of the above, I’m pretty sure you knew exactly what I meant when I left my first comment here.

    What I find rather disingenuous is the inflated outrage over TSA screening techniques coming from those who find absolutely no problem in harassing individuals on the basis of their ethnicity or skin color because such a very, very, very small number of individuals of that ethnicity or skin color commit violent crime.

  3. Hey Curtis,

    I’m not sure what you mean by “counterfactual thinking.” You seem to mean “policy analysis,” in which case, yes, that’s the whole point.

    The questions of moral or legal “blame” is irrelevant to the question of public policy impact. If your goal is to kill Americans, c.p. you should support the TSA. If not, the reverse.

    Your last paragraph seems like a vague insult that isn’t worth responding to, which you wrote in favor of clarifying your original comment.

  4. Dan,

    ” A counterfactual thought occurs when a person modifies a factual antecedent and then assesses the consequences of that mutation.” —A succinct description of the type of thought process from the Wikipedia article on counterfactual thinking.

    And so, your apparent assertion that TSA screenings have led to more traffic deaths would serve as an example, because it could be restated, “If only the TSA had not implemented such intrusive screening procedures, there would have been fewer traffic fatalities.”

    You seem to have made that claim in support of your “analysis” of the ratios of cost vs benefit comparisons for screenings (profilings) which return a false positive or a false negative—”the costs of falsely assuming someone would attack you is far less than the cost of falsely assuming an individual will not attack you.” This is an interesting starting point for weighing preemptive measures in general; but unfortunately, you use the analysis to bolster preconceptions, biases, while ignoring its application elsewhere.

    For instance, even if, for sake of argument, I granted you your claim that TSA screenings have somehow increased the incidence of traffic fatalities (and I’d be very, very generous to grant you that claim), nonetheless some people do travel by air, some people will continue to travel by air, and a false negative search procedure that offends the propriety of some travelers could be said to be a very low cost to pay for preventing the deaths of however many travelers should an explosive device or highjacking device make its way on board a flight via a determined terrorist.

    The argument you have given could also shore up an argument in favor of stronger gun-safety laws. Sure, many gun owners might be slightly more inconvenienced having to wait for longer, more stringent screening procedures; many gun owners might be inconvenienced or offended by having to insure their names are listed on gun-owner registries; etc;, but such inconvenience weighed against the results of false negatives (the assumption that any gun purchase at a gun show not subject to such screening procedures) might “prove” a la your argument that stringent gun safety laws are “worth it.”

  5. “a false negative search procedure” above should have been “a false positive search procedure” —i.e., a procedure which assumes the possibility of a positive result.

  6. Hey Curtis,

    As I suspected, you use the term ‘counterfactual thinking’ to refer to policy, or even applied social science. If you don’t like the uncertainty involved in judging policy choices in an uncertain world, that’s your prerogative, but it strikes me as ridiculous.

    Here’s a 2005 study on the TSA’s traffic death toll [1]. Feel free to critique the methodology in substantive ways.

    To answer your specific point regarding ratios, the TSA would be bodycount-neutral if screening prevented one air disaster per month. Obviously the ‘cost’ of a terrorist death is higher than the cost of an accidental death, though, so determining where the reasonable risk lay is another discussion.

    Certainly weigh risks could impact gun laws either in favor of limiting access (most deaths by firearms are accidental) or in favor of liberalizing access (a higher proportion of guns among potential victims lower the NPV of a robbery). As with the TSA example above, though, the cost to society of a criminal death is far higher than the cost to society of an accidental dealth, so there’s a real policy question there.

    of course, one might argue that when a right is enshrined in the Constitution, this intentionally increase the importance of ideological –as opposed to practical — reasoning with relation to it.

    PS: Glad you like the ‘reactionary monarchist’ line. It was thrown at me as an insult a few weeks ago ;-)

    [1] http://papers.ssrn.com/sol3/papers.cfm?abstract_id=677549

  7. “If you don’t like the uncertainty involved in judging policy choices in an uncertain world, that’s your prerogative, but it strikes me as ridiculous.”

    Dan, there is a substantial difference between acknowledging the uncertainty and failing to do so. Consequently, there is also a difference between “liking” a method by which analysts ignore the reality of uncertainty and “disliking” their willful ignorance of the reality of uncertainty. This is not a dislike of uncertainty, as you have presumed, but rather a dislike of the manner by which others choose to ignore uncertainty.

    An abridgement of the role of uncertainty which takes the form of counterfactual thinking in analysis—this, via confirmation bias, or assuming definite consequences to actions on the basis of preconceptions that fit an ideological narrative, particularly also by ignoring key variables in favor of isolating and highlighting an ideological target [1]—by constructing “100% policy” intended to forgo consideration of uncertainties would be ridiculous.

    For instance, when you say, “Here’s a 2005 study on the TSA’s traffic death toll,” you are either intentionally misleading or have accidentally been misled by your bias. The paper is in fact a study on the immediate aftereffects of the 9/11 attacks and relates far more to the fear of flying that resulted from those attacks, in the months immediately following 9/11, than on “the TSA’s traffic death toll” as you have put it. True, the authors of that study include a consideration of the effects of the inconvenience of security procedures initiated by the TSA; but they do not isolate the TSA’s procedures as the primary cause of those increased traffic fatalities.

    Indeed, the authors seem to include the possibility of a “TSA effect” mostly as a way of addressing the uncertainty of their conclusions, because the increase in traffic fatalities cannot be attributed to the fear of flying, alone, to the exclusion of potential reactions relating to the inconvenience of flying. No doubt, this uncertainty arises because of the impossibility of isolating specific thoughts and feelings occurring within travelers’ heads at that time in history. This is an uncertainty you seek to abridge by labeling the study a “study on the TSA’s traffic death toll.” To what degree was the fear of flying an effect? You ignore that question in favor of supporting an ideological bias.

    Regardless of the question of relative weight of these causes, the authors of that study also highlight the fact that distance from the 9/11 attacks, in time, correlated to a decrease in the “9/11 effect” as they so broadly term it: “These results suggest that the 9/11 effect was not long-lived….” In short, the paper addresses a short period following 9/11. You would stretch that period into perpetuity beginning from the days after 9/11 while ignoring the effects of a fear of air travel caused by the attacks on 9/11—i.e., you would draw a universal from a paper whose conclusions are extremely context-based and narrow.

    Interestingly, and more germane to the topic at hand (your formula for weighing risks between false positives and false negatives), the authors introduce the TSA effect by calling that potential effect ironic:

    “Second, the 9/11 effect may be attributable to the inconvenience of flying post-9/11, a regrettable irony given the inconvenience resulted from measures taken to prevent (air travel) deaths from terrorism.”

    This is ironic in three ways.

    First, the obvious surface irony in the fact that the TSA’s efforts to increase safety may have led to more ground traffic which is statistically a less-safe form of travel. (Again, this is “may” rather than “certainly has led.”)

    But second, and more important in my opinion, is the irony that individual travelers’ assessments of the relative safety between flying and driving may have fallen by the wayside or become irrelevant when they weighed their own safety against the convenience/inconvenience they might experience.

    Third, and most germane to the topic at hand, those travelers’ assessments may be an instance of the unreliability of your argument concerning the weighing of risks. The authors explicitly address the phenomenon earlier in their study:

    “First, the 9/11 effect may capture the fear of flying. This fear may cause travelers to substitute driving for flying if they place too much weight on the possibility of another terrorist attack, as suggested by prospect theory (Tversky and Kahneman, 1979). Consistent with this overweighting of low probability events, travelers’ behavior may also reflect what Sunstein (2003) calls ‘probability neglect,’ a condition in which large variations in the probability of an event occurring have no bearing on people’s choices. His results show that ‘when strong emotions are involved, people tend to focus on the badness of outcome, rather than on the probability that the outcome will occur.’ Similarly, Slovic (1987, 2000) documents the distortion of probability assessment when the risks are unfamiliar and beyond control, termed ‘dread risk.’ Taking a different approach, Rubinstein and Becker (2004) persuasively argue and provide empirical evidence to support the notion that consumers’ utility from travel (and other activities) may be influenced by the fear of terrorist attacks, regardless of whether or not an attack actually occurs.”

    BTW, I would also look at the dread of TSA screenings through the above lens—not just dread or fear of flying. While a large majority of air travelers may feel somewhat inconvenienced by TSA security measures—I am assuming here—I would guess that very few, having gone to the airport for a flight, suffer what have been highlighted in the media as the most extreme coercive or embarrassing incidents involving TSA agents.

    In other words, regardless of the degree that fear of flying vs dread of TSA screenings has had on an increase in traffic fatalities, those additional traffic fatalities have resulted from individuals’ assessments of false positives—that they MAY WELL DIE on a flight or MAY WELL SUFFER EXTREME INDIGNITY AT THE HANDS OF TSA AGENTS. In this case, contra the argument in your blog post, a false positive (“the flight is doomed” or “the TSA agent is going to fondle my privates”) is more dangerous than a false negative, to the degree that traveling by ground is more dangerous than traveling by air, statistically.

    [1] Purposely ignoring variables in order to increase the importance of a favored variable is one sign of counterfactual thinking. See: http://www.curtisgaleweeks.com/#Counterfactual

  8. Hey Curtis,

    Thanks for the follow-up.

    It sounds like we’re in broad agreement, that specific policies (such as TSA screenings) should be weighted on the basis of actual costs, including death tolls, and that the information to properly do this is often limited.

    As I wrote before, ” I think, that the solution to prejudice isn’t lack of prejudice — it’s more and better prejudices.” [1] That’s certainly the case in this specific example, as well.

    [1] http://www.tdaxp.com/archive/2013/07/23/prejudice-and-bias.html/comment-page-1#comment-1837382

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>