The Wary Student, Part 5: The Experiments

The experimental set-up was a variation of Alford & Hibbing (2006a) and tdaxp & Johnson (2007). 181 undergraduate students in the political science department of a large midwestern university were recruited to be participants over a period of six days. The experiment was designed and prepared with drawing software, the Audacity sound editor, and the perl programming language and finally implemented with MediaLab research software. Participants answered survey questions and played economics games. The experiment was contrived to simulate interaction with fellow classmates in distance education group work

Two games were studied as part of this research. Before the participants played these games, they were randomly assigned into a high-cognitive load or a low-cognitive load condition. The experiment differed from both tdaxp & Johnson (2007) and Alford & Hibbing (2006a) in that the tasks were framed as part of a group project, instead of as an economic game. Framing effects have been observed before (Larrick & Blount, 1997), and may have their effect, because ultimatum game performance chances depending on the norms of a people (Henrich, et al., 2005) or a workplace (Kay, Wheeler, Bagh, & Ross, 2004).

First, positive cooperation is studied. Participants were seated at computers and told they were testing new interfaces for distance education. They were told that their actions in the first part of the experiment will only effect the grades of other students. However, they would have an opportunity to gain additional extra credit at a later part of the task. The participants were instructed that the students they were assisting was at another campus, and that all identities would remain anonymous. After a structured introduction, the students were given a series of mathematical problems to solve both for themselves and for the other students. Which problem would help which student was clearly labeled. Positive cooperative behavior is measured by the number of altruist attempts participants made to solve the other student.

The students were then informed that their task was over. They were informed that another portion of the experiment was to measure cooperative behavior in distance education classes. Unbeknownst to the student, the second portion of the experiment would be an ultimatum game, “where one of the players can firmly commit himself in advance under a heavy penalty that he will insist under all conditions upon a certain specified demand (which is called his ultimatum)” (Harsanyi, 1961, 190).

The ultimatum game has been studied in educational settings (Stanley & Tran, 1998; Stodder, 1988; Oxoby, 2001), for professional populations (Bethwaite & Tompkinson, 1996), and across the world (Bowles & Gintis, 2000; Gowdy, Iorgulescu, & Onyweiwu, 2003). It has been summarized as follows:

In the Ultimatum Game, two players are offered a chance to win a certain sum of money. All they must do is divide it. The proposer suggests how to split the sum. The responder can accept or reject the deal. If the deal is rejected, neither player gets anything. The rational solution, suggested by game theory, is for the proposer to offer the smallest possible share and for the responder to accept it. If humans play the game, however, the most frequent outcome is a fair share. (Nowak, Page, & Sigumd, 2000, 1773)

The participant was then informed that the other student was given the opportunity to split extra credit points with the participant. These extra credit points were designed to reward cooperative students. The participant was informed that the other student believed that a 4-to-1split of extra points was fair. If this was accepted, the other student’s point total would be raised by 4 extra credit points while the participant’s score would be raised by only one. Alternatively if the participant refused, neither would gain these additional extra credit points. Rejection of the unfair split, an altruist act as it reduces personal gain and potentially teaches the other student a lesson that the participant could not benefit from, is defined as neutral cooperation.

Next, participants were informed they would be able to “punish” the other student with points they had earned for attending the experiment. If the participant had accepted the unfair split, then he or she had potentially six points to punish with. Alternative, if the participant had behaved neutrally cooperatively, only five such points would be available. The magnitude of retributive punishment measured is defined as negative cooperation.
Following the completion of the experiment, participants were debriefed and thanked for their time. Deception was used in the study, so that all participants received ten points of extra credit regardless of their performance.

Substantively, three hypothesis were made: increasing cognitive load will alter positive cooperation, increasing cognitive load will alter neutral cooperation, and increasing cognitive load will alter negative cooperation. In statistical notation, the null hypotheses for these can be written as follows:
Hpositive,0,: μpositive cooperation,high cognitive load = μpositive cooperation,low cognitive load,
Hneutral,0: μneutral cooperation, high cognitive load = μneutral cooperation, low cognitive load
Hnegative,0: μnegativecooperation,high cognitive load = μnegative cooperation, low cognitive load.
Because three separate questions are tested using the same sample variation, a Bonferroni adjusted was made. Therefore, results are reported using both .05 and a .017 (α = α / k = .05 / 3 = .017) levels, and .01 and .0033 ( α = α / k = .01 / 3 = .0033) levels. Through the experiment, the independent variable (IV) is cognitive load condition, and the dependent variable (DV) is the type of cooperation being examined.

The Wary Student, a tdaxp research project
1. Abstract
2. Cognitive Load
3. Cooperative Behavior
4. Method
5. The Experiments
6. Hypotheses
7. Main-Effect Results
8. Interaction-Effect Results
9. Discussion
10. Future Research
11. Bibliography

Avandia has a moderate-to-very-large practical effect on heart failure

Home, P.D., et al. Rosiglitazone evaluated for cardiovascular outcomes — an interim analysis. The New England Journal of Medicine. 5 June 2007. Available online: (via Medical News Today).

Avandia is a drug designed to treat Type II Diabetes. Type 2 Diabetes leads to heart attack, death, and a lot of other bad things. A safe drug that treats it would be very good. Many people think that Avandia (rosiglitazone maleate) is that drug. However, a recent article in The New England Journal of Medicine reported that Avandia has a large-to-very-large effect on patient death. Because this is important news, a new article was rushed to the New England Journal that reported results-so-far of a study that’s not completed.

The results section is statistics-y:

Because the mean follow-up was only 3.75 years, our interim analysis had limited statistical power to detect treatment differences. A total of 217 patients in the rosiglitazone group and 202 patients in the control group had the adjudicated primary end point (hazard ratio, 1.08; 95% confidence interval [CI], 0.89 to 1.31). After the inclusion of end points pending adjudication, the hazard ratio was 1.11 (95% CI, 0.93 to 1.32). There were no statistically significant differences between the rosiglitazone group and the control group regarding myocardial infarction and death from cardiovascular causes or any cause. There were more patients with heart failure in the rosiglitazone group than in the control group (hazard ratio, 2.15; 95% CI, 1.30 to 3.57).

Several results are reported here. The most important to consider are practical signifiance and statistical significance . From my statistics notes:

Statistical significance is concerned with whether an observed mean difference could likely be due to sampling error
Practical significance is concerned with whether an observed effect is large enough to be useful in the real world

For instance, imagine that you wish to be more productive, so you buy a new computer . You notice that you get twice as much done in an hour with the computer than without it. The practical significance would be very large (double!). However, you didn’t look at enough people to reject the notion that maybe it was just a fluke. So there would not be statistical significance.

A similar thing happened in this study. The last part of the quoted paragraph (“hazard ratio, 2.15”) means that, practically speaking, for every heart attack for diabetes type 2 patients who aren’t taking Avandia, patients taking Avandia have 2.15 heart attacks. However, the study did not meet statistical significance — the new research did not look at enough people to say whether or not this very large practical effect was due to chance or not.

A problem with the study — that the authors note — is that they are reporting their results too soon. (They are doing this because there is talk of forcing Avandia off the market, which would effect all patients who currently take Avandia and obviously hurt GlaxoSmithKline, the company that makes it.) I have heard anecdotes that one of the side-effects of Avandia is “preamature-aging.” If this is true, the negative effects of Avandia would get worse and worse over time. Thus, future research may go from the current two (where all find practical significance, but only one finds statistical significance) to a situation where all find statistical significance.