How Science Works

Diane Ravitch is a labor agitator with a focus on the education sector. She is interesting to read if you care about union politics as it applies to education from a teacher’s (as opposed to student’s, parent’s, or nation’s perspective). As such following Diane Ravitch on twitter has the same sort of twisted excitement as, say, following a NAMBLA report on age-of-consent laws: they are definitely interested in the subject area, but for all the wrong reasons.

(I’m not sure if the effect of an individual child from a terrible education is worse than the effect of rape at a young age, though I suspect it is.)

In any case, Ravitch recently linked to a blog post, titled “Regents agree to give NY students data to limited corporation run by Gates and operated by Murdoch’s Wireless Gen.” The thrust of the article is that the not-for-profit Gates Foundation is providing funds to use test data to build more effective teacher assessments. The Gates Foundation wants to improve teacher quality, so that makes sense. And Ravitch is a labor agitator, so of course she is against this. So far, so good. Presumably drill-fitters would be against measuring the productivity of drill-fittings to a similar degree.

My friend Mark Safranski was curious about the story thouge. Mark mentioned, I don’t know if their data analysis is valid or reliable. Sounds like they don’t have a model yet but have a contract, and asked Why would this be figured out on the fly?.

The reason a model would be “figured out on the fly” is that this is how science works.

There is a cartoon version of science along these lines

Science is a method for understanding Truth. To understand Truth, a great scientist thinks deeply, and using the tomes he has read along with his powers of reflection, generates a Theory. Then, with great care, an elaborate contraption is created test the Theory. If the test works, the Hypothesis is Proven, and it becomes a Fact. Otherwise, the Theory is Wrong, and the cycle begins again.

This is a ridiculous view of how science works, and that Americans believe it is on prima facia evidence that our education system is deeply broken.

Science is a method for predicting variation. To better predict variation, scientists construct Theories, which are mental models that allow brute facts to be put in some sort of framework. For instance, the theory of Gravity explains the brute fact of an apple on a tree in one moment, and the same apple on the great in the next, into a narrative. Theories are opertaionalized using hypotheses, which generate specific predictions. So gravity on Earth can be operationalized as predicting that gravity acts like an acceleration that forces all object to the ground at a rate of 32.2 feet per second per second. Run enough experiments and you will begin to see this simple hypothesis mis-predict events, which will force you to generate other hypotheses. Eventually you will have a set of hypotheses which predict events enough to be useful to you.

Many individuals who hold the first view, on hearing the news about building a database of student test scores, would be confused. What is the Theory being tested? Why won’t the Great Scientist tell us? How will we easily know if he has discovered Truth, or he is Wrong?

Because this is not how science works, people who think this way believe in Pseudoscience.

Instead, actual scientists would be approach the work in a different manner. These scientists want to know what types of education best prepare students for life outside of school. They look at outcomes, such as health, income, years in college, degrees earned, social class, criminal convictions, and so on. To recognize that to the extent that ‘success’ exists, it is a latent construct that is only imperfectly, and with error, reflected in any one of these measures, using a Theory. These scientists also look at how one can objectively measure student achievement & teacher quality (two different things, certainly), also using Theories. A simple hypothesis is created, and tested on some data. Run enough studies, and the scientists will begin to see how this simple hypothesis mis-predicts events, which forces them to create more hypothesis. Eventually, the scientists have a set of hypotheses which predict student success enough to be useful to policy makers.

It is this process that Diane Ravitch and other labor agitators are deeply opposed to. A world where teacher quality can be assessed is one where bad teachers might be forced to become good teachers or be fired. This breaks worker solidarity, and means that some teachers will work harder, when any good labor organization wants to make working conditions easier for its members.

I believe in student welfare — and having a strong nation — more than I believe in labor agitation, so I support education reform, and oppose pseudoscience.

32 thoughts on “How Science Works”

  1. If we are talking scientists, they would also want a controlled environment, otherwise it is useless to examine variables, in a changing environment.

    “These scientists want to know what types…”

    The way this is worded it sounds like there are scientists on the opposite side of the teachers union who want conformity in teachers, so they can look at this scientifically. I think this assumption would be pretty far from the truth, if we would, as deep-throat said, “follow the money”. In fact, a person might even make a case that a union would be a good step in seeing that conformity in process happens and ask why this hasn’t happen.

    On the other hand, getting rid of the teacher’s union would be a good way to see the system”blown-up”.

    I am not sure in the climate we live in today that blowing up a system that is going bankrupt would be beneficial to the children or the nation.

  2. A controlled environment would be ideal, but is often impractical and unethical in human-subjects research.

    The current system is definitely harmful to the young and the nation.

  3. “impractical and unethical in human-subjects research.” Then what is the point in introducing science? Why don’t you use what corporations used to control the environment before they shipped the jobs to a cheaper market, the unions?

  4. “They look at outcomes, such as health, income, years in college, degrees earned, social class, criminal convictions, and so on.” If your scientist have no control over the environment that the outcomes come from, then it’s a matter of GIGO.
    A student with a long criminal conviction record opposite one that has a short? Pure Garbage. Very few of these great students of quality outcomes have gone to jail in the last great time-step of our whole society almost crashing.
    Whose quality of life are you talking about, Willis? Unless you look at the environment, then you are just talking pseudoscience.
    The problem you are describing as labor agitation is one in which knowledge and learning are perpendicular forces. They are forces not pushing against each other, as you supposed (labor and management), but at odds (a person of great knowledge may not be a good teacher). You may privatize that knowledge, and produce a system where all the teachers are good and the students above average, but I don’t think the outcome will be what you are looking for.

  5. Hey Larry,

    Your comment jumps around some, so I’m not sure the best way to address it.

    You seem to view control dichotomously: either we are in a controlled environment, or we have no control over it. Of course we can’t generalize to across range of parameters we haven’t observed, and with less control error increases. This does not make science pseudoscience, but of course it does impact the confidence we can have in results.

  6. The corporations involved in marketing to public education entities, including testing companies, are not run by scientists and are not doing “science’. That’s not their objective.

    There are companies that make (and/or administer) very high quality tests in psychometric terms in which you can have confidence that the results are valid and reliable. And then there are companies that offer testing products that…well….do not meet this standard but have the attraction of being markedly cheaper to purchase and can be administered by anyone off of the street or have students self-administer via computer.

    If these much lower quality tests are used as a rough “snapshot” of academic performance as a guide to adjust instruction or direct remediation resources, that’s somewhat useful if re-testing is part of the process. To make any life-altering decisions about students or teachers on the basis of the results of one of these substandard tests is unethical and invalid.

    And then at the bottom there are testing products at the level of which the State of Illinois was recently forced to term “catastrophic vendor failure” on forms submitted to the Federal Department of Education. Psychometric quality was not part of test selection criteria under tGov. Blagojevich’s ISBE.

    *Who* gets to decides what test is used is a key decision; as is *how* the test results will be used, but low quality tests used for purposes for which they are not designed will discredit the process

  7. zenpundit,

    Merry Christmas, and thanks for the comment!

    As we agree on nearly all facts, let me outline what I think are the political dynamics, and you can correct me if I am misunderstanding them.

    Major political actors are local school boards, teachers (both as unions and as political blocs), state governments, the federal-academic complex (DOE, NSF, etc), large-scale consumers of educated workers, and textbook publishers.

    The NCLB wave of ‘reform’ was primarily a winning coalition between state governments, the federa-academic complex, large-scale consumers of educated workers, and textbook publishers against local school boards and teachers organizations. NCLB disempowered both school boards and teachers organizations.

    Different elements of this coalition had different motives.

    1. Large-scale consumers of educated workers existed in a bizarre situation in which it is impossible to fill large numbers of positions, and the blame for this was on the lack of preparation for STEM careers by public schools.
    2. The federal-academic complex’s objectives, the implementation of quality control on schools, easily aligned as it promised to solve the problem of the consumers.
    3 State governments were desirous of power for its own sake, and easy to turn against lower levels of government.
    4. Text book publishers were mechanistically trying to ‘move up the value chain,’ and sensed the opportunity to sell more goods & services to richer customers at a greater scale.

    On the losing side were school boards and teachers organizations, who were not facing a crisis from their perspective (schools were exercising powers and teachers were collecting political rent), were unprepared for the attack, unable to act empathetically to members of the winning coalition, and unable to articulate an alternative path which would have been able to break the winning coalition. Moves apparently never even considered, could have been

    1. Better preparing students for STEM careers
    2. Implementing bottom-up quality controls that bypassed the states
    3. Emphasizing the importance of state government in creating local standards
    4. Purchasing more services from text-book publishers in a way that bypassed the states and the federal-academic complex.

    (Obviously some of these steps are contradictory to others, though implementing any of them may have been enough to stop NCLB early on.)

    Of the losers, school boards appear to be generally emasculated, leaving teachers as the only voice surviving. I take Diane Ravitch to be a representative of this voice. Teacher blocs, though, are left without a rational direction or any hope of victory. Like the South Vietnamese in 1974, they are left to loot what they can with time running out, and without a path to victory. Perhaps there will be Divine (Vox Populi) Intervention and they will be saved. Perhaps not.

    As the teacher blocs are ground down, the serving winners will naturally turn on themselves in how to divide the victory. I think we both share the impression that of the four winning powers, the textbook publishers both approach the blinkered rentier position held by teacher blocs, with the states not far behind. Ravitch’s strategy appears to be to focus on this (weakest?) member of the winning coalition in hope of a popular groundswell against the winning coalition in general.

    My preference would be to articulate to large-scale consumers and the federal-academic complex specific elements of State and Publisher over-reach, and attempt to win the debate through honest (elite) engagement rather than rabble-rousing (popular) politics.

  8. “Large-scale consumers?” Debate through honest (elite) engagement rather than rabble rousing (popular) politics? Kinda of off-topic isn’t it? If there is something called Pseudo-elitism I think you got a bad case of it. I don’t mean to answer for zenpundit here, and I don’t think anyone would accuse me of being elite, but if you want to talk about honesty, then I think you need to re-investigate some of your statements.

    Take this one for instance: “Large-scale consumers of educated workers existed in a bizarre situation in which it is impossible to fill large numbers of positions, and the blame for this was on the lack of preparation for STEM careers by public schools.”

    I think, to be honest, your so called large-scale consumers of educated workers never planned on using the bulk of these workers from the US public education system in the first place. Second, our education system is student driven, which you seem to leave out of your to-do list.

    What our system of education does is produce dreamers (what could be called visionaries), which these large-scale consumers of education workers need. on the other hand, the other education systems produce workers (cheep) that the large-scale consumers of education workers want.

    What was bizarre was that the large-scale consumers of education workers were denied their usual supply of cheep labor (something about a war or two), which they were accustomed to.

    It now comes down to having to destroy what you need to get what you want.

  9. Hi Dan

    You had a fairly accurate political analysis of the correlation of forces, as the Marxists used to say. You are particularly right on in saying the union leadership was incapable of dealing with this challenge and in denial (minus one guy in the 90’s, Bob Chase, who saw all of this coming and tried to reform the NEA to no avail. His current successor is a fool and a potential sell-out to find a comfortable place for himself)

    A few caveats.

    1. Larry is right that you are not quite correct re: consumers. Part of their predicament is not the public schools but the wholesale abandonment of in-house training during the 80’s as a basic cost business they subsequently expected society to pick up for them.

    This was feasible back then only because late 70’s- early 80’s deindustrialization left a slack labor pool of highly skilled, laid-off tradesmen in niche specialties who business could pick up at progressively lower wages. Those folks, vestiges of the industrial revolution America, are now too old to work. By “training” I do not mean basic literacy/numeracy – that is a school issue – I mean training already skilled workers further to acquire the specialized industrial skills that are required for production instead expecting to find these hen’s teeth individuals just walking in their door.

    The answer corporate America is turning to is not to set up training programs again but to promote a political campaign to raise barriers to entry into college and demonize the humanities so that individuals with higher IQs in large numbers, for lack of other options, will go get specialized training in the trades on their own dime after high school (which will also drive the high wages currently paid for these increasingly rare skills downward – a “twofer” for them at the expense of everyone else)

    2. A minor point, but the old-line “Choice”, Milton Friedmanesque “free market in education” ed reformers have been thrown under the bus. Ed reform today is no more about unleashing a million inefficient church basement schools to prepare kids for the 19th century in unlimited competition than it is about a quest for “science”. The people you term “winners”, or at least some of them, need reasonably large aggregated public education entities to work with in order to maximize profit, but not so large that the entities have countervailing economic and political power to bargain effectively. If anything, corporate ed reformers would like to *reduce* the net number of school districts in states with a strong tradition of local vice county control and are funding politicians to promote school consolidation bills.

    Oh, and while they do not know it, the homeschoolers are going to be thrown under the bus too if the public ed system can be seized ( i am no longer sure it will be – I give it 50-50). Milking the revenue flows won’t work optimally if every middle-class parent with a bright kid can take advantage of liberally written homeschooling laws to withdraw their child from the corporately managed public school district. those will be tightened up and the hard-core homeschoolers in urban states will be harrassed legally.

  10. Mark,

    Excellent comment, as always. As (yet again) we are largely in agreement, I’ll focus my remarks on the roles of Large-Scale Consumers and Publishers, as I think those are the main areas you address.

    You are correct that corporate training has declined remarkably. This is part of the larger trend away from the state-capitalist State-Owned Enterprises & State-Controlled Enterprises that were common in China, the United States, Europe, and the Soviet Union following in the post-war period. For approximately 30 years there was an assumption that large enterprises could not fail, and that these large enterprises should construct mini-me welfare states encompasses health care, education, retirement, and other services.

    (One of the things that strike me as bizarre at Diane Ravitch’s rhetoric is her attack on ‘billionaires’ giving money to fund schools, when these should be public enterprises. She appears to be completely ignorant that under the ancien regime, many of the responsibilities of public educaton were expicitly ‘privitized’ to the SOEs/SCEs. As I think we agree, we are living in a moment of high-degree of direct government control over education, as opposed to a low-degree.)

    Public schools would be under a lot less pressure if we would go back to an area where they were not expected to train workers, but only prepare students for training a workers. I don’t see us coming back to that, though.

    Regarding Publishers, you are not wrong in understanding their economic motives, but I have been continually surprised by your elevation of Publishers over all other actors. It seems you weight Publishers as more powerful than all other stakeholers combined, and believe that Vox Populi divine intervention is required. Is this an accurate take?


    The US system is terrible at training at STEM. We are a laughing stock, our graduates are denigrated, and the inability of teachers to pro-actively recognize this problem is why we are at the current situation.

    Nonetheless, our system is rather wonderful at producing leaders (which you call dreamers & visionaries). Very good, in fact, though one might question how much of this is a product of the education system per se as opposed to the great degree of personal & economic freedom Americans experience very early in their lives.

    I think there’s a fair question of whether these are mutually-exclusive goals, but by recognizing both, you can be empathetic to the requirements of Large-Scale Consumers. When Large-Scale Consumers hire from abroad, there are real costs to this, often relating to leadership potential.

    Why don’t teacher organizations try to change the debate to focus on what they do well? The stupidity of their own leadership, I would think.

  11. “Regarding Publishers, you are not wrong in understanding their economic motives, but I have been continually surprised by your elevation of Publishers over all other actors. It seems you weight Publishers as more powerful than all other stakeholers combined”
    I am putting publishers and test companies together with those ed reformers who would like to hold management contracts as private for-profit companies to run public school districts (or parts of them in very large urban districts like NY, Chicago, Los Angeles etc) either directly or as charter chains (not “mom & pop” charter shops – these will also gt the heave-ho). They have a common interest in seeing school districts in fairly large-medium scale where the revenue base is at least in the high tens to low hundreds of millions of dollars per annum but small enough that they have no real bargaining power, political connections or sophistication . Larger than that – a regional or large city district with billions of dollars and the ability to hire top notch legal help – and leverage shifts decidedly to the school district, or it’s Mayoral political overseer (Rahm, Bloomberg). Unions have more leverage on that scale as well. You can fire and quickly replace 100 -500 striking teachers but not 140,000 or even 10,000

  12. zen,

    Thanks for the comment. It matches how I understand your position.

    ‘Publishers’ (which I use to include the groups you include) and teachers are in a classic stand-off over who can milk the public school system to provide excess profits.

    The States care about controlling the process, Large-Scale consumers care about outcomes, but Publishers v. Teachers are the two groups seeking rent payments from their political power over schools, and it’s probably natural at this point in time for each to see the other as the ‘real’ enemy (as opposed to simply an outside power that can be manipulated but can also toss-up interference).

    I don’t think that necessarily implies that Publishers are particularly powerful, though, but that they are empathetic to the needs of States and Large-Scale Consumers.

    If Teachers, and not Publishers, had been more empathetic when reform was still embryonic under GWHB and Clinton, the debate would look radically different.

    There is still time, but union leaders, as well as agitators like Ravitch, seem intent on digging their own graves. If schools could produce educated worker /or/ flatter State power, you probably would have a stand-off in this debate, and the Vox Populi’s sympathy for ‘our teachers’ would carry the day. Instead, Teachers seem intent on a Red Army Faction strategy of killing off everybody but true believers, and waiting for History to save them.

  13. ZenPundit wrote…
    “The corporations involved in marketing to public education entities, including testing companies, are not run by scientists and are not doing “science’. That’s not their objective.

    There are companies that make (and/or administer) very high quality tests in psychometric terms in which you can have confidence that the results are valid and reliable. And then there are companies that offer testing products that…well….do not meet this standard but have the attraction of being markedly cheaper to purchase and can be administered by anyone off of the street or have students self-administer via computer.”

    When I taught in Colorado, we administered the CSAP. It took 2 weeks to administer and was not a “bubble test.” Originally, it was designed by educators who (later after seeing what was done with it) were horrified by its use. It was criterion-referenced and the scoring was done subjectively using rubrics.

    The CSAP was published by McGraw-Hill. The state contracted that. Then, McGraw-Hill easily sold their instructional materials to the districts since they made the test. This is no different with Pearson. Pearson publishes the FCAT in Florida and then easily sells their instructional materials to the districts. It’s state by state. I’ve tried to find a document detailing which publisher makes which state’s assessment. It’s one or the other.

    When I was teaching, I was curious about who was scoring these tests. The test was almost all short answer, essays, etc. I did some research and found it was temporary hired help. I also found that the people training and managing the help were also not educators. I have an old job description somewhere to show this.

    I knew that the test’s validity was questionable since the scoring was done by temporary help who were not educators and were not properly trained. What I did not know was that these tests were scored in a factory-like way in 60 seconds. Administering this test which eventually closed our school year after year and learning this is so upsetting to me.

    Knowing what I know about testing and what it entails, I am more than a little concerned Common Core Standards are being rapidly adopted by the states and that $150 million has been invested into a CC assessment.

  14. Hey dellaccs,

    Thanks for the awesome comment!

    My bias is that teacher’s shouldn’t be writing their own, any more than teachers should be building their own cars or architecting their own houses. Testing is hard, and it’s a distinct profession from teaching. (Of course, what is being tested and what is being taught should be aligned!)

    That said, the behavior you report from the publishing company is outrageous. It would be a post by itself just to list the things that are wrong with the way they seem to be doing it!

    Now then, walking away from that deep emotion I’m feeeling about their malpractice, let me say what they are doing right, and why their efforts are valuable.

    I don’t believe that we are educating Americans appropriately. Large portions of critical industries are in the hands of foreigners because of the failures of US education. These failures are deep and systematic — all stakeholders share blame — but must be addressed.

    No matter what strategy we think is effective — whether paying a lot of money or a little — we need to be able to measure the performance of the components of our education system (teachers, principals, districts, publishers, etc). Getting this testing infrastructure in place is the hard part, partially because teachers (as a political bloc) are against their performance being measured. Publishers, without the obligation to defend their own labor force, are fine with their performance being measured (if more quality is demanded, they would just pay more for quality and pass those costs on to districts).

    The anecdote you shared is enraging. So much is being done wrong. But it is even more distressing that this wrong attempt at testing is better than the anti-testing stance of teachers, who as a political bloc are against any form of standardized testing whatsoever.

  15. The use of bad testing methods, though, encourages the teachers to dig their heels in on testing in general. As long as they see themselves a besieged minority, for this and other reasons discussed above, professional radicals are going to find them easy converts.

  16. Hey Michael,

    Thanks for the comment. I agree, though with the following provisos

    1. Teachers do not have the training to distinguish good testing from bad
    2. Teachers are a labor bloc are against any form of standardized testing, regardless of quality
    3. The only form of testing methods that teachers as a bloc support, unstandardized subjectively-graded testing, is a form of bad testing

  17. About half-way through that essay I mentioned in my last comment on the testing thread. Had this quote from an English teacher at the high school being used for an example:

    “I know that they don’t learn the English in this way, not really. In fact, most of them don’t understand the Japanese [that they translate into]. I also know that they don’t need to study English. But they need to pass because they need to graduate, and if memorizing the lesson bit by bit will allow them to do that, that’s okay with me. It’s [the curriculum] all in pieces anyway, so it’s not really very different [from the rest of the lessons]. For a school like Musashino, it’s what we have to do. Otherwise, everyone in the class fails. For students like ours, that means they cannot get a job. They at least have to graduate. I’m not going to fail them.”

    To summarize the surrounding paragraphs, students at that school have to study for college exams they will never take. So there’s no real penalty for failure to absorb the material on their part unless, as this teacher states, it leads to them not getting their state-mandated sheepskin.

    It’s not enough for the teachers to understand good testing. The federal-academic complex establishing the requirements have to understand it too:P

  18. Very true, Michael.

    Teachers are facing a fork between being professionals [1] and being drones [2]. Both cases require good tests being pushed through the federal-academic complex, but if we’re into drone teachers, they only need to be able to read the teaching-scripts given to them…


  19. Biggest issue would be privacy. I could have got some bad reports on such a system because in some classes I was bored, yet I went on to a good degree at a top university and a successful career. What damage if some bad grades earlier in my life had been allowed to stay with my report for life and hinder my progress later.

  20. Richard,

    You have a good & reasonable concern re: privacy. Certainly there’s a balance to be bad between making good evaluations and protecting individual test-takers. I’m not aware of the specifics in this area, but I have friends involved in test evaluation, and I know privacy is often a topic of concern to them.

Leave a Reply

Your email address will not be published. Required fields are marked *