Tag Archives: Publishers

What Good Tests Look Like, and Why We Don’t Have Them

Back when I was a teacher-educator, I would teach pre-service teachers what a good test looked like. This was so they could recognize one when it appeared, and when their students received standardized test scores, they could explain them to parents. I used the acronym “RSVP” (taken from this pretty good educational psychology textbook) to emphasize the quality of good testing. “RSVP” has implications for education reform, not just one-off tests.

RSVP stands for Reliable, Standard, Valid, and Practical.

  • Reliable means the test gives the same score each time. A reliable test should give the same score even as testing conditions change. It is of course hard to demonstrate the reliably for ‘high-stakes’ testing that takes place once a year. It would be far better for testing to occur on a quarterly, monthly, yearly, daily, or (preferably) nearly-continuous basis. If a standard scores high on a test over a subject-area on time, and low the next, and the same subject area is being tested at the same level, the test would have low reliability.
  • Standard means that every student gets the ‘same’ test scored the ‘same’ way. Note this doesn’t mean something foolish like, ‘Everyone gets the same test on 8 AM the same day of the year.’ It does mean that subjective “portfolios” — the only type of ‘testing’ that teachers as a politica bloc oppose — are wrong-headed. “Portfolios” aren’t a form of RSVP testing at all, but serve as a political attempt by teachers to prevent any measurement of their quality as teachers in inspiring learning.
  • Valid means the test actually measures what it is supposed to measure. So a reliable and standard testing on the Revolutionary War, composed entirely of geometry problems, probably isn’t valid. Likewise, a test of working memory capacity which would show blacks and hispanics average at the same score probably isn’t valid, owing to the strong psychometric evidence of durable gaps in general intelligence between these populations. Validity thus incorporates a very large range of issues, some almost purely political, some almost purely technical.
  • Practical means that the testing can actually be accompmlished without harming learning. The test should be easy enough to administer and score that the teacher can actually obtain scores that make sense. Likewise, information from the test should be used as feedback so teaching in that particular class can improve. A once-a-year disruptive test that takes three days and whose scores are not returned that semester would only be ‘practical’ if those three days are academcially worthless.

Most tests now used in education are not RSVP. This is because the six forces involved in the education reform debate either don’t care or are hostile to RSVP testing. Neither Districts nor States have created tests which are RSVP. On a day-to-day level, neither Parents nor Large-Scale Consumers of Educated Workers understand how to read test scores. Neither Teachers nor Publishers regularly create RSVP tests.

But between teachers and publishers, only teachers are opposed to them in principle. RSVP tests would allow districts to fire bad teachers and pay more to good ones, which wouuld be a disaster to the work rules the lobotmized teacher labor force has grown accustomed too. Teacher labor agitators like Diane Ravitch are like the longshoremen on the east coast — opposed to change and so content to watch their industry be destroyed beyond all recognition. Publishers, on the other hand, simply don’t have the skill to give RSVP tests — because they’re influence is contracts and profit, they are indifferent as to whether RSVP test will be given or not.

The federal-academic complex works as a bank for multiple interests. It allows Large-Scale Consumers of Educated Workers to convert their financial resources into power that encourages States, Districts, and Publishers to embrace measurable results, in the form of testing. It allows States to translate their power into money, which can encourage Publishers to make tests. It also creates tests which actually are RSVP, and can be embraced by Publishers as their own products.

Good tests should be RSVP – Reliable, Standard, Valid, and Practical. While we still have a way to go, for the time being teachers are against RSVP tests in principle, while Publishers are simply ignorant as to make to make and sell them. It’s easier to impart knowledge than to change hearts and minds. So until teacher groups wise up, the easier road to education reform is through empowering Publishers and working with them to create RSVP tests.

They Want Money

Different forces in the education reform debate are fighting over different resources. States and school boards are fighting over power. Parents and Large-Scale Consumers of Educated Workers are fighting over childcare. And teachers and publishers are fighting over money.

States and School Boards both focus on power. Every organization wants to exist on its own terms, without having to bow or beg from others. Both States and school boards have the ability to raise taxes, hire and fire workers, and impact the lives of many citizens through decisions related to children. Both are naturally annoyed by the power of the other. From the perspective of states and school boards, education reform is just an opportunity for States to disempower school boards and aggrandize themselves. States have been largely successful in their struggle.

Parents and Large-Scale Consumers focus on childcare. From the perspective of parents, “childcare” means a place you that will take care of children without messing up their features while parents work. What this means depends on social class. For middle and high class parents, schools should not interfere with the natural progression of children to college or other advanced training. For low class parents, schools should not teach children to become socially awkward or talk back. Large-Scale Consumers of Educated Workers, by contrast, want future laborers who are highly productive (that is, can be hired with an expectation of a large return on capital)

While States v. School Boards fight over power is relatively straightforward, the fight over child-care is more complex. First, Parents are highly mobile, and can move out in and out of school boards, while Large-Scale Consumers of Educated Workers are immobile. (While there are often multiple local schools within driving of a job, for political reasons Large-Scale Consumers of Educated Workers prefer to hire in a country proportionally to its revenues from that country.) Further, Parent are risk-adverse, while Large-Scale Consumers are risk-tolerant, when it comes to individual students.

For instance, consider these two possible trade-offs

  • All students in a school become factory drones v. More students talk back to their parents
  • All students in a school go to college v. Some go to college, some start businesses, some fall behind

While the details of these trade-offs are different (low income parents see short-term costs as catastrophic, while high income parents have a future time orientation and so are risk-adverse about future events. Because of the very high rewards for education in the modern economy (as pointed out by the ‘Occupy’ movement), the difference in return-on-investment between a very highly educated worker and a college-educated worker is higher than between a college-educated worker and a high-school-educated worker, but because middle and high class parents fear that it will be their child who does not go to college, they are intolerant of policies that would allow some students to prosper and others to flail.

This fight appears to have been conceded before it began by Large-Scale Consumers of Educated Workers. Instead, Large-Scale Consumers and Parents seem to be working together to create a public education system that creates a floor in terms of proficiency, with Large-Scale Consumers content to allow risky decision to be made after high school graduation.

Teachers and Publishers fight over money. For both Teachers and Parents, education funding is a source of money that can be milked to support lifestyles that could not otherwise be afforded. Teacher and Publishers tend to be active in the political space in order to collect “rents” — to get States and School Boards to provide a greater return-on-investment to their efforts than could be achieved in a free market. Both Teachers and Publishers are rentiers, primarily concerned with improving their own bottom-lines at the expense of children put in their care.

States and School Boards are neutral to the outcomes of education — they simply want to control it. Parents and Large Scale Consumers of Educated Workers both want good education systems, but different in their risk tolerance. Both Teachers and Publishers are essentially parasitical to schools, seeking to divert resources obtained by States and School Boards, at the behest of Parents and Large-Scale Consumers, towards themselves away from children. (Though in the best tradition of marketing, where you take your greatest weakness and claim it is a feature, both Teachers and Publishers identify their own income as being ‘for’ children.)

Education Reform in America is largely a function of the alignment and intelligence of six forces along these three axes. The future of education reform could be predicted if we only knew who would get the power, who would define proper childcare, and who profits.

The Political Economy of Education Reform

To understand the education reform in the United States, keep one thing in mind

We judge ourselves by our intentions, and everyone else by their actions

With this truism, you can see past the anguish and propaganda, and see the education reform debate clearly for the first time.


People see their successes, and they see the good they do. More than this, they see the good they intend to do, if it were not thwarted by others, or by political necessity, or the pressures of the moment. The major forces relevant to education reform, and the good they intend to do, are as follows

  • Teachers forgo higher paying careers to nurture young. It is unfair, of course, to expect teachers to willingly teach at bad schools, so teachers try their best to teach at good schools. Publishers intend to give the young a well-rounded education, to often save them from bad environments, and to teach them how to learn on their own. Their intentions can be thwarted by nonsensical regulations, overbearing administrators, and social factors. It is only fair that teachers have due process, tenure comfortable salaries, substantial time off, respect as professionals, and so on.
  • Publishers are in the business of transmitting knowledge on the written (and now electronic!) page at scale, a spirit-liberating calling that has been celebrated since Gutenberg. It is unfair, of course, to expect publishers to lose money in their calling, so they naturally tend towards profitable sectors. Teachers intend to give the young a well-rounded education, to often save them from bad environments, and to teach them how to learn on their own. Their intentions can be thwarted by nonsensical regulations, overbearing publishers, and social factors. It is only fair that publishers have healthy margins, growth business opportunities, the ability to lock-in long-term contracts, and so on.
  • States are the essence of American democracy. It is unfair, of course, to expect States to surrender the powers they retain to people who have never won elections. States intend to give the young a well-rounded education, to often save them from bad environments, and to teach them how to learn on their own. Their intentions can be thwarted by nonsensical federal regulations, overbearing voters, and social factors. It is only fair that States be immune for their actions, maintain independence from the federal government, and to be able to control the legally-created ‘creatures’ (local elected bodies and incorporated businesses) within their borders.
  • Large-scale Consumers of Educated Workers are the future of the American economy. It is unfair, of course, to expect these Consumers publishers to lose money in their calling, so they naturally tend toward hiering workers educated at public expense. Large-scale Consumers intend to revolutionize business-processes around the world through creating the careers of the future. Their intentions can be thwarted by nonsensical regulations, incompetent suppliers, and social factors. It is only fair that Large-Scale Consumers be able to inexpensively higher workers in order to provide high Return on Capital with regards to labor, however that return is measured.
  • The Federal-Academic Complex provides the largest mass of individuals who are professionally bound to consider systematic reasons for the success and failure of American education in the country. It is unfair, of course, to expect the Complex to operate without the ability to influence the practice of education. The Complex intends to use the latest scientific techniques to understand what a good education is, and how education quality in general can increase. Their intentions can be thwarted by nonsensical regulations, overbearing pre-existing stakeholders, and social factors. It is only fair that those in the Complex have due process, tenure comfortable salaries, substantial time off, respect as professionals, and so on.

Almost everything you read about education reform comes from one of these communities. Therefore, almost everything you read expresses the interests of one of these community. Members of each community judge themselves by their intentions, and each other by their actions.

Note that “Parents” aren’t in this list of stake-holders. The view education as a transient cost and risk center, not an essential part of life. As such, while Publishers, States, Large-Scale Consumers of Educated Workers, and the Federal-Academic Complex care deeply about education, High- and middle-income parents in general are happy with “good school” districts with small numbers of poor people and non-Asian minorities. Low-income parents are politically powerless anyway, and are irrelevant to a discussion of important stakeholders.