Campbell’s Law and the Fallacies of Standardized Testing

Donald Campbell was one of the leading psychologists of the second half of the twentieth century.  His was a time of optimism for planners—there was a belief that the power of technology could be brought to bear on many of the world’s ills.  And indeed they were, often with positive effects.  As a result of central planning, more people receive water, more places are electrified, more children educated, and more diseases eradicated.  All good goals with which Campbell would not quibble.

But Campbell noticed something else to, the emergence of “corruption pressures,” based on the general principle that is now known as “Campbell’s Law.”

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

      In different ways, both Michael Scroggins and Max Holland have pointed at this basic problem in their recent blogs critiquing intelligence tests ranging from the standard tests, to the SAT and college entrance exams.  Such intelligence tests are indeed used to divide up the spoils of placement spots at elite schools, and not surprisingly, ambitious parents seek to corrupt it by means both fair or not. But for use in general analysis like that done by the evolutionary psychologists, the consequences are that their data source over time is corrupted.  The cheating scandals associated the No Child Left Behind Act are a byproduct of Campbell’s Law.  So is the fact that the SAT exam was recently cancelled in South Korea due to widespread cheating.  However most of the corruption does not come from cheating.  It also comes from the fact that such standardized tests are routinely gamed by testing companies which guarantee 100 extra points on the SAT through $1000 prep courses (I used one of these classes for my daughter—it worked!).

For what it is worth, tests like the internationally administered National Association of Educational Progress (NAEP) which do not have consequences attached to them are much less likely to be gamed.  They do not have consequences for funding, admissions, etc., attached to them because they sample across broad areas, and report results on large geographical areas rather than individuals.

Which brings up the BGI Cognitive Genetics Gene Trait Association Study of Intelligence that Michael Scroggins wrote about, and which Dr. Steve Hsu is promoting as a member of the Core Team of BGI.  The Chinese company is seeking people with “high cognitive abilities,” as defined by high scores on the SAT and other standardized tests, or PhDs in a limited number of fields (e.g. physics, computer science, electrical engineering) from “top” US universities.

The implicit assumption is that these people must have DNA which makes them higher functioning than the rest of us.  There are a number of flaws with this approach, starting with those pointed to by Donald Campbell—particularly the fact that the measures they are using long ago lost the validity and reliability due to corruption pressures.  There is also the problem that Michael, Max, and I have been hammering home here at Ethnography.com, which is that “intelligence” is always culturally defined, typically by those who have the power to define people like themselves as, well, “intelligent.”  (Perhaps this is why BGI does not want people with PhD degrees in fields they have not studied, or from universities outside the US–this is who they are).

There are of course other reasons why BGI are off on a fool’s errand, some of which is described in Chapter 5 of my recent (2012) book Schooling, Childhood, and Bureaucracy: Bureaucratizing the Child.  Chapter 5 is called “The Sorting Function of Schools: Institutionalized Privilege and Why Harvard is a Social Problem for Both the Middle Class and Public School 65 in the Bronx.”  For that matter Chapter 8 “Seeing Like a State: Efficiency, Calculability, Predictivity, Control Testing Regimes, and School Administration” is also relevant.  (Sorry the book is still only out in hardcover at $90, and Kindle for $72—check your library for a copy, or wait for the paperback version).  To summarize the findings in my book:  Success on tests are inevitably associated with reproducing the status quo, whatever status quo the elites of the day might be promoting.

As for Campbell’s Law, I hope that the people organizing such projects as the Gene Trait Association Study of Intelligence read Donald Campbell’s article carefully, even if he is not an electrical engineer or physicist with a PhD from a top US University, or an 800 on the math portion of the SAT.

9 thoughts on “Campbell’s Law and the Fallacies of Standardized Testing

  1. dad

    So Tony, if you had to choose a sperm donor you would literally choose any random person in the world? Be honest. they all have the same potential intelligence, right? that means you have to live by your word and pick a random donor. I will even narrow it down for you: you have to choose between a doctor and a person who has been working at a movie theater for 30 years.

  2. Tony

    @dad. Basically, yes. I can’t imagine that the “quality” of the sperm from the doctor is that much different from that of the person working at the movie theater. Indeed, lots of doctors have fathers who worked at the equivalent of movie theaters. And virtually all doctors (and movie theater workers) are the great-grandsons of farmers. WHich is a way of saying that most doctors have cousins who are movie theater ticket takers, and vice versa. So why would their sperm be all that different with respect the genes for intelligence?

    Basically, I think occupation is a poor proxy for genetic fitness.

  3. dad

    really? r u sure? you’d take a donation from GW Bush the same as you would Richard Fenyman? What about from a donation from a psychopathic killer? you and your partner would be ok with a sperm donation from Jeffrey Dahmer just the same as Ghandi?

  4. Tony

    Yep–except for maybe Dahmer, but that is for emotional reasons, not intellectual ones. But then Dahmer’s dad was an Analytical Chemist according to Wikipedia.

    As a mind game, I thought back to all the first and second cousins, i.e. the descendants of my great-grandparents who were mostly farm people. (This is a heritage I share with most modern people). Among the cousins I know about there is a nice range of doctors, professors, holy rollers, salesmen, and knuckleheads. Yet we all share the same genes and legacy on the farm.

    Question: Why does Feynman keep coming up as being the example of being particularly smart? On Wikipedia it sounds like he was a great physicist, but is that really the only measure of human excellence? Why not Albert Schweitzer or Gandhi?

  5. Michael Scroggins

    @ Tony

    Why Feynman? Because there is a natural order of cognitive ability and it runs from physics down towards farmers.

    Call it the trickle down theory of intelligence or supply side genetics.

  6. dad

    every fetus is a potential ethnography.com writer (as long as they’re raised “correctly.”) i guess we could take it up a notch and shoot for Stephen Jay Gould level for every fetus.
    http://www.sciencedaily.com/releases/2013/05/130509142050.htm
    but then there’s stuff like this tho. raised in identical environments yet they come out differently.

  7. Tony

    @dad: The Science Daily helps make my point–thanks for the reference.

    @Michael: If farmers are below Feynman, where does this put Stephen Jay Gould relative to farmers? And to think I once liked Gould’s writing about statistical analysis!

  8. Dalliard

    “particularly the fact that the measures they are using long ago lost the validity and reliability due to corruption pressures”

    The reliabilities of SAT tests are high, generally >0.90. Paul Sackett and colleagues looked at the correlation between SAT scores and college grades in a sample of 165,000 students from 41 colleges. The correlation, corrected for range restriction, was 0.47, or, if you control for course difficulty, 0.55. The validity was very similar across social classes. SAT scores also predict equally well within each race and ethnic group, but they are somewhat biased in favor of groups with low average scores like blacks and Hispanics (paradoxically, this external bias appears to result from the fact that the tests are internally unbiased; see Roger Millsap’s work on this). The average effect size for SAT test prep is about 20 points (see Derek Brigg’s analysis of NELS data, for example).

    So your argument is simply false. It’s certainly possible that the validity of the SAT would be even higher if there was no test prep, but that’s neither here nor there. It’s valid enough as it is.

    As to the BGI project, GCTA studies by Peter Visscher and others show incontrovertibly that intelligence differences, as measured by psychometric tests, are strongly influenced by a large number of genes working in an additive fashion. Whether the BGI study has enough statistical power to detect many variants remains to be seen, but eventually they will be detected. The environmental determinist case for IQ differences is dead and buried.

  9. There is a correlation with the SAT and college grades because indeed, SAT tests measure what colleges require. The SAT is a good administrative tool for managing university enrollments, and norming the culture of the classroom.

    But for measuring “intelligence” (were there such a thing) IQ tests are tautological. I would highly recommend Gould’s book “The Mismeasure of Man.” You can take or leave the parts about cranial size which have been effectively critiqued. There is much more to his argument about the nature of intelligence testing as self-fulfilling prophecy.

    I would also recommend my own book (actually I did this in my blog) about the nature of bureaucracy and education. Tests are a great way for a culture to organize schools which will recreate a pre-existing culture. Nothing wrong with that–it is done everywhere in the modern world. But it also has little to do with individual intelligence.

    I still can’t figure out how natural selection selected for genes to do quantum physics when the science itself is 70 or 80 years old, and humans had the “i.q. test” of farming for 10,000 years before that (and hunting and gathering even before. Quantum physics is a great thing, but it emerges out of culture, not genes.

Comments are closed.