Campbell’s Law and the Fallacies of Standardized Testing

Donald Campbell was one of the leading psychologists of the second half of the twentieth century.  His was a time of optimism for planners—there was a belief that the power of technology could be brought to bear on many of the world’s ills.  And indeed they were, often with positive effects.  As a result of central planning, more people receive water, more places are electrified, more children educated, and more diseases eradicated.  All good goals with which Campbell would not quibble.

But Campbell noticed something else to, the emergence of “corruption pressures,” based on the general principle that is now known as “Campbell’s Law.”

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

      In different ways, both Michael Scroggins and Max Holland have pointed at this basic problem in their recent blogs critiquing intelligence tests ranging from the standard tests, to the SAT and college entrance exams.  Such intelligence tests are indeed used to divide up the spoils of placement spots at elite schools, and not surprisingly, ambitious parents seek to corrupt it by means both fair or not. But for use in general analysis like that done by the evolutionary psychologists, the consequences are that their data source over time is corrupted.  The cheating scandals associated the No Child Left Behind Act are a byproduct of Campbell’s Law.  So is the fact that the SAT exam was recently cancelled in South Korea due to widespread cheating.  However most of the corruption does not come from cheating.  It also comes from the fact that such standardized tests are routinely gamed by testing companies which guarantee 100 extra points on the SAT through $1000 prep courses (I used one of these classes for my daughter—it worked!).

For what it is worth, tests like the internationally administered National Association of Educational Progress (NAEP) which do not have consequences attached to them are much less likely to be gamed.  They do not have consequences for funding, admissions, etc., attached to them because they sample across broad areas, and report results on large geographical areas rather than individuals.

Which brings up the BGI Cognitive Genetics Gene Trait Association Study of Intelligence that Michael Scroggins wrote about, and which Dr. Steve Hsu is promoting as a member of the Core Team of BGI.  The Chinese company is seeking people with “high cognitive abilities,” as defined by high scores on the SAT and other standardized tests, or PhDs in a limited number of fields (e.g. physics, computer science, electrical engineering) from “top” US universities.

The implicit assumption is that these people must have DNA which makes them higher functioning than the rest of us.  There are a number of flaws with this approach, starting with those pointed to by Donald Campbell—particularly the fact that the measures they are using long ago lost the validity and reliability due to corruption pressures.  There is also the problem that Michael, Max, and I have been hammering home here at, which is that “intelligence” is always culturally defined, typically by those who have the power to define people like themselves as, well, “intelligent.”  (Perhaps this is why BGI does not want people with PhD degrees in fields they have not studied, or from universities outside the US–this is who they are).

There are of course other reasons why BGI are off on a fool’s errand, some of which is described in Chapter 5 of my recent (2012) book Schooling, Childhood, and Bureaucracy: Bureaucratizing the Child.  Chapter 5 is called “The Sorting Function of Schools: Institutionalized Privilege and Why Harvard is a Social Problem for Both the Middle Class and Public School 65 in the Bronx.”  For that matter Chapter 8 “Seeing Like a State: Efficiency, Calculability, Predictivity, Control Testing Regimes, and School Administration” is also relevant.  (Sorry the book is still only out in hardcover at $90, and Kindle for $72—check your library for a copy, or wait for the paperback version).  To summarize the findings in my book:  Success on tests are inevitably associated with reproducing the status quo, whatever status quo the elites of the day might be promoting.

As for Campbell’s Law, I hope that the people organizing such projects as the Gene Trait Association Study of Intelligence read Donald Campbell’s article carefully, even if he is not an electrical engineer or physicist with a PhD from a top US University, or an 800 on the math portion of the SAT.

A Rumination on Hillary Clinton, DNA, Cognition, and Culture in Just One Blog

Michael in the last pot here, is pointing to a book review “No Big Deal, but This Researchers’ Theory Explains Everything about How Americans Parent” in that describes something that is self-evident to anthropologists, i.e. that whatever is defined as “cognitively advanced” is in fact culturally determined.  This is a point which I tried to make, apparently unsuccessfully, to commenters on this article, which was posted in in March.  What can I say, I just don’t get it how a culturally determined characteristic like “cognitive ability” which is specific to a time and place is determined by DNA.    I also don’t trust the tools of the psychometricians when they are used indiscriminately across national, temporal, and cultural boundaries. (Note the operative word: “indiscriminate.”)


Along these same lines, though on an unrelated subject, there was another pair of articles about DNA I read today, one by the conservative columnist George Will in today’s Washington Post, “Obama is Right on Syria “.  In an outbreak of comity, the conservative Will is complementing the prudence of President Obama for not taking military action in Syria.  In making this argument, Will is in effect pointing out that there is not always an American solution for every foreign policy problem.   He contrasts this with former Secretary of State Hillary Clinton’s claim in a 2010 speech to the Council on Foreign Relations that “Throughout our history, through hot wars and cold, through economic struggles, and the long march to a more perfect union, Americans have always risen to the challenges we have faced. That is who we are. It is in our DNA. We do believe there are no limits on what is possible or what can be achieved.”  So up pops yet another biological metaphor: DNA determines that Americans always rise to challenges, and makes us believe that all can be achieved.


Now for my major stretch in logic.  Science, testing, DNA, and the genetic revolution have become a metaphor which stretches way beyond where, from a scientific perspective, it should be.  Scientists are not at fault, as Rajiv Khan seems to be pointing out to me in his comments.  However, as the book review in Slate points out, cognitive ability is perceived by American culture as being ever manipulable by parents, foreign policy by Secretarys of State, and so forth.  This is why such metaphor is so appealing—there is indeed a culturally grounded belief that “there are no limits on what is possible or what can be achieved” by Americans.  Geneticists and others have of course taken advantage of this cultural bias toward explanation via DNA, and pumped the federal government for ever-larger grants in ways that anthropology can only dream of.

Another Round of Cultural Anthropology and Population Genetics?

Last month, there was a spirited exchange on and Razib Khan’s Gene Expressions blog “Against the Cultural Anthropologists” about the relationship between cultural anthropology and population genetics.  The “conversation” started with the assertion by Razib that basically, the cultural anthropologists are a bunch of post-modern political malcontents who do nothing productive, and are anti-scientific.  A cultural anthropologist at, Michael Scroggins, responded by kindly pointing out that population genetics is a cover for “hippie bashers” who do not really understand what it means to use “the gene” as a basic unit of analysis.  Helpfully, a number of evolutionary psychologists jumped into the fray to point out that since geneticists do math, and cultural anthropologists do not, we in the social sciences have lower IQs and therefore our poor little genes are doomed to Darwinian extinction.


Thankfully, after the adrenaline charges of such ripostes, we (or I) found that the individuals involved had similar habits of arguing with evidence, theory, and references to founding principles.  It just so happens that all these of these (evidence, theory, and founding principles) are indeed different in the two fields. These are some of the differences I took away from the 100+ comments that poured into the comments sections at both the and Gene Expressions blogs:


Cultural Anthropology Population Genetics
1) Likes Qualitative Description Likes Statistics and quantification
2) Likes Steven Jay Gould Not sure that Steven Jay Gould is relevant to this discussion—and he made a couple of mistakes anyway
3) Believes that cognitive abilities are a social construction Believes that cognitive abilities can in part be explained by genetic inheritance
4) Believes that statistics are often misused in a way that reifies ethnic distinctions Basic principles involve references to Population Genetics texts which describe how to create more precise models based on latest gene sequencing techniques
5) Basic principles involve a reference to Boas, Weber, Durkheim, and Malinowski Really like the work of Luigi Cavalli-Sforza
6) Who is Cavalli-Sforza? Who cares about Boas, Weber, or Malinowski—they’re long dead and out of date—we are about the future.  Durkheim is about religion, and that’s not scientific.  He’s also dead.
7) Statistics don’t work because there is no such thing as a single fixed “ethnic” or “racial” characteristic.  Ever more precision in such circumstances is something of a fool’s errand. Statistics are a fantastic tool, particularly as the number of genes (and there variants) are identified, and the collection of data improves.  This will inevitably lead to greater validity and reliability in the models.
8) Gene flow is a product of culture—you do not generally make a baby with someone who is not from your own culture/class/status group.  So models without that assume random mating aren’t very useful. Gene flow can be evaluated independently from culture/class/status group.  So who cares about cultural anthropological variables?
9) No such simple category as a Nacirema which can be represented in a blood sample A Nacirema is a Nacirema is a Nacirema


Then there were the posters who wrote in the traditions of Evolutionary Psychology, and pointed to studies correlating economic success, genetic fitness, and the heritability of cognitive abilities as measured by various intelligence tests.  These people also didn’t like Steven Jay Gould and tended to correlate the wealth and longevity of societies/individuals wit good genes, and economic success.  These posters tended to have more in common with the Population Genetics folk, but I sense that they were in fact a third view.


Anyway, this set me to wondering what it would take to convince someone like the posters Razib, dad, or the others that Population Genetics should pay more attention to cultural anthropology? I am still convinced that their population models would be strengthened if they were to take into account the nature of inequality (ethnic, racial, class, gender) which does indeed structure who mates with who (a fact self-evident to any junior high school boy or girl).   It could also perhaps be quantified in a positivistic fashion, but then I am concerned that some precision begins to be lost, since by categorizing anything, you tend to lose some proportion of the nuance that in fact, cultural anthropology is quite good at.


As for the posters coming from Evolutionary Psychology, I do still have a really tough time understanding how “intelligence” is not inherently a cultural product. Sure some individuals have better “cognitive abilities” than others, but cognitive abilities are always created relative to pre-existing cultural values.  One cultures “success strategy” is another culture’s irrelevancy.  This applied even to math problems used trans-nationally.  My attitude comes from my experiences with African villagers who were much more capable at navigating their environment than I was.  This applied to cognitive skills like judging the weather, crops, wildfires, wildlife, and navigation by hillshapes—all complex activities which I’m not good at, and are not measured by a pencil and paper i.q. test about math or language. I just can’t imagine how a Harvard student who is a MENSA member and has 1600 SAT scores and a 150 i.q could do those things—though perhaps they maybe some of these really smart people might convince me otherwise.


But then what next?

As an academic, I have started to think about what it would take to write an academic article (not a blog) which would be acceptable to peer reviewers in all three fields.  Frankly, the thought of this exhausts me because it implies a lit review of cultural anthropology, population genetics, and evolutionary psychology.  This implies lots and lots of reading which I have not done yet.


I actually wrote such an “inter-disciplinary” article between 2007 and 2013 about mirror neurons which made reference to the fields of social psychology, neural sciences, and philosophy of mind.  My article tried to explain to neural scientists studying mirror neurons why their articles would be enriched by an understanding of one hundred years of study by sociologists of the “Looking Glass Self.”  Believe, me the scientists didn’t want to hear what I had to say, and the rejections from anonymous reviewers of the article were nasty, demeaning, and contemptuous in ways much worse than anything Razib said about cultural anthropology.  After all Razib only said cultural anthropologists should be kicked out of the academy and go work for Cultural Survival.  Big deal.


Having said that, I still am tempted to begin such an article, even though it sounds exhausting at this point.  I would need to read much more about Population Genetics, including some of the references Razib posted.  Also some of the critiques of Gould.  I hate to say it, but I also need to get a firmer grip on what the Evolutionary Psychologists have to say about the nature of intelligence.


Anyway, I guess in a backhand way, this blog is a thank you to Michael, Razib, dad, justaguy, Chuck, German, Sam, KbH and the others who posted in February and March on half a dozen different posts at both Gene Expressions, and  You have given me much to think about in the future.  Blogging fast and furious as we did has an important place to play in academia, as does the slow nasty conservative business of peer-reviewed research.  Indeed, I believe that such blogging will in the long run make the more careful and slow business of peer review stronger, particularly when it facilitates such truly-interdisciplinary efforts.

Mla Bri Genetics and Anthropology in Northern Thailand

Many anthropologists are concerned with the tendency of biologists to reduce social life in general, and culture in particular to the genes people carry.  As a sociologist, I share that concern.  I think that such reductionist approaches give a false sense of precision to the concept of culture which while very real, is often messy at the edges in very human ways.


I came up against this tendency to “reduce” everything to genetics recently in an article about the Mla Bri of Thailand, a small group (200-300 people) speaking a Khmuic language in northern Thailand.  As a group, they attract the attention of anthropologists because, until recently, they did a lot of hunting and gathering for subsistence, while also being engaged in exploitative labor practices with neighboring groups speaking a range of languages, including northern Thai, Hmong, Mien, and probably Khmu.


Anyway, in 2010, a second article about Mla Bri genetics was published in BMC Genetics using blood samples collected from Mla Bri in 1999 by visiting geneticists.  These samples were then compared to blood samples in a bank from the other ethnic groups found in northern Thailand, including northern Thai, Hmong, Mien, Khmu and so forth.  But the geneticists doing the work did all their work at the laboratory bench, and did not familiarize themselves with the geography of the Thai highlands, or the unequal relationships between the Mla Bri and the neighboring groups.  Their conclusion was that the Mla Bri have been isolated culturally and genetically from the Hmong and other groups for a long long time.  I am confident that their work at the laboratory bench was sound.


But, I did have a basis to question their lack of ethnographic context—you see, I have good friends who have lived with the Mla Bri as missionaries for the last 30 or so years.  Gene and Mary Long speak Mla Bri, and were even present with the blood samples were drawn in 1999.  They are also among the best “gut level” anthropologists I’ve ever met.  Anywya, On the basis of what they knew about the Mla Bri, we evaluated the article in BMC Genetics, and wrote a comment which was posted this week.  This posting makes the point that the genetics work published in 2010 would have been greatly enriched if the authors had talked to the people living in the village, whether it be the Mla Bri themselves, or the Longs.  Indeed, it is well-known among the Mla Bri that despite strong norms for endogamy, extra-marital relationships do exist, and that exogamy does occur.


Our Comment in BMC Genetics is available here:


The article we are commentin on is here.


A similar exchange that I had about the Mla Bri in 2005 in the pages of PLoS Biology is here.


The original 2005 article about the Mla Bri genetics is here.


If you want to know more about the Longs and their work with the Mla Bri, there will be an article about suicide among the Mla Bri in the 2013 issue of the Journal of the Siam Society which should be on-line soon!


Bottom line:  Be wary of the cultural reductionists, be they the selfish-gene type, the lab bench type, check the box type, or any other such type.  For a comment on this, see here!

“Building Bildung,” and Other Improbabilities among German University Undergrads


German has two words for the English word “education.”  Erziehung describes the school system, and the mechanics of what is taught and conveyed from the world of adults to that of children in order to “bring them up.”  Focus is on skills adults need like literacy, numeracy, history, and the factual basis citizens need to understand to participate socially, culturally, and economically in society.  The German education system is designed to educate all children in such basic skills. It is something that is done for children, and leads to practical apprenticeships/schooling which many German youth begin at ages 15 or 16, i.e. after completion of  9-10 years of schooling and in turn lead into the workforce.  This type of schooling makes for a very disciplined and skilled workforce, which is able to produce engineering wonders like Audi, BMW,  Mercedes and Siemens that power Germany’s modern export-led economy.


But there is another word in German for education, which is “Bildung,” which is a more important quality, and the one which is more highly valued even though it is not aimed directly at workforce preparation.  Bildung it is tucked into the programs of the primary, secondary, and really kicks in at the university level, including at Leuphana University where I am currently a guest professor.  Bildung roughly means “cultivation of the intellect.”  Unlike Erziehung, such cultivation is not something that is done for you, rather it is a quality that you as an individual cultivate as a matter of intellectual habit.  This is why primary and secondary schools in Germany have a curriculum in music, art, history, religion, the social sciences, philosophy, and so forth.  Famous German philosopher-types have written about this word and emphasize the quality of “cultivation of the intellect.”  Related to this, Germany prides itself on being the land of “poets and thinkers.”  Besides BMW, Mercedes, Audi, and Siemens, think also Luther, Goethe, Hegel, Schiller, Brothers Grimm, and Max Weber.  And in music, it’s Beethoven and Bach just for starters.  Indeed, in a country where political and military achievements are looked at with skepticism, bemusement, and sometimes disgust, cultivation of the Bildung is critical to a sense of adult identity.  The result for the education system that children are expected to develop habits of intellectual cultivation for their own sake–and to appreciate the cultural resources which are the product of such habits.  By the time they reach university, students are expected to do this on their own, without a lot of prompting from university professors like me.


To illustrate, here is a quick email I received from a student in my Post Colonial Theory class at Leuphana University last week, with a question about her term paper.  I know her well because she came to my classes for the last 14 weeks, even though there is no grade or “incentive” given for attendance; indeed the paper is the only graded work she will do for me this semester, and it is due after a six week writing period in which there are no classes.  As a full-time student, she will be completing 5-6 such papers during this period. She wrote:

Dear Mr. Waters,

you suggested to me to use Fanon and Wallerstein for my term paper about (Country X), but I would rather like to use Said and Spivak if possible. It seems pretty difficult to apply Fanon’s and Wallerstein’s theories on (Country X), since they are more focused on race, ethnicity and the relations between 1st and 3rd world or rather colonies which were overseas. It seems like Said and Spivak might fit better, also since (Country X) literature and articles about a postcolonial (Country X) refer to their theories. I still would like to refer to Fanon a little bit too. But of course I would like to know what you think about this?

Best regards,

Leuphana Undergrad in Culture Studies

What do I think?  After 15 years working at a mid-level American university, I think cool—go for it.  I mean an email like this from an undergrad is really cool!  I wonder how you could work in three languages (English, German and Language X), and where you came to have such a wide interest in social theory.  My students in America would have taken my cookie-cutter advice about using Wallerstein and Fanon, and left it at that.  But if the Language X literature leads to Said and Spivak, and you’re game to read that—go for it! And I am really looking forward to learning something from your paper about both Country X, and new ways of applying post-colonial theories.


The student’s years of cultivating the habits of the mind that are German Bildung in primary and secondary school are what prepared her to write such an email as a second year university student.  She has habits of reading widely, questioning sources, and engaging creatively.  What is more, she assumes that such things are normal educated humans do, as indeed it is  among my students here.  The funny thing for this US American professor is that any number of my students here in Germany could have written such an email—this is just the one that was handy.


But, I have rarely in my years of teaching in the US received an email like this.  Rather the focus of the US American student is on the formula that will get a good grade on my paper—And, truth be told, I give it to them: Have a clear introduction with a thesis, illustrative examples in the body, and a good conclusion to tie things off.  In other words mechanics and process are the issue I my conversation with US students.


Which brings me back to the subjects of Erziehung, Bildung, and what is now my new pet peeve, Chickering and Gamson’s (1987) Seven Practices of Good Undergraduate Practice.  In my personnel file at Chico State, I repeatedly reflected on these seven principles at the behest of administrators concerned that I earn my keep, i.e. be “accountable.”  As I wrote previously, I think that the insistence of US American administrators on using these “best practices” have over-emphasized process at the expense of what is in essence, Bildung.


Before I finish this blog, here are the Chickering and Gamson’s (1987) seven processes which administrators claim underpin high quality undergraduate teaching.  Note that little of this refers to student learning or acquisition of Bildung.  Rather it is about the faculty can be supervised on, and therefore held accountable for by administrators.  Good Teaching Practices include:


1. encourages contact between students and faculty,

2. develops reciprocity and cooperation among students,

3. encourages active learning,

4. gives prompt feedback,

5. emphasizes time on task,

6. communicates high expectations, and

7.respects diverse talents and ways of learning.

By and large, these are great principles of educational process.  But indeed they leave out Bildung.  Nowhere in these seven principles is there any mention of thinking, reading, capacity to think abstractly which is what my student’s email demonstrated.  Nothing in Chickering and Gamson (1987) for the land of poets and thinkers!  Rather it is designed for the land of process engineering—the world Frederick Taylor imagined when in 1911 he published The Principles of Scientific Management.  (Note to American students reading this: If you don’t know who Frederick Taylor was, check out his Wikipedia entry).

One consequence are email from German students like the one above who, by the way, has had little personal feedback from me during the semester, nor do I have any clue about how much “time on task” she has spent on any assignment.  But she does have high expectations for herself, and she learns and writes in German, English, and Language X which is certainly diverse, though perhaps not in the way Chickering and Gamson were thinking of diverse learning styles.  What she has are habits of the mind that are what the German system somehow cultivates. I guess this could be called active learning, though I have no idea what I did as a Professor to encourage it.  Which of course brings me back to Professors Chickering and Gamson (1987), and their Seven Principles.

What would happen if instead of addressing their seven principles on my next course syllabus, I was to melodramatically tell my US American students: Give me Bildung (Habits of Mind), or Give me Death!

I write about Bildung and other such improbabilities in the US American K-12 education system in my new book Schooling, Bureaucracy and Childhood: Bureaucratizing the Child.  Request it from your library, or if you have enough money, order a hardcover (or Kindle) copy from your favorite bookseller!