Campbell’s Law and the Fallacies of Standardized Testing

Donald Campbell was one of the leading psychologists of the second half of the twentieth century.  His was a time of optimism for planners—there was a belief that the power of technology could be brought to bear on many of the world’s ills.  And indeed they were, often with positive effects.  As a result of central planning, more people receive water, more places are electrified, more children educated, and more diseases eradicated.  All good goals with which Campbell would not quibble.

But Campbell noticed something else to, the emergence of “corruption pressures,” based on the general principle that is now known as “Campbell’s Law.”

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

      In different ways, both Michael Scroggins and Max Holland have pointed at this basic problem in their recent blogs critiquing intelligence tests ranging from the standard tests, to the SAT and college entrance exams.  Such intelligence tests are indeed used to divide up the spoils of placement spots at elite schools, and not surprisingly, ambitious parents seek to corrupt it by means both fair or not. But for use in general analysis like that done by the evolutionary psychologists, the consequences are that their data source over time is corrupted.  The cheating scandals associated the No Child Left Behind Act are a byproduct of Campbell’s Law.  So is the fact that the SAT exam was recently cancelled in South Korea due to widespread cheating.  However most of the corruption does not come from cheating.  It also comes from the fact that such standardized tests are routinely gamed by testing companies which guarantee 100 extra points on the SAT through $1000 prep courses (I used one of these classes for my daughter—it worked!).

For what it is worth, tests like the internationally administered National Association of Educational Progress (NAEP) which do not have consequences attached to them are much less likely to be gamed.  They do not have consequences for funding, admissions, etc., attached to them because they sample across broad areas, and report results on large geographical areas rather than individuals.

Which brings up the BGI Cognitive Genetics Gene Trait Association Study of Intelligence that Michael Scroggins wrote about, and which Dr. Steve Hsu is promoting as a member of the Core Team of BGI.  The Chinese company is seeking people with “high cognitive abilities,” as defined by high scores on the SAT and other standardized tests, or PhDs in a limited number of fields (e.g. physics, computer science, electrical engineering) from “top” US universities.

The implicit assumption is that these people must have DNA which makes them higher functioning than the rest of us.  There are a number of flaws with this approach, starting with those pointed to by Donald Campbell—particularly the fact that the measures they are using long ago lost the validity and reliability due to corruption pressures.  There is also the problem that Michael, Max, and I have been hammering home here at, which is that “intelligence” is always culturally defined, typically by those who have the power to define people like themselves as, well, “intelligent.”  (Perhaps this is why BGI does not want people with PhD degrees in fields they have not studied, or from universities outside the US–this is who they are).

There are of course other reasons why BGI are off on a fool’s errand, some of which is described in Chapter 5 of my recent (2012) book Schooling, Childhood, and Bureaucracy: Bureaucratizing the Child.  Chapter 5 is called “The Sorting Function of Schools: Institutionalized Privilege and Why Harvard is a Social Problem for Both the Middle Class and Public School 65 in the Bronx.”  For that matter Chapter 8 “Seeing Like a State: Efficiency, Calculability, Predictivity, Control Testing Regimes, and School Administration” is also relevant.  (Sorry the book is still only out in hardcover at $90, and Kindle for $72—check your library for a copy, or wait for the paperback version).  To summarize the findings in my book:  Success on tests are inevitably associated with reproducing the status quo, whatever status quo the elites of the day might be promoting.

As for Campbell’s Law, I hope that the people organizing such projects as the Gene Trait Association Study of Intelligence read Donald Campbell’s article carefully, even if he is not an electrical engineer or physicist with a PhD from a top US University, or an 800 on the math portion of the SAT.

Did Chickering and Gamson’s (1987) Seven Practices of Good Undergraduate Education Dumb Down American Education? A View from Germany

I just finished my semester teaching as a Guest Professor at Leuphana University in Germany, and am beginning to figure out an answer to a question I get asked frequently: “What is the difference between American and German universities?”  Actually, the German students who have been exchange students in the United States have helped me along.  They say:


In American universities, there’s lots of assignments, busy work really, and the students are not expected to take responsibility for their own learning.  Really it is like [German] high school, not the university.


German university classes in contrast to American universities do not involve a lot of graded work during the semester.  The credit for the entire course comes down to a few ungraded assignments during the semester (typically including a 20 minute oral presentation), and a final test or paper which is the only thing actually evaluated for a grade. Class itself is a combination of student presentation, faculty lecture, and if it is small enough, seminar.  In the case of the paper, it is usually 10-12 pages long—one for each class.  There are no incentives for attendance.  There are also is no preparation for midterms or quizzes–because there are few such tests in the American sense.  The term paper for the class is handed in is due about six weeks after the semester ends. The professor then hands in a grade sometime in the next few months, which is then reported to the student, without elaboration or feedback.  And that’s it—if you pass, you go on to the next semester.


In other words, at this German university, there are classes in which there is little encouragement for contact between students and faculty, professors do not initiate cooperation among students (that is up to the students), there is little feedback from the faculty regarding student work (and feedback need not be prompt), time on task is not monitored, and no one really cares if a student has another way of learning that does not show up on assessment.


The good news, though is that there are high expectations for German students, and if they meet the high expectations, they get a degree at the end of three years or more.  Oh, and before they can also organize complex ideas in their 10-12 pages, read voraciously, and are up to date on current events both within and outside their fields.  But they are this way not because of class content and monitoring by university faculty like me.  Rather they are like that because that is what a good student is.  The 20-30 minute class presentations are generally include high quality analysis, too.  Also, with most German undergraduates you can have an well-informed conversation about their lives or the events of the day in English, German, and perhaps another language.


But is this really good practice in undergraduate education?  According to the gurus of American undergraduate education, Professors Chickering and Gamson (1987)  the German system is a lousy system which should produce


Apathetic students, illiterate graduates, incompetent teaching, impersonal campuses — so rolls the drumfire of criticism of higher education.


The problem of course is that my German students are far from apathetic, certainly not illiterate, and judging from the fact that major political parties show up on campus to court the student vote they are far from apathetic.  As for the incompetence of teaching, I cannot judge that from my perspective, except to note that my German language teacher was pretty good!


Nevertheless in response to the problem that Professors Chickering and Gamson (1987) diagnosed regarding apathy, illiteracy, and incompetence, they prescribed seven best practices for high quality undergraduate education which American faculty are routinely held to in faculty reviews:


1. encourages contact between students and faculty,

2. develops reciprocity and cooperation among students,

3. encourages active learning,

4. gives prompt feedback,

5. emphasizes time on task,

6. communicates high expectations, and

7.respects diverse talents and ways of learning.


A German academic high schools (Gymnasium) teacher might recognize Chickering and Gamsons (1987) criteria, but the German university system I described above only meets 1/6 of Chickering and Gamson’s (1987) goals, i.e. the one about high expectations.   The other six, are considered to be the responsibility of the student, not the faculty, which is why of course my German students hold American undergraduate education in such low esteem, and typically consider their exchange experience to be “high schoolish.”


The German view is that university faculty is in charge of organizing interesting class to which students will come if they are inclined, and participate when they are prepared.  If they show up sort of regularly, and do the major assignment/test at the end of the semester, they will get credit for the course.  University faculty are not in charge of checking endless assignments, homeworks, administering quizzes and the many other tasks which are really about checking for “time on task,” the metric that is most valued by administrators and faculty?


Could it be too that the reason that the German students believe that American undergraduate education is “like high school” is the fault of Chickering and Gamson (1987)?  I bring this up because since I started teaching American undergraduates in 1996, the administrators who hired me have asked me to include Chickering and Gamson’s (1987) seven goals of good undergraduate practice in my employment dossier.  I have dutifully done this, and in a show of faith upped the number of assignments in my classes, made a point of returning assignments quickly, encouraged active learning in the assignments I make, tried to make sure that I assigned 2 hours of reading/out of class work for each Carnegie unit the students receive, and checked to see if they did it by using quizzes.  I am an accountable professor!  But does this really make me a good professor who creates students who think creatively and deeply about sociology as I see my German students doing?


And anyway, what do we in the American system get in response?  Undergraduates who are like, well, high school students in their capacity to work independently (meaning they work like well-supervised high school students).  The expectations authored by Chickering and Gamson (1987) have become a self-fulfilling prophecy.  We defined high quality as demanding a lot of busywork (not critical thinking, routine reading, or the capacity to work independently, and surprise, we get undergraduates who


–contact faculty frequently

–do group work

–equate active learning with recreational activities

–expect prompt and voluminous feedback rather than editing themselves

–spend a lot of time doing homeworks

–read for the detail on quizzes rather than for “fun”

–have high expectations of their own abilities, and

–expect learning accommodations to be made for different learning styles


This is not to say that German students are faultless.  Indeed, the German system has recently undergone major reform because students frequently floated away from a system that by American standards can be pretty brutal (it is easier to flunk a student you do not know, than one who has frequent contact with faculty).  German students also take too long to graduate, and so forth.  I should add, that my experience last semester is that American attendance patterns are a bit better than German, and punctuality is also a lot better at Chico State, where I routinely start class within 15 seconds of the class start time, a process which seems to take 5-10 minutes here…).


Still, when the German students do finally get to class, they bring intellectual preparation that my American students do not.  But then again, habits of intellectual engagement, like voluminous reading, independent thought, oral presentations, and routinely crafting 10-12 page essays are really not part of what Chickering and Gamson (1987) have defined as high quality undergraduate education, are they?



Chickering and Gamson (1987).  There are hundreds if not thousands of copies of Chickering and Gamson’s original article floating around on the internet.  If you really want to read it (I encourage you to do so), either click the link embedded in the article, or simply google it up!

Chico Rocks, and Berkeley…

I was back on the Chico State campus last week, and the new first year students are here, parents hovering nearby as they prepare to cast them out to wilds of Chico State.  The newly minted frosh are of course relishing this—they realize that Chico Rocks, and that they have finally managed to land where they are meant to be, if only they can finally ditch their parents, and seek out what the college president insists are that elusive “Chico Experience.”


So in this essay I will confirm:  junior you’ve made it.  I know that you could have gone to UC Berkeley, but somehow you managed to elude that destiny.  Some of you flat out turned down the offer of a free ride to Berkeley—and we thank you for having the gumption to do so.  Others of you were trickier and more devious in avoiding being cast into the huge classes of Berkeley, even though that was your parents’ greatest desire.  Perhaps you cut class during that 11th grade history exam in order to ruin your 4.0 gpa.  Or you got placed in detention during that stupid health exam for talking in class.  Maybe you hung out behind the gym doing who knows what.  Or perhaps you (like me) you flunked p.e.  Whatever it was, congratulations and well-done!  Now you’ve arrived at Chico State, and you can finally announce to all Chico Rocks, and Berkeley Sucks!


Now, go read this essay which will put into big words what you have known all along about the relationship between Chico and Berkeley.  Your parents are bound to be amazed at your learned erudition when you roll in next Thanksgiving.  And to top it off, the geek from you high school who made it into Berkeley (and refused to cut  the history test or hang out behind the gym) won’t really understand what it is about, and will stomp off huffing and puffing and announce “Sour Grapes!”


Back form reading the essay?  Good.  Now go out and drop that Accounting Class, and sign up for a Sociology Class.  Intro to Sociology (Sociology 100) will do—your goal should be to work yourself up to Classical Social Theory where you can learn more about the astonishing thoughts of not only Max Weber, but also Karl Marx, Emile Durkheim, W. E. B. DuBois and Mary Wollstonecraft.  (Yes, that’s the same Karl Marx you might learn about in Berkeley, but we put a slightly different spin on it at Chico!).  I guarantee you that you will really like Mary Wollstonecraft in particular—all students do.  Mary Wollstonecraft was one of the first writers who described the weird dance between men and woman—and offered a virtuous formula for getting past the nonsense you, as new students recently freed of your parents, will likely be engaging in as you prance about the dining commons seeking “true love.”


So you’ve checked Chico State’s Fall 2012 schedule and found out that there are no sections of Sociology 100 available?  Yep, that’s right, the man (that’s Governor Jerry Brown and the State Legislature) cut a bunch of Sociology 100 sections this year so your parents and their friends do not have to pay as much in taxes as your grandparents did to the same Governor Brown (he never goes away).  But don’t worry, generational inequality is a problem which you will also learn about in Sociology.  The end result is that you can’t get your sociology class, even though your parents did.  So you better stick out Accounting, but be sure to be aggressive about getting it next semester.  That way I will hopefully see you in Classical Social Theory in your sophomore year!

This Week in Ethnography: Second Digital Ethnography Week _ Trento 17-21 sept. 2012

There is not much to report in “This Week in Ethnography”, a segment I am inventing as a means of reporting on the global pulse of this most important subject.  The one item that jumped out of my feeds at me was that I missed the application deadline (of July 22, 2012) for the:

Second Digital Ethnography Week _ Trento 17-21 sept. 2012

The second “Digital Ethnography Week” (DEW), an intensive week focused on the study of digital methods and digital ethnographic approaches. The DEW is intended for Ph.D. students and researchers interested in developing advanced methodological skills to account for the digital in contemporary social life.
As their website reports, this looks like a great opportunity for aspiring digital ethnographers.
Ethnography and Journalism
Be warned: The Data Journalism School in Rome is involved in this effort.  I know the conflation of ethnography and journalism is shocking to some. During my graduate training, I recall one of the senior faculty members of my anthropology program criticizing a students’ work by referring to it as “journalism”.  The context for this event was a thesis draft presentation based on ethnographic fieldwork in a doctoral colloquium.  I believe the Professor’s intention was to imply that the student was “only out for a story” and had “little theoretical or methodological reasoning” for how they had generated the data they were reporting on.
The irony of this situation was that this was a program in applied anthropology.  In any event, let us not “throw the baby out with the bath water” or in this case, throw good data or solid technique out with the researcher using it.  Data Journalism is a fantastic means of getting at reality.  For example, check out the following TED talk by David McCandless: The beauty of data visualization and try to tell me that this “journalist” is “only out for the story”.

UCLA loses to USC, but is Still Afraid to Challenge Chico State in College Rankings!

     It is college ranking season again, sponsored by US News and World Report.  Once again, US News left Chico State out of their ranking system, I think because the big kids thought that they would lose if it came to any measure of undergraduate education.  After all as I have long asserted, Chico State beats UC Berkeley when it comes to quality of undergraduate teaching; what possible advantage could some university in southern California hope to have over any of us in northern California?! 

    Nevertheless, during this season professors steeped in the scientific method and research courses throw caution aside, and loudly brag about whatever significance such rankings may or may not have.  For whatever it is worth the University of Southern California (USC) now out ranks the University of California at Los Angeles (UCLA), a story that made the front page of the Los Angeles Times.

    Notably, even the staid Atlantic magazine seems to have taken up my cause, with their recent story which asked “What’s Wrong with the American University System?” by Jennie Rothenbert Gritz, an alumna of UC Berkeley, no less.  She agrees with my original posting that the US News ranking system is unrelated to the quality of undergraduate education, too!