Campbell’s Law and the Fallacies of Standardized Testing

Donald Campbell was one of the leading psychologists of the second half of the twentieth century.  His was a time of optimism for planners—there was a belief that the power of technology could be brought to bear on many of the world’s ills.  And indeed they were, often with positive effects.  As a result of central planning, more people receive water, more places are electrified, more children educated, and more diseases eradicated.  All good goals with which Campbell would not quibble.

But Campbell noticed something else to, the emergence of “corruption pressures,” based on the general principle that is now known as “Campbell’s Law.”

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

      In different ways, both Michael Scroggins and Max Holland have pointed at this basic problem in their recent blogs critiquing intelligence tests ranging from the standard tests, to the SAT and college entrance exams.  Such intelligence tests are indeed used to divide up the spoils of placement spots at elite schools, and not surprisingly, ambitious parents seek to corrupt it by means both fair or not. But for use in general analysis like that done by the evolutionary psychologists, the consequences are that their data source over time is corrupted.  The cheating scandals associated the No Child Left Behind Act are a byproduct of Campbell’s Law.  So is the fact that the SAT exam was recently cancelled in South Korea due to widespread cheating.  However most of the corruption does not come from cheating.  It also comes from the fact that such standardized tests are routinely gamed by testing companies which guarantee 100 extra points on the SAT through $1000 prep courses (I used one of these classes for my daughter—it worked!).

For what it is worth, tests like the internationally administered National Association of Educational Progress (NAEP) which do not have consequences attached to them are much less likely to be gamed.  They do not have consequences for funding, admissions, etc., attached to them because they sample across broad areas, and report results on large geographical areas rather than individuals.

Which brings up the BGI Cognitive Genetics Gene Trait Association Study of Intelligence that Michael Scroggins wrote about, and which Dr. Steve Hsu is promoting as a member of the Core Team of BGI.  The Chinese company is seeking people with “high cognitive abilities,” as defined by high scores on the SAT and other standardized tests, or PhDs in a limited number of fields (e.g. physics, computer science, electrical engineering) from “top” US universities.

The implicit assumption is that these people must have DNA which makes them higher functioning than the rest of us.  There are a number of flaws with this approach, starting with those pointed to by Donald Campbell—particularly the fact that the measures they are using long ago lost the validity and reliability due to corruption pressures.  There is also the problem that Michael, Max, and I have been hammering home here at Ethnography.com, which is that “intelligence” is always culturally defined, typically by those who have the power to define people like themselves as, well, “intelligent.”  (Perhaps this is why BGI does not want people with PhD degrees in fields they have not studied, or from universities outside the US–this is who they are).

There are of course other reasons why BGI are off on a fool’s errand, some of which is described in Chapter 5 of my recent (2012) book Schooling, Childhood, and Bureaucracy: Bureaucratizing the Child.  Chapter 5 is called “The Sorting Function of Schools: Institutionalized Privilege and Why Harvard is a Social Problem for Both the Middle Class and Public School 65 in the Bronx.”  For that matter Chapter 8 “Seeing Like a State: Efficiency, Calculability, Predictivity, Control Testing Regimes, and School Administration” is also relevant.  (Sorry the book is still only out in hardcover at $90, and Kindle for $72—check your library for a copy, or wait for the paperback version).  To summarize the findings in my book:  Success on tests are inevitably associated with reproducing the status quo, whatever status quo the elites of the day might be promoting.

As for Campbell’s Law, I hope that the people organizing such projects as the Gene Trait Association Study of Intelligence read Donald Campbell’s article carefully, even if he is not an electrical engineer or physicist with a PhD from a top US University, or an 800 on the math portion of the SAT.

Mla Bri Genetics and Anthropology in Northern Thailand

Many anthropologists are concerned with the tendency of biologists to reduce social life in general, and culture in particular to the genes people carry.  As a sociologist, I share that concern.  I think that such reductionist approaches give a false sense of precision to the concept of culture which while very real, is often messy at the edges in very human ways.

 

I came up against this tendency to “reduce” everything to genetics recently in an article about the Mla Bri of Thailand, a small group (200-300 people) speaking a Khmuic language in northern Thailand.  As a group, they attract the attention of anthropologists because, until recently, they did a lot of hunting and gathering for subsistence, while also being engaged in exploitative labor practices with neighboring groups speaking a range of languages, including northern Thai, Hmong, Mien, and probably Khmu.

 

Anyway, in 2010, a second article about Mla Bri genetics was published in BMC Genetics using blood samples collected from Mla Bri in 1999 by visiting geneticists.  These samples were then compared to blood samples in a bank from the other ethnic groups found in northern Thailand, including northern Thai, Hmong, Mien, Khmu and so forth.  But the geneticists doing the work did all their work at the laboratory bench, and did not familiarize themselves with the geography of the Thai highlands, or the unequal relationships between the Mla Bri and the neighboring groups.  Their conclusion was that the Mla Bri have been isolated culturally and genetically from the Hmong and other groups for a long long time.  I am confident that their work at the laboratory bench was sound.

 

But, I did have a basis to question their lack of ethnographic context—you see, I have good friends who have lived with the Mla Bri as missionaries for the last 30 or so years.  Gene and Mary Long speak Mla Bri, and were even present with the blood samples were drawn in 1999.  They are also among the best “gut level” anthropologists I’ve ever met.  Anywya, On the basis of what they knew about the Mla Bri, we evaluated the article in BMC Genetics, and wrote a comment which was posted this week.  This posting makes the point that the genetics work published in 2010 would have been greatly enriched if the authors had talked to the people living in the village, whether it be the Mla Bri themselves, or the Longs.  Indeed, it is well-known among the Mla Bri that despite strong norms for endogamy, extra-marital relationships do exist, and that exogamy does occur.

 

Our Comment in BMC Genetics is available here:

 

The article we are commentin on is here.

 

A similar exchange that I had about the Mla Bri in 2005 in the pages of PLoS Biology is here.

 

The original 2005 article about the Mla Bri genetics is here.

 

If you want to know more about the Longs and their work with the Mla Bri, there will be an article about suicide among the Mla Bri in the 2013 issue of the Journal of the Siam Society which should be on-line soon!

 

Bottom line:  Be wary of the cultural reductionists, be they the selfish-gene type, the lab bench type, check the box type, or any other such type.  For a comment on this, see here!

“Building Bildung,” and Other Improbabilities among German University Undergrads

 

German has two words for the English word “education.”  Erziehung describes the school system, and the mechanics of what is taught and conveyed from the world of adults to that of children in order to “bring them up.”  Focus is on skills adults need like literacy, numeracy, history, and the factual basis citizens need to understand to participate socially, culturally, and economically in society.  The German education system is designed to educate all children in such basic skills. It is something that is done for children, and leads to practical apprenticeships/schooling which many German youth begin at ages 15 or 16, i.e. after completion of  9-10 years of schooling and in turn lead into the workforce.  This type of schooling makes for a very disciplined and skilled workforce, which is able to produce engineering wonders like Audi, BMW,  Mercedes and Siemens that power Germany’s modern export-led economy.

 

But there is another word in German for education, which is “Bildung,” which is a more important quality, and the one which is more highly valued even though it is not aimed directly at workforce preparation.  Bildung it is tucked into the programs of the primary, secondary, and really kicks in at the university level, including at Leuphana University where I am currently a guest professor.  Bildung roughly means “cultivation of the intellect.”  Unlike Erziehung, such cultivation is not something that is done for you, rather it is a quality that you as an individual cultivate as a matter of intellectual habit.  This is why primary and secondary schools in Germany have a curriculum in music, art, history, religion, the social sciences, philosophy, and so forth.  Famous German philosopher-types have written about this word and emphasize the quality of “cultivation of the intellect.”  Related to this, Germany prides itself on being the land of “poets and thinkers.”  Besides BMW, Mercedes, Audi, and Siemens, think also Luther, Goethe, Hegel, Schiller, Brothers Grimm, and Max Weber.  And in music, it’s Beethoven and Bach just for starters.  Indeed, in a country where political and military achievements are looked at with skepticism, bemusement, and sometimes disgust, cultivation of the Bildung is critical to a sense of adult identity.  The result for the education system that children are expected to develop habits of intellectual cultivation for their own sake–and to appreciate the cultural resources which are the product of such habits.  By the time they reach university, students are expected to do this on their own, without a lot of prompting from university professors like me.

 

To illustrate, here is a quick email I received from a student in my Post Colonial Theory class at Leuphana University last week, with a question about her term paper.  I know her well because she came to my classes for the last 14 weeks, even though there is no grade or “incentive” given for attendance; indeed the paper is the only graded work she will do for me this semester, and it is due after a six week writing period in which there are no classes.  As a full-time student, she will be completing 5-6 such papers during this period. She wrote:

Dear Mr. Waters,

you suggested to me to use Fanon and Wallerstein for my term paper about (Country X), but I would rather like to use Said and Spivak if possible. It seems pretty difficult to apply Fanon’s and Wallerstein’s theories on (Country X), since they are more focused on race, ethnicity and the relations between 1st and 3rd world or rather colonies which were overseas. It seems like Said and Spivak might fit better, also since (Country X) literature and articles about a postcolonial (Country X) refer to their theories. I still would like to refer to Fanon a little bit too. But of course I would like to know what you think about this?

Best regards,

Leuphana Undergrad in Culture Studies

What do I think?  After 15 years working at a mid-level American university, I think cool—go for it.  I mean an email like this from an undergrad is really cool!  I wonder how you could work in three languages (English, German and Language X), and where you came to have such a wide interest in social theory.  My students in America would have taken my cookie-cutter advice about using Wallerstein and Fanon, and left it at that.  But if the Language X literature leads to Said and Spivak, and you’re game to read that—go for it! And I am really looking forward to learning something from your paper about both Country X, and new ways of applying post-colonial theories.

 

The student’s years of cultivating the habits of the mind that are German Bildung in primary and secondary school are what prepared her to write such an email as a second year university student.  She has habits of reading widely, questioning sources, and engaging creatively.  What is more, she assumes that such things are normal educated humans do, as indeed it is  among my students here.  The funny thing for this US American professor is that any number of my students here in Germany could have written such an email—this is just the one that was handy.

 

But, I have rarely in my years of teaching in the US received an email like this.  Rather the focus of the US American student is on the formula that will get a good grade on my paper—And, truth be told, I give it to them: Have a clear introduction with a thesis, illustrative examples in the body, and a good conclusion to tie things off.  In other words mechanics and process are the issue I my conversation with US students.

 

Which brings me back to the subjects of Erziehung, Bildung, and what is now my new pet peeve, Chickering and Gamson’s (1987) Seven Practices of Good Undergraduate Practice.  In my personnel file at Chico State, I repeatedly reflected on these seven principles at the behest of administrators concerned that I earn my keep, i.e. be “accountable.”  As I wrote previously, I think that the insistence of US American administrators on using these “best practices” have over-emphasized process at the expense of what is in essence, Bildung.

 

Before I finish this blog, here are the Chickering and Gamson’s (1987) seven processes which administrators claim underpin high quality undergraduate teaching.  Note that little of this refers to student learning or acquisition of Bildung.  Rather it is about the faculty can be supervised on, and therefore held accountable for by administrators.  Good Teaching Practices include:

 

1. encourages contact between students and faculty,

2. develops reciprocity and cooperation among students,

3. encourages active learning,

4. gives prompt feedback,

5. emphasizes time on task,

6. communicates high expectations, and

7.respects diverse talents and ways of learning.

By and large, these are great principles of educational process.  But indeed they leave out Bildung.  Nowhere in these seven principles is there any mention of thinking, reading, capacity to think abstractly which is what my student’s email demonstrated.  Nothing in Chickering and Gamson (1987) for the land of poets and thinkers!  Rather it is designed for the land of process engineering—the world Frederick Taylor imagined when in 1911 he published The Principles of Scientific Management.  (Note to American students reading this: If you don’t know who Frederick Taylor was, check out his Wikipedia entry).

One consequence are email from German students like the one above who, by the way, has had little personal feedback from me during the semester, nor do I have any clue about how much “time on task” she has spent on any assignment.  But she does have high expectations for herself, and she learns and writes in German, English, and Language X which is certainly diverse, though perhaps not in the way Chickering and Gamson were thinking of diverse learning styles.  What she has are habits of the mind that are what the German system somehow cultivates. I guess this could be called active learning, though I have no idea what I did as a Professor to encourage it.  Which of course brings me back to Professors Chickering and Gamson (1987), and their Seven Principles.

What would happen if instead of addressing their seven principles on my next course syllabus, I was to melodramatically tell my US American students: Give me Bildung (Habits of Mind), or Give me Death!

I write about Bildung and other such improbabilities in the US American K-12 education system in my new book Schooling, Bureaucracy and Childhood: Bureaucratizing the Child.  Request it from your library, or if you have enough money, order a hardcover (or Kindle) copy from your favorite bookseller!

 

 

Did Chickering and Gamson’s (1987) Seven Practices of Good Undergraduate Education Dumb Down American Education? A View from Germany

I just finished my semester teaching as a Guest Professor at Leuphana University in Germany, and am beginning to figure out an answer to a question I get asked frequently: “What is the difference between American and German universities?”  Actually, the German students who have been exchange students in the United States have helped me along.  They say:

 

In American universities, there’s lots of assignments, busy work really, and the students are not expected to take responsibility for their own learning.  Really it is like [German] high school, not the university.

 

German university classes in contrast to American universities do not involve a lot of graded work during the semester.  The credit for the entire course comes down to a few ungraded assignments during the semester (typically including a 20 minute oral presentation), and a final test or paper which is the only thing actually evaluated for a grade. Class itself is a combination of student presentation, faculty lecture, and if it is small enough, seminar.  In the case of the paper, it is usually 10-12 pages long—one for each class.  There are no incentives for attendance.  There are also is no preparation for midterms or quizzes–because there are few such tests in the American sense.  The term paper for the class is handed in is due about six weeks after the semester ends. The professor then hands in a grade sometime in the next few months, which is then reported to the student, without elaboration or feedback.  And that’s it—if you pass, you go on to the next semester.

 

In other words, at this German university, there are classes in which there is little encouragement for contact between students and faculty, professors do not initiate cooperation among students (that is up to the students), there is little feedback from the faculty regarding student work (and feedback need not be prompt), time on task is not monitored, and no one really cares if a student has another way of learning that does not show up on assessment.

 

The good news, though is that there are high expectations for German students, and if they meet the high expectations, they get a degree at the end of three years or more.  Oh, and before they can also organize complex ideas in their 10-12 pages, read voraciously, and are up to date on current events both within and outside their fields.  But they are this way not because of class content and monitoring by university faculty like me.  Rather they are like that because that is what a good student is.  The 20-30 minute class presentations are generally include high quality analysis, too.  Also, with most German undergraduates you can have an well-informed conversation about their lives or the events of the day in English, German, and perhaps another language.

 

But is this really good practice in undergraduate education?  According to the gurus of American undergraduate education, Professors Chickering and Gamson (1987)  the German system is a lousy system which should produce

 

Apathetic students, illiterate graduates, incompetent teaching, impersonal campuses — so rolls the drumfire of criticism of higher education.

 

The problem of course is that my German students are far from apathetic, certainly not illiterate, and judging from the fact that major political parties show up on campus to court the student vote they are far from apathetic.  As for the incompetence of teaching, I cannot judge that from my perspective, except to note that my German language teacher was pretty good!

 

Nevertheless in response to the problem that Professors Chickering and Gamson (1987) diagnosed regarding apathy, illiteracy, and incompetence, they prescribed seven best practices for high quality undergraduate education which American faculty are routinely held to in faculty reviews:

 

1. encourages contact between students and faculty,

2. develops reciprocity and cooperation among students,

3. encourages active learning,

4. gives prompt feedback,

5. emphasizes time on task,

6. communicates high expectations, and

7.respects diverse talents and ways of learning.

 

A German academic high schools (Gymnasium) teacher might recognize Chickering and Gamsons (1987) criteria, but the German university system I described above only meets 1/6 of Chickering and Gamson’s (1987) goals, i.e. the one about high expectations.   The other six, are considered to be the responsibility of the student, not the faculty, which is why of course my German students hold American undergraduate education in such low esteem, and typically consider their exchange experience to be “high schoolish.”

 

The German view is that university faculty is in charge of organizing interesting class to which students will come if they are inclined, and participate when they are prepared.  If they show up sort of regularly, and do the major assignment/test at the end of the semester, they will get credit for the course.  University faculty are not in charge of checking endless assignments, homeworks, administering quizzes and the many other tasks which are really about checking for “time on task,” the metric that is most valued by administrators and faculty?

 

Could it be too that the reason that the German students believe that American undergraduate education is “like high school” is the fault of Chickering and Gamson (1987)?  I bring this up because since I started teaching American undergraduates in 1996, the administrators who hired me have asked me to include Chickering and Gamson’s (1987) seven goals of good undergraduate practice in my employment dossier.  I have dutifully done this, and in a show of faith upped the number of assignments in my classes, made a point of returning assignments quickly, encouraged active learning in the assignments I make, tried to make sure that I assigned 2 hours of reading/out of class work for each Carnegie unit the students receive, and checked to see if they did it by using quizzes.  I am an accountable professor!  But does this really make me a good professor who creates students who think creatively and deeply about sociology as I see my German students doing?

 

And anyway, what do we in the American system get in response?  Undergraduates who are like, well, high school students in their capacity to work independently (meaning they work like well-supervised high school students).  The expectations authored by Chickering and Gamson (1987) have become a self-fulfilling prophecy.  We defined high quality as demanding a lot of busywork (not critical thinking, routine reading, or the capacity to work independently, and surprise, we get undergraduates who

 

–contact faculty frequently

–do group work

–equate active learning with recreational activities

–expect prompt and voluminous feedback rather than editing themselves

–spend a lot of time doing homeworks

–read for the detail on quizzes rather than for “fun”

–have high expectations of their own abilities, and

–expect learning accommodations to be made for different learning styles

 

This is not to say that German students are faultless.  Indeed, the German system has recently undergone major reform because students frequently floated away from a system that by American standards can be pretty brutal (it is easier to flunk a student you do not know, than one who has frequent contact with faculty).  German students also take too long to graduate, and so forth.  I should add, that my experience last semester is that American attendance patterns are a bit better than German, and punctuality is also a lot better at Chico State, where I routinely start class within 15 seconds of the class start time, a process which seems to take 5-10 minutes here…).

 

Still, when the German students do finally get to class, they bring intellectual preparation that my American students do not.  But then again, habits of intellectual engagement, like voluminous reading, independent thought, oral presentations, and routinely crafting 10-12 page essays are really not part of what Chickering and Gamson (1987) have defined as high quality undergraduate education, are they?

 

Reference

Chickering and Gamson (1987).  There are hundreds if not thousands of copies of Chickering and Gamson’s original article floating around on the internet.  If you really want to read it (I encourage you to do so), either click the link embedded in the article, or simply google it up!

Why Does Anthropology Worry about Jared Diamond when they have Nigel Barley?

The Anthropology blogosphere (including Ethnography.com, SavageMinds.org, anthropologyreport.com and even National Public Radio) has recently lit up with critiques of Jared Diamond’s new book The World Until Yesterday.  Jared Diamonditis seems to be a regular affliction of anthropology, re-emerging every time that the esteemed Professor of Geography (and Physiology) publishes a new tome of big picture history.  The manner that Diamond does this is something that anthros really don’t seem to like.  This is because besides his own field of Geography, Diamond borrows data liberally from all four fields of anthropology to make big generalizations in a manner a cultural geographer, comparative historian, or field ecologist might. But oh yeah, Diamond is a geographer by departmental affiliation, and a field ecologist by training and predilection.

It also seems to bother anthros that Diamond also on occasion—though not always—wanders off the reservation and lets his political views seep into his analysis.  And since these political views don’t typically jibe with those of the anthros, particularly when it comes to oil companies, well you get the idea.  But then there is a counterpoint, someone finally ends up pointing out that since no anthro since Eric Wolf has done such big picture stuff in Europe and the People without History published way back in 1982, anthro has no right to complain.  And so it goes back and forth until the next big tome from Diamond comes out, and Jared Diamonditis flares up again.

Ok, that’s my two paragraphs for the current “controversy.”  In response, I want to write about an anthropologist—an ethnographer actually—who I think is greatly undervalued in anthropology, Nigel Barley.  Barley describes well what anthropologists do best in The Innocent Anthropologist: Notes from a Mud Hut published in 1984.  This is the book I point students to when they want to understand field work, ethnography, and cultural anthropology.  As a sociologist, this is one of the anthro books I truly admire, because it reflects well on my own field experiences in Tanzania.  Oddly, I find few anthropologists who have read it, much less heard of it.

The Innocent Anthropologist is a memorably written story of Nigel Barley’s experience doing fieldwork among the Dowayo in rural Cameroon in the early 1980s.  The strength of the book is that it includes the personal problems that emerge with the frustrations, boredom, tribulations, and mis-interpretations that inevitably emerge in the context of “doing ethnography.”  In this sense the book is much different than the dispassionate, theoretical, and methodologically rigorous ethnography typically assigned undergraduates.  In such ethnograpny in  the ethnographer somehow ends up erudite, insightful, and making references to Bourdieu and Baudrillard while drinking the local brew.  Nothing wrong with this, but let’s face it, it is not the sort of thing that a 19 year-old taking your Intro to Cultural Anthro course for General Education credit identifies with.

Barley also does a great job explaining the nuts and bolts of doing ethnography in a remote Cameroonian village.  There are empathetic descriptions of coming-of-age rituals, ancestor cults, gender relations, the agricultural cycle, and a well-written nod to Malinowski.  There are also empathetic passages describing boredom, cross-cultural frustrations, and hilarious language learning errors.  And what students will really remember is Barley’s explanation of how the mechanic at the dentist’s office removed his two front teeth.  Such an account would never make its way into a standard ethnography (sorry, no spoiler here–you need to get the book!). And of course such tales, which are really the center of the ethnographic experience are left out by the likes of the ever-dignified Professor Malinowski.

But the scene from Barley’s book I spend most of my time mulling about is at the very end, and has little to do with Africa, but everything to do with ethnography, culture, and the human condition.  Barley spent a year and a half in Cameroon being bored, sick, confused, and frustrated while ostensibly “doing ethnography.” Oddly though, after returning to England, he still wants to tell everyone he meets about this wonderful world he encountered in Cameroon—something that he quickly discovers no one really cares about.  Or worse, they treat him like a raving lunatic because he approaches everyday problems with a vigor and habitus appropriate to a Cameroonian village, rather than that of a staid tweed-jacketed English lecturer.

So Barley returns to England, where he finds out that life is—as it had always been, despite his field work in the Cameroon. People ask him how Cameroon was, complain about the English weather, and then launch off into conversations about the more mundane things of life, like what was on television the previous evening, or the doings of the local football team.  Most mundane is the friend who complains because Barley left a sweater at his apartment some two years ago—could he please pick it up some time?  Like, who cares about a sweater when you have been dealing with ancestor cults, goat farts (sorry no spoiler on that one either!), shamanistic ritual, and have lost your two front teeth!?!?

But this indeed is how the big adventures of life often end: In a question about a forgotten sweater.  This happens whether we are ethnographers, archaeologists, or any other kind of long-term traveler who becomes embedded in a new culture.  Certainly it happens to my undergraduate students who leave home for Chico State the first time, and then return to the parents at Thanksgiving or Christmas brimming with tales of college life, only to be told by their parents to be sure to eat enough lettuce and clean up their room.  Indeed such dissonance happens to anyone returning from a adventure in which they embed themselves in a culture different from their own.  And this indeed is the great ethnographic lesson Barley teaches my undergraduates.  What is more, it is a lesson every bit as big as what Jared Diamond makes with his massive tomes.

Oh, despite his frustrations, whining, and moaning, did I mention that Barley returned to the Cameroon a few months later?  He was indeed hooked on field work and the need to experience new cultures, as we hope our students will—after all the complaining and lost teeth, he was back in Cameroon as quickly as he could.

It has long mystified me that The Innocent Anthropologist is not a staple of Intro to Cultural Anthropology courses.  It is well written, funny, empathetic, theoretical, and easy to read.  And students are happy to read it—the whole thing.  Most importantly, it is a fantastic introduction to what ethnographers do, why they do it, and what an anthropological viewpoint has to say about not just a small place in Cameroon, but the human condition.  I have used this book in my undergraduate social science classes a number of times, and it has always worked well to get students dreaming about the possibilities of culture and travel—i.e. the things that I would expect a good Intro to Cultural Anthropology course to do.  And the neat thing is that it can do it by celebrating what anthropology does best—while leaving poor irrelevant Jared Diamond out of the story.