Search Results : Campbell's Law

  • Campbell’s Law and the Fallacies of Standardized Testing

    Donald Campbell was one of the leading psychologists of the second half of the twentieth century.  His was a time of optimism for planners—there was a belief that the power of technology could be brought to bear on many of the world’s ills.  And indeed they were, often with positive effects.  As a result of central planning, more people receive water, more places are electrified, more children educated, and more diseases eradicated.  All good goals with which Campbell would not quibble.

    But Campbell noticed something else to, the emergence of “corruption pressures,” based on the general principle that is now known as “Campbell’s Law.”

    The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

          In different ways, both Michael Scroggins and Max Holland have pointed at this basic problem in their recent blogs critiquing intelligence tests ranging from the standard tests, to the SAT and college entrance exams.  Such intelligence tests are indeed used to divide up the spoils of placement spots at elite schools, and not surprisingly, ambitious parents seek to corrupt it by means both fair or not. But for use in general analysis like that done by the evolutionary psychologists, the consequences are that their data source over time is corrupted.  The cheating scandals associated the No Child Left Behind Act are a byproduct of Campbell’s Law.  So is the fact that the SAT exam was recently cancelled in South Korea due to widespread cheating.  However most of the corruption does not come from cheating.  It also comes from the fact that such standardized tests are routinely gamed by testing companies which guarantee 100 extra points on the SAT through $1000 prep courses (I used one of these classes for my daughter—it worked!).

    For what it is worth, tests like the internationally administered National Association of Educational Progress (NAEP) which do not have consequences attached to them are much less likely to be gamed.  They do not have consequences for funding, admissions, etc., attached to them because they sample across broad areas, and report results on large geographical areas rather than individuals.

    Which brings up the BGI Cognitive Genetics Gene Trait Association Study of Intelligence that Michael Scroggins wrote about, and which Dr. Steve Hsu is promoting as a member of the Core Team of BGI.  The Chinese company is seeking people with “high cognitive abilities,” as defined by high scores on the SAT and other standardized tests, or PhDs in a limited number of fields (e.g. physics, computer science, electrical engineering) from “top” US universities.

    The implicit assumption is that these people must have DNA which makes them higher functioning than the rest of us.  There are a number of flaws with this approach, starting with those pointed to by Donald Campbell—particularly the fact that the measures they are using long ago lost the validity and reliability due to corruption pressures.  There is also the problem that Michael, Max, and I have been hammering home here at Ethnography.com, which is that “intelligence” is always culturally defined, typically by those who have the power to define people like themselves as, well, “intelligent.”  (Perhaps this is why BGI does not want people with PhD degrees in fields they have not studied, or from universities outside the US–this is who they are).

    There are of course other reasons why BGI are off on a fool’s errand, some of which is described in Chapter 5 of my recent (2012) book Schooling, Childhood, and Bureaucracy: Bureaucratizing the Child.  Chapter 5 is called “The Sorting Function of Schools: Institutionalized Privilege and Why Harvard is a Social Problem for Both the Middle Class and Public School 65 in the Bronx.”  For that matter Chapter 8 “Seeing Like a State: Efficiency, Calculability, Predictivity, Control Testing Regimes, and School Administration” is also relevant.  (Sorry the book is still only out in hardcover at $90, and Kindle for $72—check your library for a copy, or wait for the paperback version).  To summarize the findings in my book:  Success on tests are inevitably associated with reproducing the status quo, whatever status quo the elites of the day might be promoting.

    As for Campbell’s Law, I hope that the people organizing such projects as the Gene Trait Association Study of Intelligence read Donald Campbell’s article carefully, even if he is not an electrical engineer or physicist with a PhD from a top US University, or an 800 on the math portion of the SAT.

    FacebookRedditStumbleUpon


  • Campbell’s Law, Planned Social Change, Vietnam War Deaths, and Condom Distributions in Refugee Camps

           Donald T. Campbell was a psychologist in the heyday of the 1970s. During this time, the belief emerged that society was a social engineering project that could be planned and evaluated.  The general idea was  that if you collected enough data, you could plan and control social change in a way that led to desired results.  Economists from USAID believed this about economic development, military planners in Vietnam believed it, and Sociologists in the War on Poverty believed it.  But by 1976, Campbell wasn’t so sure…

           The generation of social scientists Campbell critiqued ran around measuring poverty, illiteracy, disease, Communism, and other bad things.  Thus in the 1970s you had Wars on Poverty, Smallpox, Illiteracy, Drugs, and so forth.  There were also violent wars in Vietnam (for the Americans), and in Afghanistan (for the Russians).  When I lived in Tanzania in the 1980s, the Tanzanian government had wars on Poverty, Ignorance, and Disease, all funded by international donors living out this paradigm.  Planners in Washington, New York, Dar Es Salaam, and elsewhere calculated with statistical precision what was needed for victory in their “war,” and allocated government money to produce the desired victory.  Their decisions were “data driven” and “evidence based,”  to borrow two words common in policy making circles today.

           Campbell was involved in such projects himself.  He was so much part of them that he wrote an unfortuantely obscure paper, “Assessing the Impact of Planned Social Change” which reflects on the psychology of planners.  More interesting for this blog, though, is the fact that what he was really doing was taking the ethnographic temperature of number-assessed planners.   Campbell’s Law is as follows:

    The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

          In Vietnam, Campbell pointed out, the quantitative social indicator was “enemy killed.”  Thus, he noted, this measure was corrupted as dead civilians were re-defined as “enemy,” and occasionally villages were invaded in the hope that at the unit would have kill-metric which could be rewarded.  Two examples added by other social scientists including the following: Cardiac surgeons declined to operate on seriously ill because such patients were more likely to die (duh).  They did this because the state began issuing “scorecards” rooted in survival rates.  So, since the very sick were the most likely to die in surgery (or on their own), the doctors declined to operate on the seriously sick, and preserved their high survival rates.  Another example of Campbell’s Law comes from airline schedules.  Airlines began to be scored on the basis of “on-time arrivals” in the 1980s.  They responded by simply increasing estimated flight times, thereby driving up their “on-time” rates—anytime you arrive early at a destination, thank Campbell’s Law; tailwinds probably did not have much to do with it!

         Citing “Campbell’s Law” when critiquing the United States’ “No Child Left Behind Act” is something of a fad in education circles today.  This is because high stakes testing for science and math drives decision-making about student promotion, teacher retention, and school closures.  Thus, you get extensive test prep of students in reading and math, with resultant dilution of subjects like history, science, music and the arts which are not tested for.  And of course, the ultimate vindication of Campbell’s Law are the cheating scandals by schools and teachers concerned only about “succeeding” on test day.

          Campbell’s law also applies well to other bureaucratic endeavors, especially those of applied social scientists.  My own experience is in Tanzania where projects to assist refugees or villagers were created with quantitative goals and objectives to satisfy donors, independent of what was needed or wanted by the villagers (or refugees) they were assisting.  My favorite version of Campbell’s Law was the many broken diesel powered water projects that littered western Tanzania in the 1980s and 1990s.  Indeed a book called Watering White Elephants was written about this phenomenon.  Many of these were funded with the bureaucratic “Health for All by the year 2000” goal of WHO in mind. Quantitative reports showing that the goals of the project in terms of villagers served, villages with pumps, etc., were met. 

          For the refugees I worked with in Tanzania between 1994 and 1996, a good example was the numerical goal established for birth control in the Rwandan refugee camps.  This was right after the Rwanda genocide, and the UN was concerned about the exploding birth rates, and the costs that would be incurred by their child health programs.  The result was a bright idea: Condoms all around!  In `1995-1996, four million condoms were distributed in record time by a USAID program, a quantitative  result trumpeted at NGO meetings I attended (Quick: 4 million condoms spread across 450,000 refugees means that USAID is assuming what about the frequency of refugee sex???). 

          The visiting anthropologist hired to evaluate the program though pointed to the corruption of the condom distribution program.  The condoms, she found were not used to prevent births, which continued to rise quickly, even nine months (or more) after the big distribution.  Rather the condoms became a marker for young men to display their prowess.  The young men cut off the end of the condom and wore it as a bracelet to represent conquests.  Campbell’s Law wins again!

          Indeed, there is a ethnographic field worker’s version of Campbell’s Law which was written by the development economist Teodor Shanin in 1966 at the height of the Cold War.  Central planners in Moscow, Washington, and Beijing were running around the world applying the econometric models (Washington), or assumptions about central state planning (Moscow and Beijing) to Third World projects.  The result was Campbell’s Law writ large, as the planners with their emphasis on production targets, development plans, and so forth created goals which implementers adjusted their programs to match.  The result was that in places like the Congo, Vietnam,and  Afghanistan, all the great powers were ultimately frustrated.  Echoing Campbell’s law, Shanin wrote about the corruption of the quantitative social indictors in the following way:

    Day by day, the peasants make the economists sigh, the politicians sweat, and the strategists swear, defeating their plans and prophecies all over the world—Moscow and Washington, Peking and Delhi, Cuba and Algeria, the Congo and Vietnam”  Shanin (1966).

          Which in the end points to the strengths of the ethnographic method, since after all Campbell’s Law applies to quantiative measures, not qualitative.  As long as ethnographers are the harmless fuzzballs on the wall, they are able to write about processes, interactions, relationships, and so forth that quantitative measures typically miss.  After all, Campbell’s Law itself is ultimately an ethnographic conclusion about the nature of quantitative methods. Ethnographic method may not be grand, or easily adapted to manage large bureaucratic projects, but in its insight, it can be used to describe the limitations of more quantitative projects.

    FacebookRedditStumbleUpon


  • Human Genetics and Social Theories

    It has been a lively week on this blog. “DAD” and Razib Khan have challenged our (Michael Scroggins and myself) basic competency to discuss genetics and race/intelligence/etc.   We have responded with similar incredulity to their ability to critique anthropology.

     

    In other words, we in the social sciences think they are naïve, and they think we are dunderheads.

     

    This overall does not seem to be a very productive set of assumptions to go forward with; Razib and DAD actually appear to be relatively well-read people, though clearly we do not read the same things.  So what I propose here is that each of us propose what books/readings would be part of a graduate “Human Genetics and Social Theories” class.  I will start, while knowing that what I propose will probably annoy the geneticists (I guess I already have at some level).  They should feel free to call me on my naivete without concern:  I am a tenured full-professor fully capable of taking critique. Likewise the last Genetics course I took was in 1979 which was really a long time ago.  My last Anthropology course was about 1991.  Anyway, here goes some initial suggestions:

     

    1)  Stephen Jay Gould, The Mismeasure of Man (and yes I know that Gould has been accused of mis-representing his data)

     

    2)  Jonathan Marks What it Means to be 98% Chimpanzee

     

    3)  Michael Young The Rise of the Meritocracy, 1870-2030

     

    4)  And because I really don’t like the misuse of psychometrics: Ross Douthat, Privilege: Harvard and the Education of the Ruling Class, Daniel Golden, The Price of Admission: How America’s Buys its Way Into Elite Colleges, and Nicholas Lemann, The Secret History of the American Meritocracy.

     

    5)  Durkheim on religion social groups

     

    6)  I’m open to parts of Jared Diamond’s Guns, Germs, and Steel (and Collapse), and Steven Pinker’s new book on violence.  Both Diamond and Pinker are in places quite good (though I’m critical of Pinker’s socio-biology which appears in the latter part of the book, and Diamond’s forays into using his data to advocate social and economic policies).

     

    7)  My own critique of how geneticists use data from the Mlabri, which are here and here.

     

    8)  Reviewing psychologist Donald Campbell’s critique of the misuse of quantitative data is always a good idea too.

     

    Ground rules for proposing readings:  Please avoid charges of racism, general fraud, left-wing creationism, and so forth when it comes to the people commenting.  If you want to call the authors of the books proposed by these names, it is fine with me.  Idiot and moron are out though, since they are archaic psychological a terms (see Gould).  If you want to call me crazy, though, that’s fine.

     

    Please try to stick to questions of scholarship, and focus on advocating your own proposals, though of course brisk critique of others is welcome.

     

    Anthropologists might keep in mind what John Hawks recently wrote on his blog:

     

    Like any radicals, [anthropologists] weren’t always right. Any working scientist will be wrong about most of the details, if we revisit his work after fifty years. What makes anthropology weak today is that so many anthropologists learn nothing about scientific anthropology after Boas. They’re reactionaries against science, without knowing what today’s scientists do.

     

    [But] consider our scientific history. With sheer empirical observation, anthropologists unshuttered the folds of humanity, raising people who had been derided as “primitives” up to their rightful place beside the pampered dons of Western culture. In so doing, their science transformed “civilized” culture itself….

     

    We can be part of the future by reinvigorating anthropological science and by developing a deeper conversation with other scientists outside anthropology. Tomorrow’s anthropologists must know the field’s successes as well as its failures. The way to combat bad science is to do better science.

     

    What I am hoping for is that “deeper conversation” Hawks writes about.

     

    As for what Geneticists want to keep in mind…I’ll leave that to Razib or DAD to provide!

    FacebookRedditStumbleUpon