On Time and Evaluation

by Scott Freeman

I was recently at a bar and jokingly attacked by a couple of friends about non-quantitative data. Consultants love them some numbers. While their jests were well taken, the underlying point was also well taken. As Hervè Varenne addressed in his position paper on anthropology and education, students of anthropology often find themselves kowtowing to quantitative research, apologizing for ‘sampling limitations’. This is to say that the type of modernity that insists on brevity and a numerically constructed objectivity may be the current that we swim against most.

The problem is, many times, this current pushes in a very problematic direction. When a particular type of research or evaluation is seen as dominant, and the only type of inquisition worthwhile, what is lost in the fray? In the world of development aid, pushes towards an auditable and measurable development are more common then ever- graduate students in schools of international development pine for “Monitoring and Evaluation” expertise. The demand is high.

However, development evaluations, or audits, still retain that potent monetary thread from the inception of audits as we know them, in the realm of the financial (see Power (1997) and Strathern (2000)). As such, the imposition of audits (or site/donor visits) in the world of development are more often oriented towards the source of the money rather than the more holistic effects of their efforts. Auditing becomes an assurance to donors, an assurance that ‘something’ has been done.

In his investigations of international aid, David Mosse (2004, 2005) asks us to question not whether an aid project is successful, but rather how success is constructed. For the implementation and evaluation of short term environmental aid projects in Haiti, this question could not be more relevant.

Soil conservation projects (the subject of my dissertation research), present potent examples of the ways in which success becomes constructed. Soil conservation is the umbrella term for a number of strategies that seek to keep soil from eroding off of Haiti’s deforested hillsides. Most often, these projects involve the digging of tree lined trenches across the slope of the hill- efforts to collect and slow the downhill path of water and soil.

Aid disbursed through the UN or USAID comes with a time restriction tied to the completion of the project. A visit to the hills by UN staff might check on the work of a local association subcontracted to do dig and plant. In one case I observed, a UN representative declared the project a failure: the saplings planted in the soil conservation trenches had died within a month. But the implementing agronomist wanted the monetary reward that comes with a “successful” evaluation. When new saplings were planted, the UN evaluator quickly returned to declare the project a success.

But here is where two temporal worlds collide: the growth of trees takes much more than mere months. Five to seven years might be required for that tree to grow. For soil conservation, more than this amount of time would be needed to see if the tree lined hillsides remain tree lined, or if the young saplings were soon eaten by grazing animals (a fairly common occurrence), or cut down.

In these cases of international aid, the evaluation is not tied to the halting of erosion. Rather, it is tied to observations based on financially imposed time limits which have little to do with the continued maintenance of a soil conservation structure. Surely the world of international aid needs a check or balance (or two). But financial audits impose a particular time scale that may be completely irrelevant for the judgment of “success”. Agricultural time, and arboreal time are measured in seasons and years. A few months hardly provide adequate time to grant “success”, yet failure and success here were granted within a few weeks. According to the agronomists who implemented these projects (and in line with my observations) more often than not, these trees would soon be gone and the land unchanged.

While many in aid do recognize the need for anthropology, it is essential not to limit the conversation to previously established paremeters. Rather, we’ll need to incur a lot more joking attacks in bars to ensure that the questions we find most important continue to be asked.

Scott Freeman is a doctoral candidate in anthropology at Columbia University Teachers College, and a visiting scholar at George Washington University, where he blogs at http://focusonhaiti.org/.

Lawrence Cremin and Mara Mayor Discuss Technology and Education In 1989

Given the discussion of MOOCs that has been occurring in the blogosphere over the last year, I thought it might be helpful to get a longer perspective on technology and education. In that spirit, I have dug up this 30 minute conversation between Lawrence Cremin of Teachers College and Mara Mayor of the Annenberg CPB project on the role of technology in education.

The conversation is here.

The conversation demonstrates that the question surrounding the interplay of technology and education haven’t changed much in the last 25 years, or the last 50. However, as in many fields, the early formulation of the problem and the positions taken relative to that problem are clearly visible without smoke from the latest hype cycle obscuring the view.

There is a nice nugget in there about the chalkboard as a technology to personalize education.

Thank you to the good people at archive.org for preserving so many wonderful things that would otherwise be lost. If only we could turn the NSA to such useful pursuits…

More on Scientific Reductionism–this time from a conservative columnist

David Brooks, the center-right columnist at the New York Times today published a column about the limitations on neuron research.  He’s not against neural research, just the hubris that tends to collect around it.  Like research on DNA, research on neurons is great stuff—but no matter how enthusiastic the scientists may be, it does not explain the products of culture, sociality, or humanity.   And this conclusion is not the result of a political bias, but is a critique widely shared in anthropology, sociology, and beyond.  The article is here.

Good News for the State of Nevada!

I will be in Thailand this summer for five weeks teaching a course for the University of Nevada, Reno, as a Visiting Professor.  As part of the employment procedure, I had to sign a loyalty oath indicating “I will support, protect and defend the Constitution and Government of the United States, and the Constitution and Government of the State of Nevada, against all enemies, whether domestic or foreign…”  Unlike the other papers I signed for the employment, thisone needed to be notarized.  This created the amusing situation that I swore allegiance to Nevada in front of a German notary.

The good news for me: I get to teach in a creative study-abroad program in Chiangmai, Thailand.  I also got to think about the symbolic importance such oaths in structuring society: Not a bad thing for a social scientist to do now and then.

And the really good news is for Nevada.  I signed a similar oath to California when I started working for the State of California in the 1990s, and California is still safe from enemies foreign and domestic!  Nevada will be too.  Nevada, I got your back!

Discovering Exaptation: Or, How To Leverage Your Philosophical Baggage To Further Science

I want take up Tony’s question about this Dennett quote:

There is no such thing as philosophy-free science; there is only science whose philosophical baggage is taken on board without examination (Dennett 1995)

One way to answer this is through recourse to the literature on Science and Technology Studies (STS). We could weave our way through a dense web of philosophical and empirical work on scientific practice that demonstrates time and again that value-free inquiry is an illusion. Yet another way to answer is by making a formal argument using Weber as a guide.

But, because so many readers of this blog are big, big fans of Stephen Jay Gould, I will use Gould and Vrba’s 1982 article introducing the term exaptation to demonstrate that philosophical baggage can be used as a lever to discover better questions.

Gould and Vrba, in contrast to Jason Richwine and Steve Hsu, are fully aware that the categories through which you effect an analysis matter. And in what may be the finest use of Foucault within the pages of Palaeontology, Gould and Vrba start their article with the following passage:

We wish to propose a term for the missing term in the taxonomy of evolutionary thought. Terms in themselves are trivial, but taxonomies revised for a different ordering of thought are not without interest. Taxonomies are not neutral or arbitrary hat-racks for for a set of unvarying concepts; they reflect (or even create) different theories about the structure of the world.

The opening sets the tone for what follows. Gould and Vrba proceed to note the analytic term adaptation carries two differing connotations, which are subsumed under the prevailing classification scheme. They imply, by way of the opening paragraph, that the classification system they criticize has its roots in Victorian mores and morals.

Adaptation, they argue, has two meanings: historic origin and current utility. Gould and Vrba further note adaptation refers to both a process and a state. After introducing the philosophical baggage they will unpack and leverage, they proceed to tease apart the twin uses of adaptation using the tools of rhetoric in a way that Richard McKeon might have appreciated. That is, by using rhetoric to open a new view on an old paradigm.

What was gained is the concept exaptation. A trait or feature may have been adapted for one purpose but is found useful for another, often quite different, purpose. It is a concept which might be described as future utility.

Following the formation of the concept, Gould and Vrba use a few examples from the fossil record as illustrations of the concept. The best known example are feathers. Feathers evolved for warmth but were exaptated for flight. Another example they use is the case of extra or junk DNA. Many organisms carry around duplicate, spare, or otherwise unaccounted DNA, whose presence cannot be explained by recourse to adaptation. Like a brocoleur at his pile of spare parts, these organisms use their DNA junk piles to make new traits as needed.

But the main point I want to make here is simply that Gould and Vrba demonstrate that natural sciences and philosophy, like the Dennett quote indicates, are intertwined, and each can make good use of the other. Just compare Gould and Vrba’s rich conceptual development to Richwine’s unexamined and altogether ridiculous deployment of the category “hispanic,” or Hsu’s utterly lazy and unscientific conflation of IQ, SAT score and the g-factor.


In a similar vein, I wrote about Tools and Toolchains earlier this spring.

Why is it so Hard for a Ivy League Grad to Talk to His Plumber?

I just came across this article about elite education, and the habits of the Ivy Leaguers.  I really like the opening paragraphs which asks why the author, who is an Ivy League grad has so much trouble talking to the plumber who will fix his pipes.

This has a lot to do with what Pierre Bourdieu and the “habitus” of social class.  As the author, William Deresiewicz points out, this is tightly connected to how “intelligent” you are.

The Disadvantages of an Elite Education

Our best universities have forgotten that the reason they exist is to make minds, not careers

By William Deresiewicz

Continued here.


Philosophy and Reading Widely

There are two blogs I have read recently which make the good point that reading “classics” is important . At the New York Times, Philosopher Gary Gutting makes the point that a college education is not so much about “the content,” (or presumably the major) but about the habits of reading and inquiry developed. Or as he writes: “We should judge teaching not by the amount of knowledge it passes on, but by the enduring excitement it generates.”

The fact of the matter is that we cannot remember what we learned on a test a year or less after the final is given.  So what do we take away? Gutting’s answer is that it is habits of thinking, reading and approaches to knowledge that we “teach” at the university, not “the content.”  He seems to recommend that the “evaluation of teaching” or whatever it is we our teaching is assessed on at the college level, we should look at readership rates for popularly written magazines like The Economist, Scientific American, New York Times Book Review, and The Atlantic, presumably 5, 10, 20 years after our students graduate.  As he points out, the capacity to enjoy reading such journals is an indicator that students (and their employers) “reap the benefits of their education.”  Perhaps readers from other countries can point to similar magazines that the educated public enjoys (Der Spiegel in Germany comes to mind).

Along these same lines, Razib Khan at the Gene Expressions blog is urging his science readers to return to the classics now and then to do a little “cognitive tail chasing.”  He throws out Plato, Confucius, and Nietzsche by name, and points out that the reason to read them is not to find answers to contemporary issues, but to revisit the type of reasoning and argument underpinning modern thought.

Or in Razib’s own words, “smart opinions from people whose world views are fundamentally alien toward our own allow us to consider what dogmas and orthodoxies we hold as self-evident truths.”

In my view, natural scientists tend to hold to dogmas and orthodoxies a little too tightly.  Sometimes they seem to be more concerned with what is “cutting edge,” and as a result tend to see themselves as what Nietzsche called “new humans” untethered to the past when their analytical categories were first defined.  Sometimes tail chasing is just tail-chasing.  But other times it can challenge us to sharpen our own views—or even change them.

A Review of Jason Richwine’s Thesis by Someone Who Actually Read It

Anthropology Now has a review out by Elizabeth Chin who actually read Jason Richwine’s PhD Thesis on genetics and i.q. from Harvard’s Kennedy School.  The review is framed as the feedback Richwine should have received from one of his three committee members, but did not.  Chin raises many of the same issues that Michael Scroggins and I have been raising here on Ethnography.com about the nature of race and genetics.  From Chin’s writing, I think that it is pretty clear that we share with her a pretty mainstream view within in the social sciences.  I am still surprised though at the big epistemological gulf that persists between the social sciences, and some geneticists/biologists who write about behavior.  Fro this perspective, Chin is particularly prescient in pointing out the broad literature about race and i.q. that Richwine apparently ignores.

On a more general level, I firmly agree with Chin that the Dissertation Committee Members were remiss in reviewing Richwine’s dissertation at the proposal, review, and signing off stages.  Something seems to me to be quite wrong at the Kennedy School—was the dissertation even read by the faculty members?

And as a Post-script. Inside Higher Education weighed in on the Genetics and IQ blogging that is going on recently, and commented on here.  In particular, they ask some serious questions about the work of Steve Hsu, and his study to investigate the genetics of people who are do well on the SAT exams, and/or get PhDs from particular universities in the natural sciences, math, or some kinds of engineering.  There is a positive reference to Michael Scroggins’ blog in the body of the article.  For snark, you have to go to the follow-up comments!