On Time and Evaluation

by Scott Freeman

I was recently at a bar and jokingly attacked by a couple of friends about non-quantitative data. Consultants love them some numbers. While their jests were well taken, the underlying point was also well taken. As Hervè Varenne addressed in his position paper on anthropology and education, students of anthropology often find themselves kowtowing to quantitative research, apologizing for ‘sampling limitations’. This is to say that the type of modernity that insists on brevity and a numerically constructed objectivity may be the current that we swim against most.

The problem is, many times, this current pushes in a very problematic direction. When a particular type of research or evaluation is seen as dominant, and the only type of inquisition worthwhile, what is lost in the fray? In the world of development aid, pushes towards an auditable and measurable development are more common then ever- graduate students in schools of international development pine for “Monitoring and Evaluation” expertise. The demand is high.

However, development evaluations, or audits, still retain that potent monetary thread from the inception of audits as we know them, in the realm of the financial (see Power (1997) and Strathern (2000)). As such, the imposition of audits (or site/donor visits) in the world of development are more often oriented towards the source of the money rather than the more holistic effects of their efforts. Auditing becomes an assurance to donors, an assurance that ‘something’ has been done.

In his investigations of international aid, David Mosse (2004, 2005) asks us to question not whether an aid project is successful, but rather how success is constructed. For the implementation and evaluation of short term environmental aid projects in Haiti, this question could not be more relevant.

Soil conservation projects (the subject of my dissertation research), present potent examples of the ways in which success becomes constructed. Soil conservation is the umbrella term for a number of strategies that seek to keep soil from eroding off of Haiti’s deforested hillsides. Most often, these projects involve the digging of tree lined trenches across the slope of the hill- efforts to collect and slow the downhill path of water and soil.

Aid disbursed through the UN or USAID comes with a time restriction tied to the completion of the project. A visit to the hills by UN staff might check on the work of a local association subcontracted to do dig and plant. In one case I observed, a UN representative declared the project a failure: the saplings planted in the soil conservation trenches had died within a month. But the implementing agronomist wanted the monetary reward that comes with a “successful” evaluation. When new saplings were planted, the UN evaluator quickly returned to declare the project a success.

But here is where two temporal worlds collide: the growth of trees takes much more than mere months. Five to seven years might be required for that tree to grow. For soil conservation, more than this amount of time would be needed to see if the tree lined hillsides remain tree lined, or if the young saplings were soon eaten by grazing animals (a fairly common occurrence), or cut down.

In these cases of international aid, the evaluation is not tied to the halting of erosion. Rather, it is tied to observations based on financially imposed time limits which have little to do with the continued maintenance of a soil conservation structure. Surely the world of international aid needs a check or balance (or two). But financial audits impose a particular time scale that may be completely irrelevant for the judgment of “success”. Agricultural time, and arboreal time are measured in seasons and years. A few months hardly provide adequate time to grant “success”, yet failure and success here were granted within a few weeks. According to the agronomists who implemented these projects (and in line with my observations) more often than not, these trees would soon be gone and the land unchanged.

While many in aid do recognize the need for anthropology, it is essential not to limit the conversation to previously established paremeters. Rather, we’ll need to incur a lot more joking attacks in bars to ensure that the questions we find most important continue to be asked.

Scott Freeman is a doctoral candidate in anthropology at Columbia University Teachers College, and a visiting scholar at George Washington University, where he blogs at http://focusonhaiti.org/.


One Response to On Time and Evaluation

  1. Avatar Daniel Souleles
    Daniel Souleles says:

    A similar critique of numerical assessment occasionally comes from the rarefied world of high finance. Investors or a CEO will argue that stock traders, or markets generally are incorrectly (always under) valuing a particular company (see also Ouroussoff 2010). The grind of quarterly reports coupled with the idiot, manic scrutiny of the stock market is forcing a particular company to focus more on these inane assessments to the detriment of making quality products and profit over the long term. This is part of the argument that Ross Johnson made when he attempted to buyout shares of RJR Nabisco (Burrough and Helyar 1990), and is part of why Michael Dell is arguing shareholders should accept the offer he is leading to take Dell private (see de la Merced’s recent pieces in the NY Times Dealbook blog).

    Who would have thought though, that arguments in favor of ‘qualitative’ assessment would come from such unlikely places?