Finding the real impact of agricultural innovation support

To measure overall impact, it is best to focus on intermediate outcomes where links are clearer, says Giel Ton.
 
Impact evaluation in agricultural development is a hot topic. And so are the debates on methodology.

Most donor agencies that commission studies into the impact of agricultural development are interested in reporting on their contribution to meeting the Millennium Development Goals, with income generation and poverty reduction as main objectives.

In response, sophisticated research designs and econometric analyses (empirical testing of economic models) have been developed to isolate the net effect of development interventions and measure changes in household income. These analyses assume there is a correlation between a project's activities, or the treatment variable, and income, the outcome variable.

But often, there are so many factors that influence an outcome that attributing change to a particular intervention becomes impossible. Even with a relatively simple intervention such as agricultural extension (where research findings are promoted to improve agricultural practices) or small-grants to support experimentation, it is notoriously difficult to measure its impact on net income.

For instance, a grant can enable a farmer to buy a device that regulates milk quality, helping to meet buyers' quality requirements. This can improve market access and income.

However, the grant is just one factor behind this change in income. Impact cannot be attributed to a single part of this configuration — just as it takes all three of matches, fuel and oxygen to start a fire.

As a result, even sophisticated evaluations designed to measure the impact of agricultural development projects prove rather uninformative. So how can evaluations assess the effectiveness of development interventions that contribute to complex change processes? How can they give practitioners and policymakers the information they need to choose the right approach to facilitate innovation?

Intermediate outcomes

Rather than targeting end points, or ultimate outcomes, such as improving household incomes, evaluations should focus on the changes in immediate and intermediate outcomes that interventions bring.

Intermediate outcomes might include changes in knowledge, behaviour and organisational strength, which are more directly related to project activities. Based on these changes, one can reason through the intervention's likely net effect on the ultimate outcomes rather than attempt to measure this impact.

Take, for example, certification programmes that train farmers in good agricultural practices to try to boost income. These programmes can better 'prove' impact by measuring differences in adoption of these practices than by detailed and time-consuming measurements of income streams.

Detailed income measurements are the costliest part of data collection and data cleaning. But they often contribute little to drawing conclusions. In many studies, changes in income end up being influenced by additional factors — such as social organisation and history, changing weather patterns and volatile market price — that differ in place and time, and affect the treatment group differently from the control group.

“Collecting adequate data  would give practitioners and policymakers the information they need to decide on the merits of differing approaches to farmer innovation.”

Giel Ton

This results in, at best, an absence of statistically significant differences between the treatment and control groups — or, at worst, 'data-fishing' (selective use of data to arrive at misleading conclusions).
Instead, the effects of improved agricultural practices on income can be supported with more credible evidence by using data from trials conducted over multiple growing seasons to extrapolate from the measured changes in adoption of agricultural practices to changes in yields and income.

Comparing impact

Another question is how to compare the impact of different types of development intervention.

Recently, I reviewed with colleagues the available literature on the effectiveness of agricultural innovation grant funds, which involved having to make sense of a diverse range of interventions. [1]

Twelve studies had impact evaluation as their core objective. Of these, six were considered 'strong' because they compared groups of farmers that participated in the projects with groups that did not.
These stronger studies were less positive about the grant fund's impact than those studies that only reported the changes in the beneficiaries.

They were, however, all related to interventions that included promoting fertiliser use or supplying improved seeds — and, thus, could attribute yield and income effects to the increased use of these inputs.

A whole range of other innovation support programmes could not be included in the review because they had no systematic studies on impact. Their activities were indisputably relevant: support for experimentation, for example, or participatory technology development. However, their effectiveness and cost efficiency could not be assessed.

To facilitate comparison and increase their policy relevance, impact evaluations will need to identify common indicators that can be directly attributed to each of the innovation support activities.

Our review suggests some outcome areas for which common measurements are possible. They include knowledge of good agricultural practices, the adoption of new techniques, farmers' capacities to learn and adapt, and the capabilities of farmer groups to engage in collective action. Such indicators should be more closely related to the innovation support activities compared with indicators like yields and income.

Policymakers and grant system designers could specify assessments of these indicators as a major requirement for monitoring and evaluation plans of these programmes, to create an incentive for projects to regularly measure this human and social capital.

Collecting adequate data, and using comparable methods over a wider geographical scale than the intervention areas of each specific development project, would give practitioners and policymakers the information they need to decide on the merits of differing approaches to farmer innovation.

And it would enable them to better target policies and interventions to facilitate innovation in smallholder agriculture.

Giel Ton is a senior researcher in the agricultural economics research institute LEI Wageningen UR, the Netherlands. He is involved in the impact evaluations of various challenge funds and certification initiatives, and can be contacted at giel.[email protected].