In all OECD countries, policies are increasingly being implemented with the intention of improving the environmental performance of agriculture. Agricultural policies themselves also have important environmental impacts. But to what extent are the agri-environmental policies being evaluated to determine whether they are achieving their objectives and doing so in efficient ways? The purpose of the Workshop on Evaluating Agri-environmental Policies was to review the efforts underway in OECD member governments to evaluate their agri-environmental policies – to both share experiences and to assist the OECD deepen its analysis. This publication closely follows the structure of the Workshop...

The three papers in Part I raise the major issues involved in evaluating agri-environmental policies, from both a government and academic perspective, and from a theoretical and practical application. The following comments are based on my experience from participating in the OECD Environmental Performance Reviews, both as a review team member and as a Finnish delegate to the OECD Working Party responsible for the reviews...

The increasing number and complexity of agri-environmental policies is an important reason for the OECD interest in evaluating such measures. The two principal criteria used are environmental effectiveness and economic efficiency, with the later including elements such as administration and compliance costs, dynamic and innovative effects, and changes in farmer attitudes. The OECD has only undertaken a limited number of specific evaluations, including the Permanent Cover Program (Canada), Landcare (Australia), the nutrient quota scheme (the Netherlands) and manure management regulations (various). These evaluations have focused on environmental effectiveness, relied largely on quantitative assessments, and generally use a simple “before and after” approach to establish the impact. Future work will focus on developing models and statistical methods to evaluate the cause and effect relationship between policies and environmental outcomes...

Policy evaluation is difficult because the words have become “plastic” and the dilemma of public sector accountability. Despite these problems, evaluations should be undertaken because they help build up the collective knowledge of what works and why. The establishment of a cause and effect logic, illustrated for example by Bennett’s Hierarchy, greatly assists the development of effective evaluations by determining specific outcomes or targets at various points in the chain of events. It also identifies the various “communities”, such as the target community (e.g. farmers), partners with an interest in the success of the programme (e.g. community groups) that need to be considered. The involvement of such communities is not only vital to the evaluation process, but crucially important to the successful implementation of the policy. Evaluations must be seen as part of a learning process that leads to the development of better agri-environmental policies...

Rational appraisal of agri-environmental policy requires a comparison of costs and benefits. Some fundamental features of such a comparison are explored. First, some of the indicators currently used to measure benefits or effectiveness are not proper measures of environmental effect. Indicators such as farmer attitudes and the level of participation in agri-environment schemes (AESs) are intermediate, not final, outputs. Benefits may or may not be measured in monetary terms. Where they are not so measured, the relevant methodologies are cost-effectiveness analysis (CEA) and multicriteria analysis (MCA). Where they are measured in money terms, the relevant evaluation procedure is cost-benefit analysis (CBA). However, CEA cannot answer questions about whether any agrienvironment scheme is worth pursuing: they can select only from alternative schemes given that one or more schemes must be executed. MCA can account for “do nothing” options but will diverge from CBA unless restrictive conditions are met: (a) scores on attributes must be the same, and (b) MCA weights must be the same as shadow prices. Any appraisal methodology must also account for time issues: (a) discounting, (b) selection of terminal period and post-policy scenarios, and (c) relative price effects. CBA offers a more comprehensive basis for comparison, but has uncertainties that primarily relate to the credibility of willingness to pay estimates of benefits. The use of expert rather than citizen indicator choice is discussed. The paper concludes that (a) CBA is to be preferred where credible benefit estimates can be secured, but that (b) CEA and MCA are second-best appraisal methodologies where CBA cannot, for one reason or another, be applied. Nonetheless, CEA and MCA carry with them considerable risks of inefficient decision-making...

The recent expansion of agri-environmental payment approaches in European countries provides a rich set of experiments to discover ways to improve programme performance. The mid-term evaluations of agri-environmental measures in the Belgian, French, Greek, and Italian Rural Development Plans offer early insights on improving their efficacy and cost effectiveness, and suggest ways to strengthen future evaluations. The authors are to be commended for contributing new knowledge to a critical agri-environmental policy arena in all OECD countries...

Agri-environmental measures (AEM) in Belgium are implemented through Rural Development Plans (RDP) established under the Rural Development Regulation of the Second Pillar of the Common Agricultural Policy (CAP) of the European Union (EU).2 There is a separate RDP for the country’s two regions: Flanders and Wallonia. Flanders was late in implementing AEM: Regulation 2078/92 was never implemented. With its RDP 2000-06, Flanders has tried to catch up with other EU countries which use AEM. Despite little experience, twelve AEM have been implemented. In 2003, a mid-term evaluation by an independent evaluator reviewed the implementation of the RDP over 2000-02.3 The evaluation was done in the framework of the EU common monitoring and evaluation approach. For this purpose, the European Commission (EC) had defined common evaluation questions, criteria and indicators to be used by RDP evaluators.4 Most of the evaluation approach went to fulfilling the EC demands, which aimed mostly at assessing the environmental effects of the AEM. The evaluator was assisted full-time by a staff member from the Division for Agricultural Policy Analysis. This set-up proved to be very successful. While the evaluation took place too early to measure results and impacts, some useful recommendations for RDP management were made. The importance of an integrated monitoring and evaluation framework was shown. A survey of farmers delivered useful information about what’s happening on the field.

While various agri-environmental measures have been available in France since the early 1990s, a single scheme based on regional farming contracts (the contrat territorial d’exploitation or CTE) was established in 2000 as part of the Rural Development Plan 2000-06.2 This was a major shift to broaden the scope of applicability to cover the whole country. As required, a mid-term evaluation of the RDP was conducted in 2003. Within this, the evaluation of the CTE was split into 22 contracts (one national and 21 regional) which were awarded to eleven evaluation consultancies, and cost over EUR 1 million. At this stage, the main findings indicate that the measures appear to be working adequately with regard to maintaining biodiversity and the quality of landscape in grazing and mixed crop/livestock zones but fairly poorly in terms of mitigating the negative impacts of agriculture, such as water pollution. As a follow-up to the evaluation, working groups were established to consider changes to the measures to improve their performance.

Agri-environmental measures have been developed in Italy by the 21 regional authorities.2 These schemes offer farmers voluntary, multi-annual contracts where they are paid for delivering environmental goods and services which go beyond the “reference level” of good agricultural. Three measures account for 89% of total expenditure: integrated production, organic production, and active management of pastures and meadows. This paper analyses the evolution of agri-environmental measure evaluation since they began in 1994, considering the socio-economic, agricultural and environmental impacts. The paper concludes that the crucial point, whatever the methodology used to evaluate the socio-economic, agricultural or environmental impacts, is the availability of reference data and a local level monitoring system. The setting up of a detailed geographic information system is a precondition for the impact evaluation of any environmental policy. The Italian experience demonstrates that an indirect analysis of the environmental impact based on contextual indicators, administrative data and scientific data coming from literature or specific research, can provide good results at relative lower cost. The survey also shows that good evaluation results can be reached by combining different methods and different criteria (e.g. economic, social and environmental). Considering the costs and the analytical skills required the combination of state and pressure indicators seems to be a good alternative to the estimation of impact indicators in the case of large scale programmes.

The papers included in Part III cover a range of agri-environmental measures that reward farmers for the supply of public benefits (e.g. environment, biodiversity values and nature management). Measures include: 1) the adoption of a new Environmental Stewardship Scheme to be put in the broader context of agri-environmental programmes in England; 2) buffer strips as a very specific and targeted agri-environmental measure in Finland; 3) a comparison of agri-environmental measures relative to other agricultural policy programmes that could contribute to meeting environmental targets in Germany, Sweden and Switzerland; and 4) a specific programme (the CRP) in the United States to control soil erosion and conserve wildlife...

This paper starts with a short summary of the development of agri-environment schemes2 in England and then describes a two-year review intended to evaluate the existing schemes and to design a replacement for them. Evaluation of the existing schemes was carried out using a combination of data from scheme monitoring, externally commissioned reviews, other published and unpublished surveys and a major public consultation exercise. The design of the new scheme, Environmental Stewardship, built on the results of this review and was itself subject to two rounds of public consultation intended to ensure that the design was robust and widely supported. The design provides for a two-tier scheme, the Entry Level of which is intended to be a simple scheme, open to all farmers and offering a fixed payment per hectare in return for a package of management measures chosen by the farmer from a standard menu of options. Because this type of scheme is new to England, a live pilot was run to evaluate the design. Success criteria were agreed in advance covering uptake, farmer reactions, and likely environmental outcomes. The performance of the pilot against these criteria was carefully monitored during the first six months of the pilot. It was concluded that the pilot had met all its success criteria. The design of Environmental Stewardship is now complete and has been approved by the European Commission (EC). The design of the scheme incorporates features intended to make it easier to measure environmental outcomes in future and these are briefly described, along with the overall strategy for evaluating the performance of the scheme. It is stressed that in future it will be important to shorten the cycle time between monitoring, evaluation and changes to schemes.

This paper reviews alternative evaluation techniques used to assess environmental and economic performance of Finnish buffer strip policy. Narrow (3-metre) buffer strips are one of the basic mandatory measures in the Finnish agri-environmental programme. Their major objective is to improve surface water quality through reducing surface run-offs of sediment and sediment bound pollutants. Wide (15-metre) buffer strips are one of the environmentally more effective special agrienvironmental measures. All evaluation techniques reviewed in the paper (experiments, field survey data, expert surveys and economic analysis) showed that buffer strips are environmentally effective agri-environmental measures. Economic analysis revealed, however, that the establishment of wide buffer strips is not profitable for a farmer at current compensation level. Economic analysis also showed that uniform narrow buffer strips performed well under heterogeneous conditions whereas uniform fertiliser restrictions produced significant welfare losses.

This contribution describes the development of agri-environmental measures in Germany and presents an evaluation approach for the assessment of long-term impacts.2 Farm accounts of 18 600 farms over 13 years (1989-2002) have been used to select samples of farms participating in agri-environmental schemes and similar farms not participating in these schemes, using a cluster analysis. For each farm with high agri-environmental payments, five similar farms receiving no or low levels of environmental payments (“non-participants”) were selected. The results show that farms participating in agri-environmental schemes have reduced their land use intensity and production per hectare, compared to non-participants, with organic farms showing notably higher rates of extensification. Although considerable income effects can be observed, there is no clear “windfall profit” situation, because participants significantly improve their environmental performance. For such management changes, appropriate incentives are needed in order to compensate for risks involved. Analysis of farm accounts can provide valuable insights into long-term farm developments. However, certain impacts of farming, e.g. in the area of erosion and biodiversity, are not “visible” in the accounts. Therefore, the presented approach is only one element in a methodology mix to be applied when evaluating agri-environmental schemes.

Several of the Swedish environmental objectives comprise positive and negative environmental effects of agriculture. One of the most important positive effects is the management and preservation of semi-natural pastures. Those lands provide a considerable share of the biodiversity and cultural values of Sweden’s agricultural landscape. An important negative effect is nitrogen leaching and ammonia emissions regarding both land and water. Simulations using an economic model of Swedish farming show that most forms of support related to grazing animals or to grassland have a positive effect on the pasture area. At the same time, those measures contribute to negative environmental load in the form of nitrogen leaching and ammonia emission. The relationship between positive and negative effects is very different for different forms of support. Simulations indicate a significantly higher environmental efficiency with more targeted measures. Different environmental objectives require different policy measures to deal with the objective-specific problem. The targeted measure that compensates for using a particular cropping technique in areas sensitive to leaching has proved to be cost-efficient. Good models are essential to evaluation. When economic models fail to predict the observed development, additional evaluation methods must be used. This can be the case when developments in agriculture are affected by social factors, in addition to agro-economical ones.

In 1994, Switzerland launched an extensive evaluation of agri-environmental measures that will be completed in 2005. Over time, various scientific studies commenced to analyse the effects of these measures on biodiversity; nitrogen, phosphorus and pesticide use; and the farm economic situation. A system of agri-environmental indicators is being established. Every four years, Switzerland decides on important modifications in agricultural policy. The various projects have to be finished and exploited in such a way that they can be used in the next round of agricultural policy modification.

The two papers in Part IV provide interesting insights into the implementation of taxes as part of an agri-environmental policy portfolio in Denmark and Norway. The research presented is a mix of policy simulation for nitrogen and phosphorus control in Denmark, and the reporting of actual responses by farmers to the introduction of a tax on pesticides in Denmark and Norway...

Part V contains two excellent papers that evaluate agri-environmental regulatory policies. The papers take mainly a technical standpoint, focusing on the measurement of the principal environmental issue that was the objective of policy makers, i.e. soil erosion in the US and nitrogen loss in Denmark. Both papers show some attention to the dynamics of policy impact, particularly the Danish paper that discusses the trend in nitrogen reduction. The two papers emphasise the good results of the policies under evaluation, even with delays and the presence of climate effects that somehow disturb the clear interpretation of the environmental changes detected. They also emphasise the importance of simple indicators, regularly used over time, with possibilities for benchmarking and comparisons (such as nitrogen balances) and the use of a mix of information sources (primary ambient data, input statistics, etc.). The papers focus on one or a few indicators very much confined to the main target of the policy...

Since 1985, a number of Action Plans have been implemented in Denmark to reduce nitrate leaching from agriculture. The regulatory measures applied include area-related measures (e.g. wetlands and afforestation) and nutrient-related measures (e.g. mandatory fertiliser plans and improved utilisation of nitrogen in manure). These have been implemented alongside research programmes and dialogue between authorities and the agricultural community. To monitor the development in nitrogen losses from the agricultural system three national indicators are used: nitrogen surplus, nitrogen efficiency and nitrate leaching. Over the period 1979-2002, the nitrogen surplus has fallen by over one-third, nitrogen efficiency has increased from 27% to 36%, and model calculations estimate a 48% reduction in nitrate leaching. Environmental monitoring programmes show a decrease in the nitrogen concentration of water leaving the root zone in both rivers and in coastal waters. The Danish approach to regulating nutrient losses from agriculture has proven successful but with a delay in achieving the environmental objectives. To date the regulation has been performed on a national scale. A more regional or local approach is believed to be necessary in future.

Conservation compliance, enacted as part of the 1985 Food Security Act, required farmers to apply approved conservation systems on highly erodible (HEL) cropland by 1995 or risk loss of all federal agricultural programme payments, including income support. Between 1982 and 1997, the annual rate of cropland soil erosion dropped by nearly 40%. What portion of this drop in cropland soil erosion can be attributed to conservation compliance? We analyse this question using existing data and a new method of linking soil erosion data from the National Resources Inventory (NRI) with data on farm programme participation and payments from the Agricultural Resources Management Survey (ARMS).

All three papers in Part VI discuss, and in some cases value, the benefits that result from agrienvironmental policies (AEP). For instance, the social environmental benefits of the Canadian Shelterbelt Program exceed the cost, and in New Zealand, the potential economic gains to farmers resulting from the Sustainable Farming Fund exceed the funds provided. But we have learned that there are uncertainties with regard to the appropriate economic evaluation tools...

The Natural Heritage Trust and the National Action Plan for Salinity and Water Quality are Australia’s principal programmes aimed at protecting and restoring the environment and furthering the sustainability of agriculture and other uses of natural resources. The majority of programme investments are made through fifty-seven regional community based organisations covering the entire continent. Each regional body is required to prepare a natural resources management plan for joint accreditation by governments and programme investments are made in response to the targets and priorities identified in those plans. Governments have established nationally agreed frameworks for resource condition targets, for monitoring and evaluation, and for indicators and protocols. The practicality of these policies for assessing the impact of programme investments and improving the broad understanding of landscape condition and trends has been reviewed through regional trials conducted by the National Land and Water Resources Audit focussing particularly on data availability and regional community capacity. Co-operative effort is underway within regional, State and national partnerships to build an integrated and enduring monitoring and evaluation system capable of meeting needs at all these levels.

This paper describes two evaluations that have been completed on the Sustainable Farming Fund (SFF).2 The particular shape of the SFF requires an evaluation methodology that takes into account the complex nature of the supported projects. An adaptation of the Bennett’s Hierarchy was used. Experience with these two evaluations shows that to be effective, the methodology will need to continue to adapt as the SFF proceeds through time – the methodology used in the early-stage evaluation was further adapted in Evaluation 2004. Evaluation is then a work in progress...

There are a number of possible interpretations of what is a “policy mix”. In a broad sense a policy mix can be a particular combination of policies, objectives, and instruments to achieve these objectives. Thus, examples of policy mixes include the following...

This paper provides an overview of the evaluation methodology which is applied to all agriculture, forestry and fisheries policies in Japan, with specific reference to agri-environmental policies. The Ministry of Agriculture, Forestry and Fisheries (MAFF) establishes policy evaluation target indicators and their values for each policy area. For example, for the policy area “environment protection measures in crop production”, the two target indicators “maintenance of organic matter use” and “reduction of chemical fertiliser use” have been adopted. The target indicators cover the outcome of various policy measures including regulatory measures, economic measures, information provision and voluntary measures. MAFF has also a policy evaluation framework for each policy measure.

In 2003/04 five instruments of Dutch manure and fertiliser policy were evaluated: the Minerals Accounting System (MINAS, the core of Dutch manure policy), the Manure Transfer Contracts system, the Production Quota system, the Buying Up Scheme and the Nitrate Projects Action Scheme. A great number of data sources and research and evaluation tools were used. The evaluation showed that MINAS was for important sectors within agriculture both an effective and efficient instrument. The system of Manure Transfer Contracts did not add to the effect of the Production Quota system and therefore was not efficient. MINAS has contributed to a reduction in the use of chemical nitrogen fertilisers by 25% and phosphate fertilisers by 10-20%. Since 1998 nitrogen surpluses in dairy farming have steadily reduced by 15-30 kg ha–1 yr–1 and present surpluses average about 150 kg ha–1 yr-1. Nitrate concentrations in upper groundwater have decreased substantially since 1990, but the target value of 50 mg L-1 is exceeded on at least 60% of all farms on sandy soils in the Netherlands.