Executive Summary
Government accountability bodies in various OECD countries point to a dearth of reliable evidence on the impacts of SME and entrepreneurship policy. Either evaluations have not been undertaken or their methodologies have not been of high enough standard. However, SME and entrepreneurship policy is a youthful field where questions are raised about likely effectiveness, to do for example with low survivability of start-ups and low motivations for SME growth.
The response must be to develop systematic, high-quality evaluation in the area. This requires taking a number of steps – including establishing clear objectives for policies and programmes at the outset, measuring changes on a common set of core impact indicators alongside possible additional indicators to measure specifics, setting up control and treatment groups, and tracking survivors and non-survivors.
Reliable evaluation is certainly achievable in SME and entrepreneurship policy, and there is no reason not to develop it, particularly when taking into account recent improvements in data and analytical methods. For example, some governments now associate single numerical identifiers with individual entrepreneurs and small businesses, facilitating control group studies. More sophisticated statistical techniques such as propensity score matching have diffused, and Randomised Control Trial (RCT) studies are becoming common.
Evidence can also be drawn from those meta-evaluations and individual high-quality evaluations that do exist. Adopting an international lens to this offers the prospect of learning from a critical mass of evaluation evidence.
This Framework represents the main OECD guidance on SME and entrepreneurship policy evaluation. It highlights the foundations of systematic and reliable impact evaluation, and it demonstrates that reliable evaluation is achievable in the field by offering the examples of 50 reliable evaluations across different OECD countries and SME and entrepreneurship policy areas. It reviews the evidence of existing meta-evaluations in the field, and profiles the methodologies and findings of the 50 individual high-quality evaluation studies.
Overall, the report offers information on the key features of evaluation, the lack of reliable evaluation evidence, the mixed findings of reliable evaluations, and reasons why reliable evaluations are delivering mixed findings.
It examines what needs to be improved in evaluation practice, such as exploiting new techniques and data, specifying objectives, and assessing impacts against expenditures. It also explores what evaluation evidence is suggesting may be wrong with some aspects of SME and entrepreneurship policy, such as its targeting and delivery and the mix of policy interventions used.
The report also explores the role of evaluation for steering government response measures to major economic shocks, taking the example of recent government COVID-19 crisis response measures, and asks what could be done better on evaluation.
Improving evaluation practice
The major message of the report is that current weaknesses in the practice of SME and entrepreneurship policy evaluation need to be addressed. This can be achieved by making better use of existing data within government for evaluation purposes and adopting more sophisticated evaluation techniques using control groups. Work is also needed to specify the objectives and targets of policy in advance of evaluation, to benchmark programmes against others based on their evaluation results, and to evaluate the impacts of macro interventions such as changes in the tax and regulatory regime as well as expenditure programmes aimed directly at specific groups SMEs and entrepreneurs. Evaluation evidence must also frame future policy decisions.
The implications of mixed findings from evaluation evidence
Although the evidence shows that many of the policy interventions worked, a smaller number did not. In particular, 3 of 11 evaluations of purely “Soft” programmes (based on training, advice, mentoring etc.) reported “no/negative” outcomes, compared with only 2 out of 33 “Hard” programmes (with a significant financial aspect of the package). While the numbers are not sufficiently high to make any conclusion, this indicates the need for more comparative assessment of the impact of “Soft” and “Hard” policies. Furthermore, policy success may be affected by the diversity of SMEs and entrepreneurs. For example, if a programme is not selective of firms with growth and survival prospects, or selects poorly, its impact is likely to be weak.
Evaluation and COVID-19 policy support measures
Governments internationally introduced substantial temporary support for SMEs and entrepreneurs to weather the COVID-19 crisis from the early months of 2020. However, there were evaluation weaknesses which can be addressed in forthcoming evaluation work or in preparing interventions for future economic shocks. The key issues are lack of clear specification of policy objectives, lack of information on expenditures, and lack of generalised impact evaluations.
The report makes the following recommendations:
1. Governments should specify in advance the Objectives and Targets for each policy measure introduced.
2. Three core metrics – Sales, Employment and Survival – should be specified and assessed in all evaluations. These can be complemented with additional measures for other Objectives, where targeted, such as environmental and social benefits.
3. Expenditure data should be made available to evaluators for each policy measure to facilitate cost-effectiveness assessments.
4. Governments should establish a central monitoring and evaluation unit and a co-ordination process for the monitoring and evaluation of SME and entrepreneurship policy across government ministries and bodies.
5. Every three years, all major SME and entrepreneurship programmes should be the subject of a reliable evaluation, defined as a minimum of Step V, only the very “short-lifers” being excluded.
6. Governments should look carefully, using at least Step V methods, at the impact of their existing, and any new proposed, “Soft” programmes.
7. Governments should review the role played by “Macro” policies.
8. Evaluations should provide the evidence for making decisions on the scale and nature of selective support.
9. Evaluations should identify exceptional performers and the role such firms have in reaching a judgment on the overall effectiveness of a programme.
10. Evaluations should systematically include the performance of non-surviving SMEs and start-ups in their assessments of treatment and control group performance.
11. Governments should investigate the use of the data they collect for tax and other purposes with a view to making it more widely available to those conducting policy evaluations.
12. Lessons from reliable evaluations should be shared between countries, with the OECD CSMEE being an ideal vehicle for facilitating this exchange.
13. Internationally-co-ordinated policy evaluation should be undertaken on the impact of COVID-19 SME and entrepreneurship policy responses.