To help equality bodies improve their work and effectiveness, Equinet has recently commissioned a paper on processes and indicators for measuring the impact of equality bodies.
The paper is entitled Processes and indicators for measuring the impact of equality bodies, and was written by Niall Crowley - an independent expert on equality issues.
Its primary purpose is to:
Devise and recommend practical processes and indicators that equality bodies could use to measure the impact of their work at national level.
Support national equality bodies in the further development of systems of evaluation of the impact of their work.
More broadly, the paper aims to contribute to:
Supporting national equality bodies to improve their work and effectiveness.
Strengthening the capacity of equality bodies to effectively promote and communicate their potential and impact.
Creating a context of wider recognition for the potential and contribution of equality bodies.
The paper provides:
an overview of existing literature on evaluation of the work of national human rights institutions and equality bodies,
the results of a survey on the experience of equality bodies in terms of evaluating their work, and
actions that could be taken to measure the impact of equality bodies.
The digital version of the paper can be downloaded below, while some of its conclusions are included in the rest of this article. To order hardcopies of this paper click here.
There is a limited range of literature in this field. However, it is well developed in relation to the work of national human rights institutions. Nonetheless, there is limited evaluation carried out by national human rights institutions and by equality bodies.
There are issues with evaluation. Evaluation methodologies can exercise a pressure on what work the body prioritises and how it is pursued. The work of the body can be driven by what is measurable rather than by what is important. Key achievements such as just sustaining a voice for equality and non-discrimination or preventing invidious change can be rendered invisible.
There are pitfalls that need to be addressed. Causality between the work done by the body and the impact measured can be impossible to establish. Data deficits can preclude the measurement of some impacts. Many equality bodies engage in a broad range of interventions that could demand a complex evaluation methodology.
However, evaluation offers real benefits. It enables the body to assess what is working well. It provides a basis from which to set targets for the work of the body. It provides a means for the body to communicate its objectives and its achievements. It enhances the quality of the work done by the body and improves its internal processes.
National Human Rights Institutions are identified as having the potential to achieve a transformative effect in society. Equality bodies are identified as having the potential to achieve social change. It is important that evaluation is focused on the extent to which this potential is being realised.
A more detailed potential has been established for equality bodies that serves as a framework for developing an evaluation strategy and indicators. Equality bodies have the potential to achieve change in the situation of individuals who experience discrimination; the policies and practices of organisations that employ and/or provide services; the content of and process for policy and legislation; the engagement by a range of stakeholders in promoting equality and combating discrimination; and attitudes to equality, diversity, discrimination and rights held by the general public.
Evaluation is an integral part of a broader planning/evaluation cycle. A theory of change approach can enable the body to better identify the change it seeks and the steps that are required to achieve such change. A range of evaluation methods emerges from the literature review. These include stakeholder consultations, public surveys, media reviews, and fact-finding studies. Peer review processes also emerge as of value.
Many equality bodies do not evaluate their work (50% of those surveyed). Some evaluate specific projects (18% of those surveyed). A number have developed more substantive forms of evaluation that cover the full range of their work or specific areas or portfolios of their work and that focus on impact (32% of those surveyed. This latter group could reflect a growing trend. There is evidence of significant new thinking and creativity in the approaches these equality bodies are developing.
The planning/evaluation cycle developed by equality bodies is identified as a key driving force for this evaluation work. A strategic plan and performance indicators (both quantitative and qualitative) are key elements in the evaluation process. It is, therefore, significant that many equality bodies have not developed strategic plans. This makes evaluation difficult if not impossible.
Evaluation is seen to inform the choices made by equality bodies and to support learning within the equality body. It is seen to enhance the standing of the equality body and to empower their staff in their work.
A logical chain is usefully identified where evaluation explores outputs, then it explores change and finally it assesses the link between the two. Informed judgment is viewed as a key element in evaluation. Some equality bodies are deploying proxy indicators. Proxy indicators reflect that an impact can be claimed from an output, for example, on the basis of evidence that has already established that certain actions lead to particular outcomes.
Surveys, stakeholder interviews, self-assessment, and follow-up to cases emerge as valuable tools for evaluation. Simplicity in the approach to evaluation is encouraged. Claims for impact should be modest. Care is required with self-assessment lest it be subjective rather than evidence based. An external and an internal dimension is often involved in the evaluation done.
Different perspectives on assessing impact are apparent. Impact at an institutional level is clearly seen as important by most equality bodies. The impact of the voice and image of an equality body or on perceptions held about the equality body and its work are also identified as a useful focus. In one instance a framework of potential impacts is used - economic, social, regulatory and deliberative/aspirational.
A number of difficulties in implementing evaluation are identified. These include the issues of causality, data deficits and complexity (particularly for small organisations) already identified. They also include the challenge of managing the diversity of interests held by stakeholders and the lack of adequate human and financial resources in the equality body. There are cultural barriers, in particular where there is no culture of strategic planning or no culture of evidence based processes within the equality body.