Topics: big data

Report by Andrew Weston, Anne Kazimirski, Anoushka Kenley, Rosie McLeod, Ruth Gripper: “Measurement and evaluation is core to good impact practice. It helps us understand what works, how it works and how we can achieve more. Good measurement and evaluation involves reflective, creative, and proportionate approaches. It makes the most of existing theoretical frameworks as well as new digital solutions, and focuses on learning and improving. We researched the latest changes in theory and practice based on both new and older, renascent ideas. We spoke to leading evaluation experts from around the world, to ask what’s exciting them, what people are talking about and what is most likely to make a long lasting contribution to evaluation. And we found that new thinking, techniques, and technology are influencing and improving practice.

Technology is enabling us to gather different types of data on bigger scales, helping us gain insights or spot patterns we could not see before. Advances in systems to capture, manage and share sensitive data are helping organisations that want to work collaboratively, while moves towards open data are providing better access to data that can be linked together to generate even greater insight. Traditional models of evaluating a project once it has finished are being overtaken by methods that feed more dynamically into service design. We are learning from the private sector, where real-time feedback shapes business decisions on an ongoing basis asking: ‘is it working?’ instead of ‘did it work?’.

And approaches that focus on assessing not just if something works but how and why, for whom, and under what conditions are also generating more insight into the effectiveness of programmes. Technology may be driving many of the innovations we highlight here, but some of the most exciting developments are happening because of changes in the ideologies and cultures that inform our approach to solving big problems. This is resulting in an increased focus on listening to and involving users, and on achieving change at a systemic level—with technology simply facilitating these changes.

Some of the pressures that compel measurement and evaluation activity remain misguided. For example, there can be too big a focus on obtaining a cost-benefit ratio—regardless of the quality of the data it is based on—and not enough encouragement from funders for charities to learn from their evaluation activity. Even the positive developments have their pitfalls: new technologies pose new data protection risks, ethical hazards, and the possibility of exclusion if participation requires high levels of technical ability. It is important that, as the field develops and capabilities increase, we remain focused on achieving best practice.

This report highlights the developments that we think have the greatest potential to improve evaluation and programme design, and the careful collection and use of data. We want to celebrate what is possible, and encourage wider application of these ideas. Choosing the innovations In deciding which trends to include in this report, we considered how different approaches contributed to better evaluation by:

• overcoming previous barriers to good evaluation practice, eg, through new technologies or skills;
• providing more meaningful or robust data;
• using data to support decision-making, learning and improving practice;
• increasing equality between users, service deliverers and funders; and
• offering new contexts for collaboration that improve the utility of data.

… Eight key trends emerged from our research that we thought to be most exciting, relevant and likely to have a long-lasting contribution. Some of these are driven by cutting-edge technology; others reflect growing application of ideas that push practice beyond ‘traditional’ models of evaluation. User-centric and shared approaches are leading to better informed measurement and evaluation design. Theory-based evaluation and impact management embolden us to ask better research questions and obtain more useful answers. Data linkage, the availability of big data, and the possibilities arising from remote sensing are increasing the number of questions we can answer. And data visualisation opens up doors to better understanding and communication of this data. Here we present each of these eight innovations and showcase examples of how organisations are using them to better understand and improve their work….(More)”