Emily Bazalgette at FutureGov: “…Impact evaluation involves using a range of research methodologies to investigate whether our products and services are having an impact on users’ lives. ….Rigorous academic impact evaluation wasn’t really designed for rapidly iterating products made by a fast-moving digital and design company like FutureGov. Our products can change significantly over short periods of time — for instance, in a single workshop Doc Ready evolved from a feature-rich social media platform to a stripped-down checklist builder — and that can create a tension between our agile process and traditional evaluation methodologies, which tend to require a fixed product to support a long-term evaluation plan.
We’ve decided to embrace this tension by using Theories of Change, a useful evaluation tool recommended to us by our investors and partners Nesta Impact Investments. To give you a flavour (excuse the pun), below we have Casserole Club’s Theory of Change.
The problem we’re trying to solve (reducing social isolation) doesn’t tend to change, but the way we solve it might (the inputs and short to medium-term outcomes). In future, we may find that we need to adapt to serve new user groups, or operate in different channels, or that there are mediating outcomes for social isolation that Casserole Club produces other than social contact with a Casserole Club cook. Theories of Change allow us to stay focused on big-picture outcomes, while being flexible about how the product delivers on these outcomes.
Another lesson is to make evaluation everyone’s business. Like many young-ish companies, FutureGov is not at the stage where we have the resources to support a full-time, dedicated Head of Impact. But we’ve found that you can get pretty far if you’ve got a flat structure and lots of passionate people (both of which, luckily, we have). Our lack of hierarchy means that anyone can take up a project and run with it, and collaboration across the company is encouraged. Product impact evaluation is owned by the product teams who manage the product over time. This means we can get more done, that research design benefits from the deep knowledge of our product teams, and that evaluation skills (like how to design a decent survey or depth interview) have started to spread across the organisation….(More)”