An Ethnographic Approach to Impact Evaluation: Stop Measuring Outputs, Start Understanding Experiences

Panthea Lee, a principal at Reboot, in TechPresident: “Is open government working? I asked the question in a previous post …Too often, assessing the impact of open government initiatives amounts to measuring outputs: how many developers flocked to a civic tech hackathon; the amount of procurement records feeding corruption hawks and socially-minded graphic designers; or the number of tweets or media mentions about a particular initiative, regardless of whether they are from the same industry blogs and actors covering open government.
Quantitative metrics have their place. They may be useful for gauging the popularity of an initiative. They are almost always used to justify funding for an initiative. But, ultimately, these studies say very little about open government’s actual impact on people….We need to move beyond measuring outputs and toward understanding experiences….
Applied ethnography holds great potential for understanding how individuals experience open government initiatives. Ethnography––“a portrait of people”––is the study of people within their social and cultural contexts. It embraces context, examining how results can be explained by human factors and situational interactions. Ethnography allows us to understand the meaning of participation for different individuals––who is affected or not, and why. (Ethnographic research is often mistakenly equated with “interview studies” or other types of qualitative research. An immersive research approach, it uses techniques such as participant observation, unstructured interviews, and artifact collection to attempt a holistic analysis of human behaviours, interactions, and perceptions over time.)
Take, for example, this ethnographic study of a participatory budgeting initiative in Rome. The study found that through engagement with the participatory budgeting process, some participants “discovered a passion for politics,” leading them to join neighborhood associations and local political parties. Other participants, however, left the budgeting process feeling more cynical about and disengaged from participatory democracy.”