Measuring impact by design: A guide to methods for impact measurement

Curated on Posted on by Stefaan Verhulst

Privy Council Office (Canada): “…This document is intended to be both an accessible introduction to the topic, as well as a reference for those involved in the design, delivery, procurement or appraisal of impact measurement strategies for Impact Canada projects. Drawing on best practices, Measuring Impact by Design was written to guide its readers to think differently about measuring impact than we have traditionally done within the federal public service.

In its role leading Impact Canada as a whole-of-government effort, the IIU works with an ever-expanding network of partners to deliver a range of innovative, outcomes-based program approaches. We are aware that program spending is an investment that we are making on behalf of, and directly for Canadians, and we need to place a greater emphasis on understanding what differences these investments make in improving the lives of citizens. That means we need a better understanding of what works, for whom, and in what contexts; and we need a better understanding of what kinds of investments are likely to maximize the social, economic and environmental returns we seek.

“We are aware that program spending is an investment that we are making on behalf of, and directly for Canadians, and we need to place a greater emphasis on understanding what differences these investments make in improving the lives of citizens.”

Good impact measurement practices are fundamental to these understandings and it is incumbent upon us to be rigorous in our efforts. We recognize that we are still building our capacity in government deliver on these approaches. It is why we built flexibility within Impact Canada authorities to use grants and contributions to fund research organizations with expertise in the kinds of techniques outlined in this guide. We encourage our partner departments to consider taking up these flexibilities.

Measuring Impact by Design is one of a number of supports that the IIU provides to deliver on its commitment to improve measurement practices for Impact Canada. We look forward to continued collaboration with our partners in the delivery of these important outcomes-based approaches across the public sector….(More)”.

The Tricky Ethics of Using YouTube Videos for Academic Research

Curated on Posted on by Stefaan Verhulst

Jane C.Hu in P/S Magazine: “…But just because something is legal doesn’t mean it’s ethical. That doesn’t mean it’s necessarily unethical, either, but it’s worth asking questions about how and why researchers use social media posts, and whether those uses could be harmful. I was once a researcher who had to obtain human-subjects approval from a university institutional review board, and I know it can be a painstaking application process with long wait times. Collecting data from individuals takes a long time too. If you could just sub in YouTube videos in place of collecting your own data, that saves time, money, and effort. But that could be at the expense of the people whose data you’re scraping.

But, you might say, if people don’t want to be studied online, then they shouldn’t post anything. But most people don’t fully understand what “publicly available” really means or its ramifications. “You might know intellectually that technically anyone can see a tweet, but you still conceptualize your audience as being your 200 Twitter followers,” Fiesler says. In her research, she’s found that the majority of people she’s polled have no clue that researchers study public tweets.

Some may disagree that it’s researchers’ responsibility to work around social media users’ ignorance, but Fiesler and others are calling for their colleagues to be more mindful about any work that uses publicly available data. For instance, Ashley Patterson, an assistant professor of language and literacy at Penn State University, ultimately decided to use YouTube videos in her dissertation work on biracial individuals’ educational experiences. That’s a decision she arrived at after carefully considering her options each step of the way. “I had to set my own levels of ethical standards and hold myself to it, because I knew no one else would,” she says. One of Patterson’s first steps was to ask herself what YouTube videos would add to her work, and whether there were any other ways to collect her data. “It’s not a matter of whether it makes my life easier, or whether it’s ‘just data out there’ that would otherwise go to waste. The nature of my question and the response I was looking for made this an appropriate piece [of my work],” she says.

Researchers may also want to consider qualitative, hard-to-quantify contextual cues when weighing ethical decisions. What kind of data is being used? Fiesler points out that tweets about, say, a television show are way less personal than ones about a sensitive medical condition. Anonymized written materials, like Facebook posts, could be less invasive than using someone’s face and voice from a YouTube video. And the potential consequences of the research project are worth considering too. For instance, Fiesler and other critics have pointed out that researchers who used YouTube videos of people documenting their experience undergoing hormone replacement therapy to train an artificial intelligence to identify trans people could be putting their unwitting participants in danger. It’s not obvious how the results of Speech2Face will be used, and, when asked for comment, the paper’s researchers said they’d prefer to quote from their paper, which pointed to a helpful purpose: providing a “representative face” based on the speaker’s voice on a phone call. But one can also imagine dangerous applications, like doxing anonymous YouTubers.

One way to get ahead of this, perhaps, is to take steps to explicitly inform participants their data is being used. Fiesler says that, when her team asked people how they’d feel after learning their tweets had been used for research, “not everyone was necessarily super upset, but most people were surprised.” They also seemed curious; 85 percent of participants said that, if their tweet were included in research, they’d want to read the resulting paper. “In human-subjects research, the ethical standard is informed consent, but inform and consent can be pulled apart; you could potentially inform people without getting their consent,” Fiesler suggests….(More)”.

How to use data for good — 5 priorities and a roadmap

Curated on Posted on by Stefaan Verhulst

Stefaan Verhulst at apolitical: “…While the overarching message emerging from these case studies was promising, several barriers were identified that if not addressed systematically could undermine the potential of data science to address critical public needs and limit the opportunity to scale the practice more broadly.

Below we summarise the five priorities that emerged through the workshop for the field moving forward.

1. Become People-Centric

Much of the data currently used for drawing insights involve or are generated by people.

These insights have the potential to impact people’s lives in many positive and negative ways. Yet, the people and the communities represented in this data are largely absent when practitioners design and develop data for social good initiatives.

To ensure data is a force for positive social transformation (i.e., they address real people’s needs and impact lives in a beneficiary way), we need to experiment with new ways to engage people at the design, implementation, and review stage of data initiatives beyond simply asking for their consent.

(Photo credit: Image from the people-led innovation report)

As we explain in our People-Led Innovation methodology, different segments of people can play multiple roles ranging from co-creation to commenting, reviewing and providing additional datasets.

The key is to ensure their needs are front and center, and that data science for social good initiatives seek to address questions related to real problems that matter to society-at-large (a key concern that led The GovLab to instigate 100 Questions Initiative).

2. Establish Data About the Use of Data (for Social Good)

Many data for social good initiatives remain fledgling.

As currently designed, the field often struggles with translating sound data projects into positive change. As a result, many potential stakeholders—private sector and government “owners” of data as well as public beneficiaries—remain unsure about the value of using data for social good, especially against the background of high risks and transactions costs.

The field needs to overcome such limitations if data insights and its benefits are to spread. For that, we need hard evidence about data’s positive impact. Ironically, the field is held back by an absence of good data on the use of data—a lack of reliable empirical evidence that could guide new initiatives.

The field needs to prioritise developing a far more solid evidence base and “business case” to move data for social good from a good idea to reality.

3. Develop End-to-End Data Initiatives

Too often, data for social good focus on the “data-to-knowledge” pipeline without focusing on how to move “knowledge into action.”

As such, the impact remains limited and many efforts never reach an audience that can actually act upon the insights generated. Without becoming more sophisticated in our efforts to provide end-to-end projects and taking “data from knowledge to action,” the positive impact of data will be limited….

4. Invest in Common Trust and Data Steward Mechanisms 

For data for social good initiatives (including data collaboratives) to flourish and scale, there must be substantial trust between all parties involved; and amongst the public-at-large.

Establishing such a platform of trust requires each actor to invest in developing essential trust mechanisms such as data governance structures, contracts, and dispute resolution methods. Today, designing and establishing these mechanisms take tremendous time, energy, and expertise. These high transaction costs result from the lack of common templates and the need to each time design governance structures from scratch…

5. Build Bridges Across Cultures

As C.P. Snow famously described in his lecture on “Two Cultures and the Scientific Revolution,” we must bridge the “two cultures” of science and humanism if we are to solve the world’s problems….

To implement these five priorities we will need experimentation at the operational but also institutional level. This involves the establishment of “data stewards” within organisations that can accelerate data for social good initiative in a responsible manner integrating the five priorities above….(More)”

We should extend EU bank data sharing to all sectors

Curated on Posted on by Stefaan Verhulst

Carlos Torres Vila in the Financial Times: “Data is now driving the global economy — just look at the list of the world’s most valuable companies. They collect and exploit the information that users generate through billions of online interactions taking place every day. 


But companies are hoarding data too, preventing others, including the users to whom the data relates, from accessing and using it. This is true of traditional groups such as banks, telcos and utilities, as well as the large digital enterprises that rely on “proprietary” data. 
Global and national regulators must address this problem by forcing companies to give users an easy way to share their own data, if they so choose. This is the logical consequence of personal data belonging to users. There is also the potential for enormous socio-economic benefits if we can create consent-based free data flows. 
We need data-sharing across companies in all sectors in a real time, standardised way — not at a speed and in a format dictated by the companies that stockpile user data. These new rules should apply to all electronic data generated by users, whether provided directly or observed during their online interactions with any provider, across geographic borders and in any sector. This could include everything from geolocation history and electricity consumption to recent web searches, pension information or even most recently played songs. 

This won’t be easy to achieve in practice, but the good news is that we already have a framework that could be the model for a broader solution. The UK’s Open Banking system provides a tantalising glimpse of what may be possible. In Europe, the regulation known as the Payment Services Directive 2 allows banking customers to share data about their transactions with multiple providers via secure, structured IT interfaces. We are already seeing this unlock new business models and drive competition in digital financial services. But these rules do not go far enough — they only apply to payments history, and that isn’t enough to push forward a data-driven economic revolution across other sectors of the economy. 

We need a global framework with common rules across regions and sectors. This has already happened in financial services: after the 2008 financial crisis, the G20 strengthened global banking standards and created the Financial Stability Board. The rules, while not perfect, have delivered uniformity which has strengthened the system. 

We need a similar global push for common rules on the use of data. While it will be difficult to achieve consensus on data, and undoubtedly more difficult still to implement and enforce it, I believe that now is the time to decide what we want. The involvement of the G20 in setting up global standards will be essential to realising the potential that data has to deliver a better world for all of us. There will be complaints about the cost of implementation. I know first hand how expensive it can be to simultaneously open up and protect sensitive core systems. 

The alternative is siloed data that holds back innovation. There will also be justified concerns that easier data sharing could lead to new user risks. Security must be a non-negotiable principle in designing intercompany interfaces and protecting access to sensitive data. But Open Banking shows that these challenges are resolvable. …(More)”.

France Bans Judge Analytics, 5 Years In Prison For Rule Breakers

Curated on Posted on by Stefaan Verhulst

Artificial Lawyer: “In a startling intervention that seeks to limit the emerging litigation analytics and prediction sector, the French Government has banned the publication of statistical information about judges’ decisions – with a five year prison sentence set as the maximum punishment for anyone who breaks the new law.

Owners of legal tech companies focused on litigation analytics are the most likely to suffer from this new measure.

The new law, encoded in Article 33 of the Justice Reform Act, is aimed at preventing anyone – but especially legal tech companies focused on litigation prediction and analytics – from publicly revealing the pattern of judges’ behaviour in relation to court decisions.

A key passage of the new law states:

‘The identity data of magistrates and members of the judiciary cannot be reused with the purpose or effect of evaluating, analysing, comparing or predicting their actual or alleged professional practices.’ *

As far as Artificial Lawyer understands, this is the very first example of such a ban anywhere in the world.

Insiders in France told Artificial Lawyer that the new law is a direct result of an earlier effort to make all case law easily accessible to the general public, which was seen at the time as improving access to justice and a big step forward for transparency in the justice sector.

However, judges in France had not reckoned on NLP and machine learning companies taking the public data and using it to model how certain judges behave in relation to particular types of legal matter or argument, or how they compare to other judges.

In short, they didn’t like how the pattern of their decisions – now relatively easy to model – were potentially open for all to see.

Unlike in the US and the UK, where judges appear to have accepted the fait accompli of legal AI companies analysing their decisions in extreme detail and then creating models as to how they may behave in the future, French judges have decided to stamp it out….(More)”.

Can we nudge farmers into saving water? Evidence from a randomised experiment

Curated on Posted on by Stefaan Verhulst

Paper by Sylvain Chabé-Ferret, Philippe Le Coent, Arnaud Reynaud, Julie Subervie and Daniel Lepercq: “We test whether social comparison nudges can promote water-saving behaviour among farmers as a complement to traditional CAP measures. We conducted a randomised controlled trial among 200 farmers equipped with irrigation smart meters in South-West France. Treated farmers received weekly information on individual and group water consumption over four months. Our results rule out medium to large effect-sizes of the nudge. Moreover, they suggest that the nudge was effective at reducing the consumption of those who irrigate the most, although it appears to have reduced the proportion of those who do not consume water at all….(More)”.

Journalism Initiative Crowdsources Feedback on Failed Foreign Aid Projects

Curated on Posted on by Stefaan Verhulst

Abigail Higgins at SSIR: “It isn’t unusual that a girl raped in northeastern Kenya would be ignored by law enforcement. But for Mary, whose name has been changed to protect her identity, it should have been different—NGOs had established a hotline to report sexual violence just a few years earlier to help girls like her get justice. Even though the hotline was backed by major aid institutions like Mercy Corps and the British government, calls to it regularly went unanswered.

“That was the story that really affected me. It touched me in terms of how aid failures could impact someone,” says Anthony Langat, a Nairobi-based reporter who investigated the hotline as part of a citizen journalism initiative called What Went Wrong? that examines failed foreign aid projects.

Over six months in 2018, What Went Wrong? collected 142 reports of failed aid projects in Kenya, each submitted over the phone or via social media by the very people the project was supposed to benefit. It’s a move intended to help upend the way foreign aid is disbursed and debated. Although aid organizations spend significant time evaluating whether or not aid works, beneficiaries are often excluded from that process.

“There’s a serious power imbalance,” says Peter DiCampo, the photojournalist behind the initiative. “The people receiving foreign aid generally do not have much say. They don’t get to choose which intervention they want, which one would feel most beneficial for them. Our goal is to help these conversations happen … to put power into the hands of the people receiving foreign aid.”

What Went Wrong? documented eight failed projects in an investigative series published by Devex in March. In Kibera, one of Kenya’s largest slums, public restrooms meant to improve sanitation failed to connect to water and sewage infrastructure and were later repurposed as churches. In another story, the World Bank and local thugs struggled for control over the slum’s electrical grid….(More)”

Here’s a prediction: In the future, predictions will only get worse

Curated on Posted on by Stefaan Verhulst

Allison Schrager at Quartz: “Forecasts rely on data from the past, and while we now have better data than ever—and better techniques and technology with which to measure them—when it comes to forecasting, in many ways, data has never been more useless. And as data become more integral to our lives and the technology we rely upon, we must take a harder look at the past before we assume it tells us anything about the future.

To some extent, the weaknesses of data has always existed. Data are, by definition, information about what has happened in the past. Because populations and technology are constantly changing, they alter how we respond to incentives, policy, opportunities available to us, and even social cues. This undermines the accuracy of everything we try to forecast: elections, financial markets, even how long it will take to get to the airport.

But there is reason to believe we are experiencing more change than before. The economy is undergoing a major structural change by becoming more globally integrated, which increases some risks while reducing others, while technology has changed how we transact and communicate. I’ve written before how it’s now impossible for the movie industry to forecast hit films. Review-aggregation site Rotten Tomatoes undermines traditional marketing plans and the rise of the Chinese market means film makers must account for different tastes. Meanwhile streaming has changed how movies are consumed and who watches them. All these changes mean data from 10, or even five, years ago tell producers almost nothing about movie-going today.

We are in the age of big data that offers to promise of more accurate predictions. But it seems in some of the most critical aspects of our lives, data has never been more wrong….(More)”.

Public Value: How can it be measured, managed and grown?

Curated on Posted on by Stefaan Verhulst

Geoff Mulgan et al at Nesta: “It builds on work Nesta has done in many fields – from health and culture to public services – to find more rounded and realistic ways of capturing the many dimensions of value created by public action. It is relevant to our work influencing governments and charities as well as to our own work as a funder, since our status as a charity commits us to creating public benefit.

Our aim in this work is to make value more transparent and more open to interrogation, whether that concerns libraries, bicycle lanes, museums, primary health services or training programmes for the unemployed. We recognise that value may come from government action; it can also be created by others, in civil society and business. And we recognise that value can often be complex, whether in terms of who benefits, or how it relates to values, as well as more technical issues such as what discount rates to apply.

But unless value is attended to explicitly, we risk ending up with unhappy results….(More)”.

Democracy (Re)Imagined

Curated on Posted on by Stefaan Verhulst

Chapter by Oldrich Bubak and Henry Jacek in Trivialization and Public Opinion: “Democracy (Re)Imagined begins with a brief review of opinion surveys, which, over the recent decades, indicate steady increases in the levels of mistrust of the media, skepticism of the government’s effectiveness, and the public’s antipathy toward politics. It thus continues to explore the realities and the logic behind these perspectives. What can be done to institute good governance and renew the faith in the democratic system? It is becoming evident that rather than relying on the idea of more democracy, governance for the new age is smart, bringing in people where they are most capable and engaged. Here, the focus is primarily on the United States providing an extreme case in the evolution of democratic systems and a rationale for revisiting the tenets of governance.

Earlier, we have identified some deep lapses in public discourse and alluded to a number of negative political and policy outcomes across the globe. It may thus not be a revelation that the past several decades have seen a disturbing trend apparent in the views and choices of people throughout the democratic world—a declining political confidence and trust in government. These have been observed in European nations, Canada as well as the United States, countries different in their political and social histories (Dalton 2017). Consider some numbers from a recent US poll, the 2016 Survey of American Political Culture. The survey found, for example, that 64% of the American public had little or no confidence in the federal government’s capacity to solve problems (up from 60% in 1996), while 56% believed “the government in Washington threatens the freedom of ordinary Americans.” About 88% of respondents thought “political events these days seem more like theater or entertainment than like something to be taken seriously” (up from 79% in 1996). As well, 75% of surveyed individuals thought that one cannot “believe much” the mainstream media content (Hunter and Bowman 2016). As in other countries, such numbers, consistent across polls, tell a story much different than responses collected half a century ago.

Some, unsurprised, argue citizens have always had a level of skepticism and mistrust toward their government but appreciated their regime legitimacy, a democratic capacity to exercise their will and choose a new government. However, other scholars are arriving at a more pessimistic conclusion: People have begun questioning the very foundations of their systems of government—the legitimacy of liberal democratic regimes. Foa and Mounk, for example, examined responses from three waves of cross-national surveys (1995–2014) focusing on indicators of regime legitimacy: “citizens’ express support for the system as a whole; the degree to which they support key institutions of liberal democracy, such as civil rights; their willingness to advance their political causes within the existing political system; and their openness to authoritarian alternatives such as military rule” (2016, 6). They find citizens to be not only progressively critical of their government but also “cynical about the value of democracy as a political system, less hopeful that anything they do might influence public policy, and more willing to express support for authoritarian alternatives” (2016, 7). The authors point out that in 2011, 24% of those born in the 1980s thought democracy 1 was a “bad” system for the US, while 26% of the same cohort believed it is unimportant 2 for people to “choose their leaders in free elections.” Also in 2011, 32% of respondents of all ages reported a preference for a “strong leader” who need not “bother with parliament and elections” (up from 24% in 1995). As well, Foa and Mounk (2016) observe a decrease in interest and participation in conventional (including voting and political party membership) and non-conventional political activities (such as participation in protests or social movement).

These responses only beckon more questions, particularly as some scholars believe that “[t]he changing values and skills of Western publics encourage a new type of assertive or engaged citizen who is skeptical about political elites and the institutions of representative democracy” (Dalton 2017, 391). In this and the next chapter, we explore the realities and the logic behind these perspectives. Is the current system working as intended? What can be done to renew the faith in government and citizenship? What can we learn from how public comes to their opinions? We focus primarily on the developments in the United States, providing an extreme case in an evolution of a democratic system and a rationale for revisiting the tenets of governance. We will begin to discern the roots of many of the above stances and see that regaining effectiveness and legitimacy in modern governance demands more than just “more democracy.” Governance for the new age is smart, bringing in citizens where they are most capable and engaged. But change will demand a proper understanding of the underlying problems and a collective awareness of the solutions. And getting there requires us to cope with trivialization….(More)”