Global innovations in measurement and evaluation


Report by Andrew WestonAnne KazimirskiAnoushka KenleyRosie McLeodRuth Gripper: “Measurement and evaluation is core to good impact practice. It helps us understand what works, how it works and how we can achieve more. Good measurement and evaluation involves reflective, creative, and proportionate approaches. It makes the most of existing theoretical frameworks as well as new digital solutions, and focuses on learning and improving. We researched the latest changes in theory and practice based on both new and older, renascent ideas. We spoke to leading evaluation experts from around the world, to ask what’s exciting them, what people are talking about and what is most likely to make a long lasting contribution to evaluation. And we found that new thinking, techniques, and technology are influencing and improving practice.

Technology is enabling us to gather different types of data on bigger scales, helping us gain insights or spot patterns we could not see before. Advances in systems to capture, manage and share sensitive data are helping organisations that want to work collaboratively, while moves towards open data are providing better access to data that can be linked together to generate even greater insight. Traditional models of evaluating a project once it has finished are being overtaken by methods that feed more dynamically into service design. We are learning from the private sector, where real-time feedback shapes business decisions on an ongoing basis asking: ‘is it working?’ instead of ‘did it work?’.

And approaches that focus on assessing not just if something works but how and why, for whom, and under what conditions are also generating more insight into the effectiveness of programmes. Technology may be driving many of the innovations we highlight here, but some of the most exciting developments are happening because of changes in the ideologies and cultures that inform our approach to solving big problems. This is resulting in an increased focus on listening to and involving users, and on achieving change at a systemic level—with technology simply facilitating these changes.

Some of the pressures that compel measurement and evaluation activity remain misguided. For example, there can be too big a focus on obtaining a cost-benefit ratio—regardless of the quality of the data it is based on—and not enough encouragement from funders for charities to learn from their evaluation activity. Even the positive developments have their pitfalls: new technologies pose new data protection risks, ethical hazards, and the possibility of exclusion if participation requires high levels of technical ability. It is important that, as the field develops and capabilities increase, we remain focused on achieving best practice.

This report highlights the developments that we think have the greatest potential to improve evaluation and programme design, and the careful collection and use of data. We want to celebrate what is possible, and encourage wider application of these ideas. Choosing the innovations In deciding which trends to include in this report, we considered how different approaches contributed to better evaluation by:

  • overcoming previous barriers to good evaluation practice, eg, through new technologies or skills;
  • providing more meaningful or robust data;
  • using data to support decision-making, learning and improving practice;
  • increasing equality between users, service deliverers and funders; and
  • offering new contexts for collaboration that improve the utility of data.

… Eight key trends emerged from our research that we thought to be most exciting, relevant and likely to have a long-lasting contribution. Some of these are driven by cutting-edge technology; others reflect growing application of ideas that push practice beyond ‘traditional’ models of evaluation. User-centric and shared approaches are leading to better informed measurement and evaluation design. Theory-based evaluation and impact management embolden us to ask better research questions and obtain more useful answers. Data linkage, the availability of big data, and the possibilities arising from remote sensing are increasing the number of questions we can answer. And data visualisation opens up doors to better understanding and communication of this data. Here we present each of these eight innovations and showcase examples of how organisations are using them to better understand and improve their work….(More)”

How open data can help the Global South, from disaster relief to voter turnout


Stefaan Verhulst and Andrew Young in The Conversation Global: “The modern era is marked by growing faith in the power of data. “Big data”, “open data”, and “evidence-based decision-making” have become buzzwords, touted as solutions to the world’s most complex and persistent problems, from corruption and famine to the refugee crisis.

While perhaps most pronounced in higher income countries, this trend is now emerging globally. In Africa, Latin America, Asia and beyond, hopes are high that access to data can help developing economies by increasing transparency, fostering sustainable development, building climate resiliency and the like.

This is an exciting prospect, but can opening up data actually make a difference in people’s lives?

Getting data-driven about data

The GovLab at New York University spent the last year trying to answer that question….

Our conclusion: the enthusiasm is justified – as long as it’s tempered with a good measure of realism, too. Here are our six major takeaways:

1. We need a framework – Overall, there is still little evidence to substantiate the enthusiastic claims that open data can foment sustainable development and transform governance. That’s not surprising given the early stage of most open data initiatives.

It may be early for impact evaluation, but it’s not too soon to develop a model that will eventually allow us to assess the impact of opening up data over time.

To that end, the GovLab has created an evidence-based framework that aims to better capture the role of open data in developing countries. The Open Data Logic Framework below focuses on various points in the open data value cycle, from data supply to demand, use and impact.

Logic model of open data. The GovLab

2. Open data has real promise – Based on this framework and the underlying evidence that fed into it, we can guardedly conclude that open data does in fact spur development – but only under certain conditions and within the right supporting ecosystem.

One well-known success took place after Nepal’s 2015 earthquake when open data helped NGOs map important landmarks such as health facilities and road networks, among other uses.

And in Colombia, the International Centre for Tropical Agriculture launched Aclímate Colombia, a tool that gives smallholder farmers data-driven insight into planting strategies that makes them more resilient to climate change….

3. Open data can improve people’s lives Examining projects in a number of sectors critical to development, including health, humanitarian aid, agriculture, poverty alleviation, energy and education, we found four main ways that data can have an impact….

4. Data can be an asset in development While these impacts are apparent in both developed and developing countries, we believe that open data can have a particularly powerful role in developing economies.

Where data is scarce, as it often is in poorer countries, open data can lead to an inherently more equitable and democratic distribution of information and knowledge. This, in turn, may activate a wider range of expertise to address complex problems; it’s what we in the field call “open innovation”.

This quality can allow resource-starved developing economies to access and leverage the best minds around.

And because trust in government is quite low in many developing economies, the transparency bred of releasing data can have after-effects that go well beyond the immediate impact of the data itself…

5. The ingredients matter To better understand why some open data projects fail while others succeed, we created a “periodic table” of open data (below), which includes 27 enabling factors divided into five broad categories….

6. We can plan for impact Our report ends by identifying how development organisations can catalyse the release and use of open data to make a difference on the ground.

Recommendations include:

· Define the problem, understand the user, and be aware of local conditions;

· Focus on readiness, responsiveness and change management;

· Nurture an open data ecosystem through collaboration and partnerships;

· Have a risk mitigation strategy;

· Secure resources and focus on sustainability; and

· Build a strong evidence base and support more research.

Next steps

In short, while it may still be too early to fully capture open data’s as-of-yet muted impact on developing economies, there are certainly reasons for optimism.

Much like blockchaindrones and other much-hyped technical advances, it’s time to start substantiating the excitement over open data with real, hard evidence.

The next step is to get systematic, using the kind of analytical framework we present here to gain comparative and actionable insight into if, when and how open data works. Only by getting data-driven about open data can we help it live up to its potential….(More)

Why and How Open Data Matters for Developing Economies


The GovLab is pleased to announce the launch of a new report that assesses and explores ways open data can be used in developing economies, “Open Data in Developing Economies: Toward Building an Evidence Base on What Works and How.” The new report co-authored by Stefaan Verhulst and Andrew Young seeks to :

(1) provides an evidence-based framework governments, NGOs, donors, and others can use to assess the impacts resulting from the use of open data in developing economies;

(2) outlines four key impact areas gleaned from 12 case studies that feature real-world examples from 12 countries, ranging from Colombia to Nepal; and

(3) identifies 27 critical factors that help to determine the success (or failure) of open data uses in developing economies, organized into a newly-created Periodic Table of Open Data, along with recommendations for both practitioners and decision-makers, including donor agencies.

table

The “Open Data in Developing Economies” report is the result of a months-long collaboration between The GovLab and several partners, including the U.S. Agency for International Development, FHI 360, and the World Wide Web Foundation.

The full report and cases studies are available at ODimpact.org…”

 

Government initiative offers Ghanaians chance for greater participation


Springwise: “Openness and transparency are key ingredients in building an accountable and effective democratic government. An “open” government is transparent, accessible to anyone, anytime, anywhere; and is responsive to new ideas and demands. The key to this is providing access to accurate data to all citizens. However, in many countries, a low rate of citizen participation and involvement has led to poor accountability from government officials. In Ghana, a new project, TransGov, is developing innovative tools to foster participation in local governance of marginalised groups, and improve government accountability to those who need it most.

TransGov’s research found that many Ghanaians were not aware of the status of local development projects, and this has led to a general public apathy, where people felt they had no influence on getting the government to work for them. TransGov created a platform to enhance information disclosure, dissemination and to create ways for citizens to engage with the local leaders in their communities. The TransGov platform allows all citizens to track the progress of government projects in their area and to publish information about those projects. TransGov has four integrated platforms, including a website, mobile app, voice response technology (IVR) and SMS – to allow the participation of people from a wide range of socio-economic backgrounds.

The organization has recently partnered with the government-sponsored Ghana Open Data Initiative, to share resources, tools, and research and hold workshops and seminars. This is aimed to strengthen various government agencies in collecting and managing data for public use. The hope is that making this information more accessible will help create more business opportunities and drive innovation, as well as increasing democratic participation. We have seen this in educational radio broadcasts in Cairo subways and an app that allows citizen feedback on city development….(More)”.

How Africa’s Data Revolution Can Deliver Sustainable Development Outcomes


Donald Mogeni at Huffington Post: “…As a demonstration of this political will, several governments in Africa are blazing the trail in numerous ways. For instance, the Government of Senegal now considers investment in data as important as it would treat investment in physical infrastructure such as roads. In Ghana and Sierra Leone, more policy-makers and legislators are now using data to inform their work and make planning is continuously evidence-based.

Despite the progressive developments, several cautionary statements are worth noting. Firstly, data is not a silver-bullet to addressing present development challenges and/or problems. To be transformative, use of data and evidence must include political agency and citizen mobilization. Thus, while data may highlight important development cleavages, it may not guarantee change if not used appropriately within the various political contexts. ‘Everyone Counts’, a new global initiative by CARE, KWANTU and World Vision (that was also showcased in the meeting) seeks to contribute to this agenda.

Secondly, there is need for data ‘experts’ to move beyond the chronic obsession with big numbers to ensure greater inclusion of marginalised and vulnerable segments of the population. Achieving this will require a ‘business unusual’ approach that devises better data collection methodologies and technologies that must collect more and better than ever before. This ‘new’ data should then be used together with administrative and open data to ensure that ‘no one is left behind’.

Thirdly, the utility of citizen-generated data is still contentious – especially within state institutions. Increasing the value of this data must therefore involve standardization of data collection tools and methodologies across the board (to the extent possible), making consideration for ethical approvals, subjecting this data to quality audits and triangulation, as well as adhering to quality assurance standards.

Fourthly, the emergence of various data communities within African countries has made the roles of National Statistical Offices in the data ecosystem even more crucial. However, significant capacity and technical disparities exist between the various National Statistical Offices (NSOs) in Africa. To realise the potential of data and statistics in achieving sustainable development outcomes, financial and human capacities of these institutions must to be enhanced….(More)”.

Open data on universities – New fuel for transformation


François van Schalkwyk at University World News: “Accessible, usable and relevant open data on South African universities makes it possible for a wide range of stakeholders to monitor, advise and challenge the transformation of South Africa’s universities from an informed perspective.

Some describe data as the new oil while others suggest it is a new form of capital or compare it to electricity. Either way, there appears to be a groundswell of interest in the potential of data to fuel development.

Whether the proliferation of data is skewing development in favour of globally networked elites or disrupting existing asymmetries of information and power, is the subject of ongoing debate. Certainly, there are those who will claim that open data, from a development perspective, could catalyse disruption and redistribution.

Open data is data that is free to use without restriction. Governments and their agencies, universities and their researchers, non-governmental organisations and their donors, and even corporations, are all potential sources of open data.

Open government data, as a public rather than a private resource, embedded in principles of universal access, participation and transparency, is touted as being able to restore the deteriorating levels of trust between citizens and their governments.

Open data promises to do so by making the decisions and processes of the state more transparent and inclusive, empowering citizens to participate and to hold public institutions to account for the distribution of public services and resources.

Benefits of open data

Open data has other benefits over its more cloistered cousins (data in private networks, big data, etc). By democratising access, open data makes possible the use of data on, for example, health services, crime, the environment, procurement and education by a range of different users, each bringing their own perspective to bear on the data. This can expose bias in the data or may improve the quality of the data by surfacing data errors. Both are important when data is used to shape government policies.

By removing barriers to reusing data such as copyright or licence-fees, tech-savvy entrepreneurs can develop applications to assist the public to make more informed decisions by making available easy-to-understand information on medicine prices, crime hot-spots, air quality, beneficial ownership, school performance, etc. And access to open research data can improve quality and efficiency in science.

Scientists can check and confirm the data on which important discoveries are based if the data is open, and, in some cases, researchers can reuse open data from other studies, saving them the cost and effort of collecting the data themselves.

‘Open washing’

But access alone is not enough for open data to realise its potential. Open data must also be used. And data is used if it holds some value for the user. Governments have been known to publish server rooms full of data that no one is interested in to support claims of transparency and supporting the knowledge economy. That practice is called ‘open washing’. …(More)”

Features of Parliamentary Websites in Selected Jurisdictions


Report by The Law Library of Congress, Global Legal Research Center: “In recent years, parliaments around the world have enhanced their websites in order to improve access to legislative information and other parliamentary resources. Innovative features allow constituents and researchers to locate and utilize detailed information on laws and lawmaking in various ways. These include tracking tools and alerts, apps, the use of open data technology, and different search functions. In order to demonstrate some of the developments in this area, staff from the Global Legal Research Directorate of the Law Library of Congress surveyed the official parliamentary websites of fifty countries from all regions of the world, plus the website of the European Parliament. In some cases, information on more than one website is provided where separate sites have been established for different chambers of the national parliament, bringing the total number of individual websites surveyed to seventy.

While the information on the parliamentary websites is primarily in the national language of the particular country, around forty of the individual websites surveyed were found to provide at least limited information in one or more other languages. The European Parliament website can be translated into any of the twenty-four official languages of the members of the European Union.

All of the parliamentary websites included in the survey have at least basic browse tools that allow users to view legislation in a list format, and that may allow for viewing in, for example, date or title order. All of the substantive websites also enable searching, often providing a general search box for the whole site at the top of each page as well as more advanced search options for different types of documents. Some sites provide various facets that can be used to further narrow searches.

Around thirty-nine of the individual websites surveyed provide users with some form of tracking or alert function to receive updates on certain documents (including proposed legislation), parliamentary news, committee activities, or other aspects of the website. This includes the ability to subscribe to different RSS feeds and/or email alerts.

The ability to watch live or recorded proceedings of different parliaments, including debates within the relevant chamber as well as committee hearings, is a common feature of the parliamentary websites surveyed. Fifty-eight of the websites surveyed featured some form of video, including links to dedicated YouTube channels, specific pages where users can browse and search for embedded videos, and separate video services or portals that are linked to or viewable from the main site. Some countries also make videos available on dedicated mobile-friendly sites or apps, including Denmark, Germany, Ireland, the Netherlands, and New Zealand. In total, apps containing parliamentary information are provided in just fourteen of the countries surveyed. In comparison, the parliamentary websites of thirty countries are available in mobile-friendly formats, enabling easy access to information and different functionalities using smartphones and tablets.

The table also provides information on some of the additional special features available on the surveyed websites. Examples include dedicated sites or pages that provide educational information about the parliament for children (Argentina, El Salvador, Germany, Israel, Netherlands, Spain, Taiwan, Turkey); calendar functions, including those that allow users to save information to their personal calendars or otherwise view information about different types of proceedings or events (available on at least twenty websites); and open data portals or other features that allow information to be downloaded in bulk for reuse or analysis, including through the use of APIs (application programming interfaces) (at least six countries)….(More)”.

Open Data Blueprint


ODX Canada: “In Canada, the open data environment should be viewed as a supply chain. The movement of open data from producers to consumers involves many different organizations, people, activities, projects and initiatives, all of which work together to push out a final product. Naturally, if there is a break or hurdle in this supply chain, it doesn’t work efficiently. A fundamental hurdle highlighted by companies across the country was the inability to scale their business at the provincial, national and international levels.

This blueprint aims to address the challenges Canadian entrepreneurs are facing by encouraging municipalities to launch open data initiatives. By sharing best practices, we hope to encourage the accessibility of datasets within existing jurisdictions. The structured recommendations in this Open Data Blueprint are based on feedback and best practices seen in major cities across Canada collected through ODX’s primary research….(More)”

(Read more about the OD150 initiative here)

Connecting societal issues, users and data. Scenario-based design of open data platforms


Erna Ruijer et al in Government Information Quarterly: “Governments around the world make their data available through platforms but, disappointingly, the use of this data is lagging behind. This problem has been recognized in the literature and to facilitate use of open datasets, scholars have focused on identifying general user requirements for open data platform design. This approach however fails to take into account the variation of open data practices and specific contexts of usage. This study, therefore, argues that next to general requirements: we also need to collect context-specific user requirements for open data platforms. We take different societal issues as the starting point for open data platform design. To illustrate the value of this context-specific approach, we apply scenario-based design methodology in the Province of Groningen in The Netherlands. The results show that different scenarios result partly in similar but also partly in different user requirements, leading to a deeper and richer understanding of user requirements. We conclude that a context-specific approach thereby connecting data, users and societal issues can be used to guide government agencies and designers in efforts to develop open data platforms that actually meet the needs of citizens….(More)”.

Madrid as a democracy lab


Bernardo Gutiérrez at OpenDemocracy: “…The launch of Decide Madrid, the city participation platform running on the Consul free software, signaled a real revolution. On the one hand, it paved the way for democracy from the bottom up, through direct and binding mechanisms. Unlike other historical participatory budgets, the 100 million Euros devoted to Decide Madrid participatory budgets in 2017 are allocated according to proposals coming from below. The proposals that get the most votes, whenever technically feasible, are approved. The platform also carries a section for “citizen proposals”. …

The Decide Madrid platform was not initially well received by the traditional neighbourhood associations, used to face-to-face participation and to mediating between citizens and government. In order to tackle this, a number of face-to-face deliberation spaces are being set up, such as the Local Forums (physical participation spaces in the districts), and also projects such as If you feel like a cat (participation for children and teenagers), or processes such as G1000, which aims at promoting collective deliberation and fostering proposals from below on the basis of a representative sample of the population, so that the participants’ diversity and plurality is guaranteed.

Most projects are being carried out with the support of the new Laboratories of Citizen Innovation of the prestigious Medialab-Prado. The Participa LAB(Collective Intelligence for Democracy), the DataLab (open data) and the InciLab (Citizen Innovation Lab) are joint public/common initiatives, acting as a bridge between local government and citizens. The Participa LAB, which is the one working more closely on participation, is collaborating with Decide Madridin a number of projects (Codat Madrid hackathons, If you feel like a cat, community lines, gamification, G1000, narrative groups…) and coordinates the Collective Intelligence for Democracy international call. InciLab has launched, among many other initiatives, the Madrid Listens project, to connect City Hall officials with citizens on concrete projects, blending disintermediation and the citizen lab philosophy.

More than 300.000 users strong, Decide Madrid is consolidating itself as the hegemonic space for participation in the city. It activates a variety of processes, debates, proposals, and projects. Its free software means that any city can adapt Consul to its needs, without any substantial investment, and set up a platform. From Barcelona to A Coruña, from Rome to Paris and Buenos Aires, dozens of institutions around the world have replicated the initial Decide Madrid core, thus setting up what Audrey Tang, Taiwan’s digital minister, calls a “liquid federation of cities”. Ada Colau, the mayor of Barcelona, praising the cooperative network of participation cities says: “It is very interesting that in Barcelona we have been able to carry out our first experience of digital participation, Decidim Barcelona, adapting Madrid’s base code. Once we have had a first proposal, we have shared it with many municipalities throughout Catalonia”.

Distributed democracy

The brain as a metaphor. A map of Hamburg (Germany) as a symbol of the networked, decentralized city. Neurons and neighbourhoods connected by flows, inevitably synchronized. Both images are to be found in Emergency, Steven Johnson’s classic book on collective intelligence processes. The city as a brain, as a whole made of decentralized nodes. The city as an open network, where any neighbourhood-node can connect with any other. Caio Vassão’s concept of a distributed city rounds the edges of the city with no centre, “networked, open, fluid, flexible, adaptable, reconfigurable”. A city where the neighbourhoods in the suburbs dialogue and relate to each other without the mediation of a historical center.

Left: Diagram of the human brain. Image source: Mittermeier. Right: Map of Hamburg, circa 1850. Image source: Princeton Architectural Press.

Madrid has kick-started a forceful decentralization policy. Distributed democracy in Madrid can be seen in how budgets are allocated, how city districts have multiplied their resources and partly manage cultural festivals (like the Summers in the City) and cultural projects (Madrid District).

At the same time, the launching of the Local Forums is a clear move to decentralize power and participation in the city. Through projects such as Experiment District (travelling citizen laboratories), Imagine Madrid (rethinking 10 territories) or the M.A.R.E.S project, Spain’s capital city is redrawing its neighbourhood fabric, its economic relations, and citizen involvement in decision making. The successful Medialab-Prado’s Experiment District project, which has already visited Villaverde, Moratalaz and Fuencarral, is in full expansion. It is about to even launch a global call, as dozens of cities around the world want to replicate it. Medialab-Prado, one of the city innovation centres, defines Experiment District as a set of “citizen labs for experimenting and collaborative learning in which anyone can participate”. Citizen (neighbourhood) labs based on the prototyping culture, an open and collaborative way of developing projects. Citizen (neighbourhood) labs for learning and teaching, where the result is not a perfect product, but a process that can be improved in real time through the collaboration of citizens from the Madrid neighbourhoods….(More)”