Using Crowds for Evaluation Tasks: Validity by Numbers vs. Validity by Expertise


Paper by Christoph Hienerth and Frederik Riar:Developing and commercializing novel ideas is central to innovation processes. As the outcome of such ideas cannot fully be foreseen, the evaluation of them is crucial. With the rise of the internet and ICT, more and new kinds of evaluations are done by crowds. This raises the question whether individuals in crowds possess necessary capabilities to evaluate and whether their outcomes are valid. As empirical insights are not yet available, this paper deals with the examination of evaluation processes and general evaluation components, the discussion of underlying characteristics and mechanism of these components affecting evaluation outcomes (i.e. evaluation validity). We further investigate differences between firm- and crowd-based evaluation using different cases of applications, and develop a theoretical framework towards evaluation validity, i.e. validity by numbers vs. the validity by expertise. The identified factors that influence the validity of evaluations are: (1) the number of evaluation tasks, (2) complexity, (3) expertise, (4) costs, and (5) time to outcome. For each of these factors, hypotheses are developed based on theoretical arguments. We conclude with implications, proposing a model of evaluation validity.”

City 72 Toolkit


“An effective preparedness platform customizable to your city. City72 is an open-source emergency preparedness platform that promotes community resilience and connection. This Toolkit is designed specifically for emergency preparedness organizations and provides the information and resources to create a customized City72 site for any city or region. It includes: how to create localized content, access to the code to build and install your City72 website, and tips for how to manage and promote your site.”

Digital Government: Turning the Rhetoric into Reality


Miguel Carrasco and Peter Goss at BCG Perspectives: “Getting better—but still plenty of room for improvement: that’s the current assessment by everyday users of their governments’ efforts to deliver online services. The public sector has made good progress, but most countries are not moving nearly as quickly as users would like. Many governments have made bold commitments, and a few countries have determined to go “digital by default.” Most are moving more modestly, often overwhelmed by complexity and slowed by bureaucratic skepticism over online delivery as well as by a lack of digital skills. Developing countries lead in the rate of online usage, but they mostly trail developed nations in user satisfaction.
Many citizens—accustomed to innovation in such sectors as retailing, media, and financial services—wish their governments would get on with it. Of the services that can be accessed online, many only provide information and forms, while users are looking to get help and transact business. People want to do more. Digital interaction is often faster, easier, and more efficient than going to a service center or talking on the phone, but users become frustrated when the services do not perform as expected. They know what good online service providers offer. They have seen a lot of improvement in recent years, and they want their governments to make even better use of digital’s capabilities.
Many governments are already well on the way to improving digital service delivery, but there is often a gap between rhetoric and reality. There is no shortage of government policies and strategies relating to “digital first,” “e-government,” and “gov2.0,” in addition to digital by default. But governments need more than a strategy. “Going digital” requires leadership at the highest levels, investments in skills and human capital, and cultural and behavioral change. Based on BCG’s work with numerous governments and new research into the usage of, and satisfaction with, government digital services in 12 countries, we see five steps that most governments will want to take:

1. Focus on value. Put the priority on services with the biggest gaps between their importance to constituents and constituents’ satisfaction with digital delivery. In most countries, this will mean services related to health, education, social welfare, and immigration.

2. Adopt service design thinking. Governments should walk in users’ shoes. What does someone encounter when he or she goes to a government service website—plain language or bureaucratic legalese? How easy is it for the individual to navigate to the desired information? How many steps does it take to do what he or she came to do? Governments can make services easy to access and use by, for example, requiring users to register once and establish a digital credential, which can be used in the future to access online services across government.

3. Lead users online, keep users online. Invest in seamless end-to-end capabilities. Most government-service sites need to advance from providing information to enabling users to transact their business in its entirety, without having to resort to printing out forms or visiting service centers.

4. Demonstrate visible senior-leadership commitment. Governments can signal—to both their own officials and the public—the importance and the urgency that they place on their digital initiatives by where they assign responsibility for the effort.

5. Build the capabilities and skills to execute. Governments need to develop or acquire the skills and capabilities that will enable them to develop and deliver digital services.

This report examines the state of government digital services through the lens of Internet users surveyed in Australia, Denmark, France, Indonesia, the Kingdom of Saudi Arabia, Malaysia, the Netherlands, Russia, Singapore, the United Arab Emirates (UAE), the UK, and the U.S. We investigated 37 different government services. (See Exhibit 1.)…”

Why This Simple Government Website Was Named the Best Design of the Year


Kelsey Campbell-Dollaghan on Gizmodo: “When was the last time you tried to find a government form on the Internet? For me, it was a few months back, trying to track down an absentee ballot. And while I love American flag GIFs as much as the next patriot, I was amazed at the labyrinth of independent sites I had to visit before I found what I was looking for. Bringing the web presence of an entire government under one roof is a Sisyphean task, and the UK is one of the only countries that’s managed to do it, with Gov.uk, a one-stop-web-shop that launched earlier this year.

Today, at a ceremony in London, the site was named the 2013 Best Design of the Year by the Design Museum, beating out 99 shortlisted buildings, inventions, and cars for the honor. It’s the first website to ever win the six-year-old title, too—which illustrates just how remarkable the achievement really is.

Here’s what makes it so deceivingly special.

 

Why This Simple Government Website Was Named the Best Design of the Year

 


Why does a straightforward, cut-and-dry website deserve the award? Because of that straightforwardness, actually. “There were thousands of websites, and we folded them into Gov.uk to make just one,” says Ben Terrett, head of design at the UK’s Government Digital Service, in a Dezeen-produced video. “Booking a prison stay should be as easy as booking a driver’s license test.”…

Terrett describes Gov.uk as an attempt to bring web design up to speed with technology like Glass, where the user interfacer all but disappears. “We haven’t achieved that yet with most web interfaces, [where] you can still see the graphic design,” he says. “But technology will change, and we’ll get past that.”

Design in public and social innovation


New paper by Geoff Mulgan (Nesta): “What’s going right and what’s going wrong? Is design a key to more efficient and effective public services, or a costly luxury, good for conferences and consultants but not for the public?
This paper looks at the elements of the design method; the strengths of current models; some of their weaknesses and the common criticisms made of them; and what might be the way forward.

Contents:

  • Strengths of the design method in social innovation and public service
  • Social design tools table
  • Weaknesses of design projects and methods
  • The challenge
  • Design in the context of innovation”

E-Government and Its Limitations: Assessing the True Demand Curve for Citizen Public Participation


Paper by David Karpf: “Many e-government initiatives start with promise, but end up either as digital “ghost towns” or as a venue exploited by organized interests. The problem with these initiatives is rooted in a set of common misunderstandings about the structure of citizen interest in public participation – simply put, the Internet does not create public interest, it $2 public interest. Public interest can be high or low, and governmental initiatives can be polarized or non-polarized. The paper discusses two common pitfalls (“the Field of Dreams Fallacy” and “Blessed are the Organized”) that demand alternate design choices and modified expectations. By treating public interest and public polarization as variables, the paper develops a typology of appropriate e-government initiatives that can help identify the boundary conditions for transformative digital engagement.”

E-Government and Its Limitations: Assessing the True Demand Curve for Citizen Public Participation


Paper by David Karpf:  “Many e-government initiatives start with promise, but end up either as digital “ghost towns” or as a venue exploited by organized interests.  The problem with these initiatives is rooted in a set of common misunderstandings about the structure of citizen interest in public participation – simply put, the Internet does not create public interest, it reveals public interest.  Public interest can be high or low, and governmental initiatives can be polarized or non-polarized.  The paper discusses two common pitfalls (“the Field of Dreams Fallacy” and “Blessed are the Organized”) that demand alternate design choices and modified expectations.  By treating public interest and public polarization as variables, the paper develops a typology of appropriate e-government initiatives that can help identify the boundary conditions for transformative digital engagement….

 Typology

Figure 1: Typology of Appropriate E-government Projects”

Online public services and Design Thinking for governments


Ela Alptekin: “The digital era has changed the expectations citizens have regarding the communication of public services and their engagement with government agencies. ‘Digital Citizenship’ is common place and this is a great opportunity for institutions to explore the benefits this online presence offers.

Most government agencies have moved their public services to digital platforms by applying technology to the exact same workflow they had earlier. They’ve replaced hard copies with emails and signatures with digital prints. However, Information Technologies don’t just improve the efficiency of governments, they also have the power to transform how governments work by redefining their engagement with citizens. With this outlook they can expand the array of services that could be provided and implemented.

When it comes to online public services there are two different paths to building-up a strategy: Governments can either: Use stats, trends and quantitative surveys to measure and produce “reliable results”; or they can develop a deeper understanding of the basic needs of their consumers for a specific problem. With that focus, they may propose a solid solution that would satisfy those needs.

Two of the primary criteria of evaluation in any measurement or observation are:

  1. Does the same measurement process yields the same results?

  2. Are we measuring what we intend to measure?

These two concepts are reliability and validity.

According to Roger Martin, author of “The Design of Business”, truly innovative organisations are those that have managed to balance the “reliability” of analytical thinking with the “validity” of abductive thinking. Many organisations often don’t find this balance between reliability and validity and choose only the reliable data to move on with their future implementations.

So what is the relationship between reliability and validity? The two do not necessarily go hand-in-hand.

At best, we have a measure that has both high validity and high reliability. It yields consistent results in repeated application and it accurately reflects what we hope to represent.

It is possible to have a measure that has high reliability but low validity – one that is consistent in getting bad information or consistent in missing the mark. *It is also possible to have one that has low reliability and low validity – inconsistent and not on target.

Finally, it is not possible to have a measure that has low reliability and high validity – you can’t really get at what you want or what you’re interested in if your measure fluctuates wildly.” – click here for further reading.

Many online, government, public services are based on reliable data and pay no attention to the validity of the results ( 1st figure “reliable but not valid” ).

What can government agencies use to balance the reliability and validity when it comes to public services? The answer is waiting in Design Thinking and abductive reasoning.

….Design thinking helps agencies to go back to the basics of what citizens need from their governments. It can be used to develop both reliable and valid online public services that are able to satisfy their needs….

As Government accelerates towards a world of public services that are digital by default, is this going to deliver the kind of digital services that move the public with them?

To find out, thinkpublic partnered with Consumer Focus (UK) to undertake detailed research into some of the fundamental questions and issues that users of digital public services are interested in. The findings have been published today in the Manifesto for Online Public Services, which sets out simple guiding principles to be placed at the heart of online service design.”

Creating Networked Cities


New Report by Alissa Black and Rachel Burstein, New America Foundation: “In April 2013 the California Civic Innovation Project released a report, The Case for Strengthening Personal Networks in California Local Governments, highlighting the important role of knowledge sharing in the diffusion of innovations from one city or county to another, and identifying personal connections as a significant source of information when it comes to learning about and implementing innovations.
Based on findings from CCIP’s previous study, Creating Networked Cities makes recommendations on how local government leaders, professional associations, and foundation professionals might promote and improve knowledge sharing through developing, strengthening and leveraging their networks. Strong local government networks support the continual sharing and advancement of projects, emerging practices, and civic innovation…Download CCIP’s recommendations for strengthening local government networks and diffusing innovation here.”

What should we do about the naming deficit/surplus?


in mySociety Blog: “As I wrote in my last post, I am very concerned about the lack of comprehensible, consistent language to talk about the hugely diverse ways in which people are using the internet to bring about social and political change….My approach to finding an appropriate name was to look at the way that other internet industry sectors are named, so that I could choose a name that sits nicely next to very familiar sectoral labels….

Segmenting the Civic Power sector

Choosing a single sectoral name – Civic Power – is not really the point of this exercise. The real benefit would come from being able to segment the many projects within this sector so that they are more easy to compare and contrast.

Here is my suggested four part segmentation of the Civic Power sector…:

  1. Decision influencing organisations try to directly shape or change particular decisions made by powerful individuals or organisations.
  2. Regime changing organisations try to replace decision makers, not persuade them.
  3. Citizen Empowering organisations try to give people the resources and the confidence required to exert power for whatever purpose those people see fit, both now and in the future.
  4. Digital Government organisations try to improve the ways in which governments acquire and use computers and networks. Strictly speaking this is just a sub-category of ‘decision influencing organisation’, on a par with an environmental group or a union, but more geeky.”

See also: Open Government – What’s in a Name?