New book on Open Data and the Future of Civic Innovation: The rise of open data in the public sector has sparked innovation, driven efficiency, and fueled economic development. And in the vein of high-profile federal initiatives like Data.gov and the White House’s Open Government Initiative, more and more local governments are making their foray into the field with Chief Data Officers, open data policies, and open data catalogs.
While still emerging, we are seeing evidence of the transformative potential of open data in shaping the future of our civic life. It’s at the local level that government most directly impacts the lives of residents—providing clean parks, fighting crime, or issuing permits to open a new business. This is where there is the biggest opportunity to use open data to reimagine the relationship between citizens and government.
Beyond Transparency is a cross-disciplinary survey of the open data landscape, in which practitioners share their own stories of what they’ve accomplished with open civic data. It seeks to move beyond the rhetoric of transparency for transparency’s sake and towards action and problem solving. Through these stories, we examine what is needed to build an ecosystem in which open data can become the raw materials to drive more effective decision-making and efficient service delivery, spur economic activity, and empower citizens to take an active role in improving their own communities….
This book is a resource for (and by) practitioners inside and outside government—from the municipal chief information officer to the community organizer to the civic-minded entrepreneur. Beyond Transparency is intended to capture and distill the community’s learnings around open data for the past four years. And we know that the community is going to continue learning. That’s why, in addition to the print version of the book which you can order on Amazon, we’ve also published the digital version of this book on this site under a Creative Commons license. The full text of this site is on GitHub — which means that anyone can submit a pull request with a suggested edit. Help us improve this resource for the community and write the next edition of Beyond Transparency by submitting your pull requests.
Code for America is a national nonprofit committed to building a government for the people, by the people, that works in the 21st century. Over the past four years, CfA has worked with dozens of cities to support civic innovation through open data. You can support this work by contributing to the book on GitHub, joining the CfA volunteer community (the Brigade), or connecting your city with CfA.
Special issue of FirstMonday: "Making data — Big data and beyond"
Introduction by Rasmus Helles and Klaus Bruhn Jensen: “Data are widely understood as minimal units of information about the world, waiting to be found and collected by scholars and other analysts. With the recent prominence of ‘big data’ (Mayer–Schönberger and Cukier, 2013), the assumption that data are simply available and plentiful has become more pronounced in research as well as public debate. Challenging and reflecting on this assumption, the present special issue considers how data are made. The contributors take big data and other characteristic features of the digital media environment as an opportunity to revisit classic issues concerning data — big and small, fast and slow, experimental and naturalistic, quantitative and qualitative, found and made.
Data are made in a process involving multiple social agents — communicators, service providers, communication researchers, commercial stakeholders, government authorities, international regulators, and more. Data are made for a variety of scholarly and applied purposes, oriented by knowledge interests (Habermas, 1971). And data are processed and employed in a whole range of everyday and institutional contexts with political, economic, and cultural implications. Unfortunately, the process of generating the materials that come to function as data often remains opaque and certainly under–documented in the published research.
The following eight articles seek to open up some of the black boxes from which data can be seen to emerge. While diverse in their theoretical and topical focus, the articles generally approach the making of data as a process that is extended in time and across spatial and institutional settings. In the common culinary metaphor, data are repeatedly processed, rather than raw. Another shared point of attention is meta–data — the type of data that bear witness to when, where, and how other data such as Web searches, e–mail messages, and phone conversations are exchanged, and which have taken on new, strategic importance in digital media. Last but not least, several of the articles underline the extent to which the making of data as well as meta–data is conditioned — facilitated and constrained — by technological and institutional structures that are inherent in the very domain of analysis. Researchers increasingly depend on the practices and procedures of commercial entities such as Google and Facebook for their research materials, as illustrated by the pivotal role of application programming interfaces (API). Research on the Internet and other digital media also requires specialized tools of data management and analysis, calling, once again, for interdisciplinary competences and dialogues about ‘what the data show.’”
See Table of Contents
The Brave New World of Good
Brad Smith: “Welcome to the Brave New World of Good. Once almost the exclusive province of nonprofit organizations and the philanthropic foundations that fund them, today the terrain of good is disputed by social entrepreneurs, social enterprises, impact investors, big business, governments, and geeks. Their tools of choice are markets, open data, innovation, hackathons, and disruption. They cross borders, social classes, and paradigms with the swipe of a touch screen. We seemed poised to unleash a whole new era of social and environmental progress, accompanied by unimagined economic prosperity.
As a brand, good is unassailably brilliant. Who could be against it? It is virtually impossible to write an even mildly skeptical blog post about good without sounding well, bad — or at least a bit old-fashioned. For the record, I firmly believe there is much in the brave new world of good that is helping us find our way out of the tired and often failed models of progress and change on which we have for too long relied. Still, there are assumptions worth questioning and questions worth answering to ensure that the good we seek is the good that can be achieved.
…
Open Data
Second only to “good” in terms of marketing genius is the concept of “open data.” An offspring of previous movements such as “open source,” “open content,” and “open access,” open data in the Internet age has come to mean data that is machine-readable, free to access, and free to use, re-use, and re-distribute, subject to attribution. Fully open data goes way beyond posting your .pdf document on a Web site (as neatly explained by Tim Berners Lee’s five-star framework).
When it comes to government, there is a rapidly accelerating movement around the world that is furthering transparency by making vast stores of data open. Ditto on the data of international aid funders like the United States Agency for International Development, the World Bank, and the Organisation for Economic Co-operation and Development. The push has now expanded to the tax return data of nonprofits and foundations (IRS Forms 990). Collection of data by government has a business model; it’s called tax dollars. However, open data is not born pure. Cleaning that data, making it searchable, and building and maintaining reliable user interfaces is complex, time-consuming, and often expensive. That requires a consistent stream of income of the kind that can only come from fees, subscriptions, or, increasingly less so, government.
Foundation grants are great for short-term investment, experimentation, or building an app or two, but they are no substitute for a scalable business model. Structured, longitudinal data are vital to social, environmental, and economic progress. In a global economy where government is retreating from the funding of public goods, figuring how to pay for the cost of that data is one of our greatest challenges.”
Global Open Data Initiative moving forward
“The Global Open Data Initiative will serve as a guiding voice internationally on open data issues. Civil society groups who focus on open data have often been isolated to single national contexts, despite the similar challenges and opportunities repeating themselves in countries across the globe. The Global Open Data Initiative aims to help share valuable resources, guidance and judgment, and to clarify the potential for government open data across the world.
Provide a leading vision for how governments approach open data. Open data commitments are among the most popular commitments for countries participating in the Open Government Partnership. The Global Open Data Initiative recommendations and resources will help guide open data initiatives and others as they seek to design and implement strong, effective open data initiatives and policies. Global Open Data Initiative resources will also help civil society actors who will be evaluating government initiatives.
Increase awareness of open data. Global Open Data Initiative will work to advance the understanding of open data issues, challenges, and resources by promoting best practices, engaging in online and offline dialogue, and supporting networking between organizations both new and familiar to the open data arena.
Support the development of the global open data community especially in civil society. Civil society organizations (CSOs) have a key role to play as suppliers, intermediaries, and users of open data, though at present, relatively few organizations are engaging with open data and the opportunities it presents. Most CSOs lack the awareness, skills and support needed to be active users and providers of open data in ways that can help them meet their goals. The Global Open Data Initiative aims to help CSOs, to engage with and use open data whether whatever area they work on – be it climate change, democratic rights, land governance or financial reform.
Our immediate focus is on two activities:
Provide a leading vision for how governments approach open data. Open data commitments are among the most popular commitments for countries participating in the Open Government Partnership. The Global Open Data Initiative recommendations and resources will help guide open data initiatives and others as they seek to design and implement strong, effective open data initiatives and policies. Global Open Data Initiative resources will also help civil society actors who will be evaluating government initiatives.
Increase awareness of open data. Global Open Data Initiative will work to advance the understanding of open data issues, challenges, and resources by promoting best practices, engaging in online and offline dialogue, and supporting networking between organizations both new and familiar to the open data arena.
Support the development of the global open data community especially in civil society. Civil society organizations (CSOs) have a key role to play as suppliers, intermediaries, and users of open data, though at present, relatively few organizations are engaging with open data and the opportunities it presents. Most CSOs lack the awareness, skills and support needed to be active users and providers of open data in ways that can help them meet their goals. The Global Open Data Initiative aims to help CSOs, to engage with and use open data whether whatever area they work on – be it climate change, democratic rights, land governance or financial reform.
Our immediate focus is on two activities:
- To consult with members of the CSO community around the world about what they think is important in this area
- Develop a set of principles in collaboration with the CSO community to guide open government data policies and approaches and to help initiate, strengthen and further elevate conversations between governments and civil society.”
Coase’s theories predicted Internet’s impact on how business is done
Don Tapscott in The Globe and Mail: “Renowned economist Ronald Coase died last week at the age of 102. Among his many achievements, Mr. Coase was awarded the 1991 Nobel Prize in Economics, largely for his inspiring 1937 paper The Nature of the Firm. The Nobel committee applauded the academic for his “discovery and clarification of the significance of transaction costs … for the institutional structure and functioning of the economy.”
Mr. Coase’s enduring legacy may well be that 60 years later, his paper and theories help us understand the Internet’s impact on business, the economy and all our institutions… Mr. Coase wondered why there was no market within the firm. Why is it unprofitable to have each worker, each step in the production process, become an independent buyer and seller? Why doesn’t the draftsperson auction their services to the engineer? Why is it that the engineer does not sell designs to the highest bidder? Mr. Coase argued that preventing this from happening created marketplace friction.
Mr. Coase argued that this friction gave rise to transaction costs – or to put it more broadly, collaboration or relationship costs. There are three types of these relationship costs. First are search costs, such as the hunt for appropriate suppliers. Second are contractual costs, including price and contract negotiations. Third are the co-ordination costs of meshing the different products and processes.
The upshot is that most vertically integrated corporations found it cheaper and simpler to perform most functions in-house, rather than incurring the cost, hassle and risk of constant transactions with outside partners….This is no longer the case. Many behemoths have lost market share to more supple competitors. Digital technologies slash transaction and collaboration costs. Smart companies are making their boundaries porous, using the Internet to harness knowledge, resources and capabilities outside the company. Everywhere,leading firms set a context for innovation and then invite their customers, partners and other third parties to co-create their products and services.
Today’s economic engines are Internet-based clusters of businesses. While each company retains its identity, companies function together, creating more wealth than they could ever hope to create individually. Where corporations were once gigantic, new business ecosystems tend toward the amorphous.
Procter & Gamble now gets 60 per cent of its innovation from outside corporate walls. Boeing has built a massive ecosystem to design and manufacture jumbo jets. China’s motorcycle industry, which consists of dozens of companies collaborating with no single company pulling the strings, now comprises 40 per cent of global motorcycle production.
Looked at one way, Amazon.com is a website with many employees that ships books. Looked at another way, however, Amazon is a vast ecosystem that includes authors, publishers, customers who write reviews for the site, delivery companies like UPS, and tens of thousands of affiliates that market products and arrange fulfilment through the Amazon network. Hundreds of thousands of people are involved in Amazon’s viral marketing network.
This is leading to the biggest change to the corporation in a century and altering how we orchestrate capability to innovate, create goods and services and engage with the world. From now on, the ecosystem itself, not the corporation per se, should serve as the point of departure for every business strategist seeking to understand the new economy – and for every manager, entrepreneur and investor seeking to prosper in it.
Nor does the Internet tonic apply only to corporations. The Web is dropping transaction costs everywhere – enabling networked approaches to almost every institution in society, from government, media, science and health care to our energy grid, transportation systems and institutions for global problem solving.
Governments can change from being vertically integrated, industrial-age bureaucracies to become networks. By releasing their treasures of raw data, governments can now become platforms upon which companies, NGOs, academics, foundations, individuals and other government agencies can collaborate to create public value…”
Confronting Wicked Problems in the Metropolis
An APSA 2013 Annual Meeting Paper by Jered Carr and Brent Never: “These problems facing many metropolitan regions in the U.S. are complex, open-ended and seemingly intractable. The obstacles to regional governance created by these “wicked” problems are the root of the criticisms of the consensus-based “self-organizing” strategies described by frameworks such as New Regionalism and Institutional Collective Action. The self-organized solutions described by these frameworks require substantial consensus exist among the participants and this creates a bias toward solving low-conflict problems where consensus already exists. We discuss the limitations of these two influential research programs in the context of wicked problems and draw on the concept of nested institutional action situations to suggest a research agenda for studying intergovernmental collaboration on problems requiring the development of consensus about the nature of the problem and acceptable solutions. The Advocacy Coalitions and Institutional Analysis and Development frameworks have been effectively used to explain regional collaboration on wicked environmental problems and likely have insights for confronting the wicked fiscal and social problems of regional governance. The implications are that wicked problems are tamed through iterated games and that institution-making at the collective-choice level can then be scaled up to achieve agreement at the constitutional level of analysis.”
Government Is a Good Venture Capitalist
Wall Street Journal: “In a knowledge-intensive economy, innovation drives growth. But what drives innovation? In the U.S., most conservatives believe that economically significant new ideas originate in the private sector, through either the research-and-development investments of large firms with deep pockets or the inspiration of obsessive inventors haunting shabby garages. In this view, the role of government is to secure the basic conditions for honest and efficient commerce—and then get out of the way. Anything more is bound to be “wasteful” and “burdensome.”
The real story is more complex and surprising. For more than four decades, R&D magazine has recognized the top innovations—100 each year—that have moved past the conceptual stage into commercial production and sales. Economic sociologists Fred Block and Matthew Keller decided to ask a simple question: Where did these award-winning innovations come from?
The data indicated seven kinds of originating entities: Fortune 500 companies; small and medium enterprises (including startups); collaborations among private entities; government laboratories; universities; spinoffs started by researchers at government labs or universities; and a grab bag of other public and nonprofit agencies.
Messrs. Block and Keller randomly selected three years in each of the past four decades and analyzed the resulting 1,200 innovations. About 10% originated in foreign entities; the sociologists focused on the domestic innovations, more than 1,050.
Two of their findings stand out. First, the number of award winners originating in Fortune 500 companies—either working alone or in collaboration with others—has declined steadily and sharply, from an annual average of 44 in the 1970s to only nine in the first decade of this century.
Second, the number of top innovations originating in federal laboratories, universities or firms formed by former researchers in those entities rose dramatically, from 18 in the 1970s to 37 in the 1980s and 55 in the 1990s before falling slightly to 49 in the 2000s. Without the research conducted in federal labs and universities (much of it federally funded), commercial innovation would have been far less robust…”
A promising phenomenon of open data: A case study of the Chicago open data project
Paper by Maxat Kassen in Government Information Quarterly: “This article presents a case study of the open data project in the Chicago area. The main purpose of the research is to explore empowering potential of an open data phenomenon at the local level as a platform useful for promotion of civic engagement projects and provide a framework for future research and hypothesis testing. Today the main challenge in realization of any e-government projects is a traditional top–down administrative mechanism of their realization itself practically without any input from members of the civil society. In this respect, the author of the article argues that the open data concept realized at the local level may provide a real platform for promotion of proactive civic engagement. By harnessing collective wisdom of the local communities, their knowledge and visions of the local challenges, governments could react and meet citizens’ needs in a more productive and cost-efficient manner. Open data-driven projects that focused on visualization of environmental issues, mapping of utility management, evaluating of political lobbying, social benefits, closing digital divide, etc. are only some examples of such perspectives. These projects are perhaps harbingers of a new political reality where interactions among citizens at the local level will play an more important role than communication between civil society and government due to the empowering potential of the open data concept.”
Strengthening Local Capacity for Data-Driven Decisionmaking
A report by the National Neighborhood Indicators Partnership (NNIP): “A large share of public decisions that shape the fundamental character of American life are made at the local level; for example, decisions about controlling crime, maintaining housing quality, targeting social services, revitalizing low-income neighborhoods, allocating health care, and deploying early childhood programs. Enormous benefits would be gained if a much larger share of these decisions were based on sound data and analysis.
In the mid-1990s, a movement began to address the need for data for local decisionmaking.Civic leaders in several cities funded local groups to start assembling neighborhood and address-level data from multiple local agencies. For the first time, it became possible to track changing neighborhood conditions, using a variety of indicators, year by year between censuses. These new data intermediaries pledged to use their data in practical ways to support policymaking and community building and give priority to the interests of distressed neighborhoods. Their theme was “democratizing data,” which in practice meant making the data accessible to residents and community groups (Sawicki and Craig 1996).
The initial groups that took on this work formed the National Neighborhood Indicators Partnership (NNIP) to further develop these capacities and spread them to other cities. By 2012, NNIP partners were established in 37 cities, and similar capacities were in development in a number of others. The Urban Institute (UI) serves as the secretariat for the network. This report documents a strategic planning process undertaken by NNIP in 2012 and early 2013. The network’s leadership and funders re-examined the NNIP model in the context of 15 years of local partner experiences and the dramatic changes in technology and policy approaches that have occurred over that period. The first three sections explain NNIP functions and institutional structures and examine the potential role for NNIP in advancing the community information field in today’s environment.”
The Participatory Turn: Participatory Budgeting Comes to America
Thesis by Hollie Russon Gilman: “Participatory Budgeting (PB) has expanded to over 1,500 municipalities worldwide since its inception in Porto Alege, Brazil in 1989 by the leftist Partido dos Trabalhadores (Workers’ Party). While PB has been adopted throughout the world, it has yet to take hold in the United States. This dissertation examines the introduction of PB to the United States with the first project in Chicago in 2009, and proceeds with an in-depth case study of the largest implementation of PB in the United States: Participatory Budgeting in New York City. I assess the outputs of PB in the United States including deliberations, governance, and participation. I argue that PB produces better outcomes than the status quo budget process in New York City, while also transforming how those who participate understand themselves as citizens, constituents, Council members, civil society leaders and community stakeholders. However, there are serious challenges to participation, including high costs of engagement, process exhaustion, and perils of scalability. I devise a framework for assessment called “citizenly politics,” focusing on: 1) designing participation 2) deliberation 3) participation and 4) potential for institutionalization. I argue that while the material results PB produces are relatively modest, including more innovative projects, PB delivers more substantial non-material or existential results. Existential citizenly rewards include: greater civic knowledge, strengthened relationships with elected officials, and greater community inclusion. Overall, PB provides a viable and informative democratic innovation for strengthening civic engagement within the United States that can be streamlined and adopted to scale.”