Talking About a (Data) Revolution


Dave Banisar at Article 19: “It is important to recognize the utility that data can bring. Data can ease analysis, reveal important patterns and facilitate comparisons. For example, the Transactional Access Clearing House (TRAC – http://www.trac.org) at Syracuse University uses data sets from the US Department of Justice to analyze how the federal government enforces its criminal and civil laws, showing how laws are applied differently across the US.
The (somewhat ICT-companies manufactured) excitement over “E-government” in the late 1990s imagined a brave new e-world where governments would quickly and easily provide needed information and services to their citizens. This was presented as an alternative to the “reactive” and “confrontational” right to information laws but eventually led to the realization that ministerial web pages and the ability to pay tickets online did not lead to open government. Singapore ranks near the top every year on e-government but is clearly not an ‘open government’. Similarly, it is important to recognize that governments providing data through voluntary measures is not enough.
For open data to promote open government, it needs to operate within a framework of law and regulation that ensures that information is collected, organized and stored and then made public in a timely, accurate and useful form.   The information must be more than just what government bodies find useful to release, but what is important for the public to know to ensure that those bodies are accountable.
Otherwise, it is in danger of just being propaganda, subject to manipulation to make government bodies look good. TRAC has had to sue the USA federal government dozens of times under the Freedom of Information Act to obtain the government data and after they publish it, some government bodies still claim that the information is incorrect.  Voluntary systems of publication usually fail when they potentially embarrass the bodies doing the publication.
In the countries where open data has been most successful such as the USA and UK, there also exists a legal right to demand information which keeps bodies honest. Most open government laws around the world now have requirements for affirmative publication of key information and they are slowly being amended to include open data requirements to ensure that the information is more easily usable.
Where there is no or weak open government laws, many barriers can obstruct open data. In Kenya, which has been championing their open data portal while being slow to adopt a law on freedom of information, a recent review found that the portal was stagnating. In part, the problem was that in the absence of laws mandating openness, there remains a culture of secrecy and fear of releasing information.
Further, mere access to data is not enough to ensure informed participation by citizens and enable their ability to affect decision-making processes.  Legal rights to all information held by governments – right to information laws – are essential to tell the “why”. RTI reveals how and why decisions and policy are made – secret meetings, questionable contracts, dubious emails and other information. These are essential elements for oversight and accountability. Being able to document why a road was built for political reasons is as crucial for change as recognizing that it’s in the wrong place. The TRAC users, mostly journalists, use the system as a starting point to ask questions or why enforcement is so uneven or taxes are not being collected. They need sources and open government laws to ask these questions.
Of course, even open government laws are not enough. There needs to be strong rights for citizen consultation and participation and the ability to enforce those rights, such as is mandated by the UNECE Convention on Access to Environment Information, Public Participation and Access to Justice (Aarhus Convention). A protocol to that convention has led to a Europe-wide data portal on environmental pollution.
For open data to be truly effective, there needs to be a right to information enshrined in law that requires that information is made available in a timely, reliable format that people want, not just what the government body wants to release. And it needs to be backed up with rights of engagement and participation. From this open data can flourish.  The OGP needs to refocus on the building blocks of open government – good law and policy – and not just the flashy apps.”

Beyond Transparency


New book on Open Data and the Future of Civic Innovation: The rise of open data in the public sector has sparked innovation, driven efficiency, and fueled economic development. And in the vein of high-profile federal initiatives like Data.gov and the White House’s Open Government Initiative, more and more local governments are making their foray into the field with Chief Data Officers, open data policies, and open data catalogs.
While still emerging, we are seeing evidence of the transformative potential of open data in shaping the future of our civic life. It’s at the local level that government most directly impacts the lives of residents—providing clean parks, fighting crime, or issuing permits to open a new business. This is where there is the biggest opportunity to use open data to reimagine the relationship between citizens and government.
Beyond Transparency is a cross-disciplinary survey of the open data landscape, in which practitioners share their own stories of what they’ve accomplished with open civic data. It seeks to move beyond the rhetoric of transparency for transparency’s sake and towards action and problem solving. Through these stories, we examine what is needed to build an ecosystem in which open data can become the raw materials to drive more effective decision-making and efficient service delivery, spur economic activity, and empower citizens to take an active role in improving their own communities….
This book is a resource for (and by) practitioners inside and outside government—from the municipal chief information officer to the community organizer to the civic-minded entrepreneur. Beyond Transparency is intended to capture and distill the community’s learnings around open data for the past four years. And we know that the community is going to continue learning. That’s why, in addition to the print version of the book which you can order on Amazon, we’ve also published the digital version of this book on this site under a Creative Commons license. The full text of this site is on GitHub — which means that anyone can submit a pull request with a suggested edit. Help us improve this resource for the community and write the next edition of Beyond Transparency by submitting your pull requests.
Code for America is a national nonprofit committed to building a government for the people, by the people, that works in the 21st century. Over the past four years, CfA has worked with dozens of cities to support civic innovation through open data. You can support this work by contributing to the book on GitHub, joining the CfA volunteer community (the Brigade), or connecting your city with CfA.

GitHub and Government


New site: “Make government better, together. Stories of open source, open data, and open government.
This site is an open source effort to showcase best practices of open sourcing government. See something that you think could be better? Want to submit your own story? Simply fork the project and submit a pull request.

Ready to get started on GitHub? Here are some ideas that are easy to get your feet wet with.

Feedback Repository

GitHub’s about connecting with developers. Whether you’re an API publishing pro, or just getting started, creating a “feedback” repository can go a long way to connect your organization with the community. Get feedback from current and potential data consumers by creating a specific repository for them to contribute ideas and suggestions for types of data or other information they’d like to see opened. Here’s how:

  1. Create a new repository
    • Choose your organization as the Owner
    • Name the repository “feedback” or similar
    • Click the checkbox to automatically create a README.md file
  2. Set up your Readme
    • Click README.md within your newly created repository
    • Click Edit
    • Introduce yourself, describe why you’ve joined GitHub, what you’re hoping to do and what you’d like to learn from the development community. Encourage them to leave feedback through issues on the repository.

Sample text for your README.md:

# City of Gotham Feedback
We've just joined GitHub and want to know what data would be interesting to our development community?
Leave us comments via issues!

Open source a Dataset

Open sourcing a dataset can be as simple as uploading a .csv to GitHub and letting people know about it. Rather than publishing data as a zip file on your website or an FTP server, you can add the files through the GitHub.com web interface, or via the GitHub for Windows or GitHub for Mac native clients. Create a new repository to store your datasets – in many cases, it’s as easy as drag, drop, sync.
GitHub can host any file type (although open, non-binary files like .csvs tend to work best). Plus, GitHub supports rendering certain open data formats interactively such as the popular geospacial .geojson format. Once uploaded, citizens can view the files, and can even open issues or submit pull requests with proposed fixes.

Explore Open Source Civic Apps

There are many open source applications freely available on GitHub that were built just for government. Check them out, and see if it fits a need. Here are some examples:

  • Adopt-a – This open source web app was created for the City of Boston in 2011 by Code for America fellows. It allows residents to “adopt” a hydrant and make sure it’s clear of snow in the winter so that emergency crews can locate them when needed. It has since been adopted in Chicago (for sidewalks), Seattle (for storm drains), and Honolulu (for tsunami sirens).
  • StreetMix – Another creation of Code for America fellows (2013) this website, www.streetmix.net, allows anyone to create street sections in a way that is not only beautiful but educational, too. No downloading, no installing, no paying – make and save your creations right at the website. Great for internal or public community planning meetings.
  • We The PeopleWe The People, the White House’s petitions application hosted at petitions.whitehouse.gov is a Drupal module to allow citizens to submit and digitally sign petitions.

Open source something small

Chances are you’ve got something small you can open source. Check in with your web or new media team, and see if they’ve got something they’ve been dying to share or blog about, no matter how small. It can be snippet of analytics code, or maybe a small script used internally. It doesn’t even have to be code.
Post your website’s privacy policy, comment moderation policy, or terms of service and let the community weigh in before your next edit. No matter how small it is, getting your first open source project going is a great first step.

Improve an existing project

Does you agency use an existing open source project to conduct its own business? Open an issue on the project’s repository with a feature request or a bug you spot. Better yet, fork the project, and submit your improvements. Even if it’s one or two lines of code, such examples are great to blog about to showcase your efforts.
Don’t forget, this site is an open source project, too. Making an needed edit is another great way to get started.”

And Data for All: On the Validity and Usefulness of Open Government Data


Paper presented at the the 13th International Conference on Knowledge Management and Knowledge Technologies: “Open Government Data (OGD) stands for a relatively young trend to make data that is collected and maintained by state authorities available for the public. Although various Austrian OGD initiatives have been started in the last few years, less is known about the validity and the usefulness of the data offered. Based on the data-set on Vienna’s stock of trees, we address two questions in this paper. First of all, we examine the quality of the data by validating it according to knowledge from a related discipline. It shows that the data-set we used correlates with findings from meteorology. Then, we explore the usefulness and exploitability of OGD by describing a concrete scenario in which this data-set can be supportive for citizens in their everyday life and by discussing further application areas in which OGD can be beneficial for different stakeholders and even commercially used.”

Choose Your Own Route on Finland's Algorithm-Driven Public Bus


Brian Merchant at Motherboard: “Technology should probably be transforming public transit a lot faster than it is. Yes, apps like Hopstop have made finding stops easier and I’ve started riding the bus in unfamiliar parts of town a bit more often thanks to Google Maps’ route info. But these are relatively small steps, and it’s all limited to making scheduling information more widely available. Where’s the innovation on the other side? Where’s the Uber-like interactivity, the bus that comes to you after a tap on the iPhone?
In Finland, actually. The Kutsuplus is Helsinki’s groundbreaking mass transit hybrid program that lets riders choose their own routes, pay for fares on their phones, and summon their own buses. It’s a pretty interesting concept. With a ten minute lead time, you summon a Kutsuplus bus to a stop using the official app, just as you’d call a livery cab on Uber. Each minibus in the fleet seats at least nine people, and there’s room for baby carriages and bikes.
You can call your own private Kutsuplus, but if you share the ride, you share the costs—it’s about half the price of a cab fare, and a dollar or two more expensive than old school bus transit. You can then pick your own stop, also using the app.
The interesting part is the scheduling, which is entirely automated. If you’re sharing the ride, an algorithm determines the most direct route, and you only get charged as though you were riding solo. You can pay with a Kutsuplus wallet on the app, or, eventually, bill the charge to your phone bill.”

Crowdsourcing Mobile App Takes the Globe’s Economic Pulse


Tom Simonite in MIT Technology Review: “In early September, news outlets reported that the price of onions in India had suddenly spiked nearly 300 percent over prices a year before. Analysts warned that the jump in price for this food staple could signal an impending economic crisis, and the Research Bank of India quickly raised interest rates.
A startup company called Premise might’ve helped make the response to India’s onion crisis timelier. As part of a novel approach to tracking the global economy from the bottom up, the company has a daily feed of onion prices from stores around India. More than 700 people in cities around the globe use a mobile app to log the prices of key products in local stores each day.

Premise’s cofounder David Soloff says it’s a valuable way to take the pulse of economies around the world, especially since stores frequently update their prices in response to economic pressures such as wholesale costs and consumer confidence. “All this information is hiding in plain sight on store shelves,” he says, “but there’s no way of capturing and aggregating it in any meaningful way.”
That information could provide a quick way to track and even predict inflation measures such as the U.S. Consumer Price Index. Inflation figures influence the financial industry and are used to set governments’ monetary and fiscal policy, but they are typically updated only once a month. Soloff says Premise’s analyses have shown that for some economies, the data the company collects can reliably predict monthly inflation figures four to six weeks in advance. “You don’t look at the weather forecast once a month,” he says….
Premise’s data may have other uses outside the financial industry. As part of a United Nations program called Global Pulse, Cavallo and PriceStats, which was founded after financial professionals began relying on data from an ongoing academic price-indexing effort called the Billion Prices Project, devised bread price indexes for several Latin American countries. Such indexes typically predict street prices and help governments and NGOs spot emerging food crises. Premise’s data could be used in the same way. The information could also be used to monitor areas of the world, such as Africa, where tracking online prices is unreliable, he says.”

Free Software Ties the Internet of Things Together


Rachel Metz in MIT Technology Review: “OpenRemote is an open-source Internet of Things platform that could help spur smarter homes and cities.
If you buy several Internet-connected home gadgets—say, a “smart” thermostat, “smart” door lock, and “smart” window blinds—you’ll likely have to control each one with a separate app, meaning it exists in its own little silo.
That’s not how Elier Ramirez does it. In his home, an iPad app controls his lights, ceiling fans, and TV and stereo. Pressing a single button within the app can shut off all his lights and gadgets when he leaves.
Ramirez can tap a lamp in an image to turn an actual lamp off and on in his apartment, and at the same time he’ll see the picture on the tablet’s screen go dark or become illuminated. Ramirez also set up a presence-sensing feature that uses his cell phone to determine if he’s home (it checks whether or not he has connected to his home Wi-Fi network). This can automatically turn on the lights if he’s there. Ramirez runs the whole setup from a small computer in his home.
The software behind all this interconnection comes from a company called OpenRemote, which is plugging away on an open-source software platform for linking Internet-connected gadgets, making it easier to control all kinds of smart home devices, regardless of who made them. And it makes it easy to automate actions like lowering your connected window blinds if the temperature sensed in your living room goes above 75 degrees….
OpenRemote also sees a moneymaking opportunity beyond the home in providing its software to cities, which are becoming increasingly interested in using technology for everything from communicating with citizens to monitoring traffic. Last year, OpenRemote conducted a small test in Eindhoven, in hopes of using automation and crowdsourcing to monitor a city. This included people-tracking with cameras, sound-level tracking, social-media monitoring, and an app that people in the area could use to rate what the atmosphere was like. The company is currently working on a larger-scale project in Eindhoven, Kil says. “If you put four walls around a city, it’s a big room, if you know what I mean,” he says.”

Seven Principles for Big Data and Resilience Projects


PopTech & Rockefeler Bellagio Fellows: “The following is a draft “Code of Conduct” that seeks to provide guidance on best practices for resilience building projects that leverage Big Data and Advanced Computing. These seven core principles serve to guide data projects to ensure they are socially just, encourage local wealth- & skill-creation, require informed consent, and be maintainable over long timeframes. This document is a work in progress, so we very much welcome feedback. Our aim is not to enforce these principles on others but rather to hold ourselves accountable and in the process encourage others to do the same. Initial versions of this draft were written during the 2013 PopTech & Rockefeller Foundation workshop in Bellagio, August 2013.
Open Source Data Tools – Wherever possible, data analytics and manipulation tools should be open source, architecture independent and broadly prevalent (R, python, etc.). Open source, hackable tools are generative, and building generative capacity is an important element of resilience….
Transparent Data Infrastructure – Infrastructure for data collection and storage should operate based on transparent standards to maximize the number of users that can interact with the infrastructure. Data infrastructure should strive for built-in documentation, be extensive and provide easy access. Data is only as useful to the data scientist as her/his understanding of its collection is correct…
Develop and Maintain Local Skills – Make “Data Literacy” more widespread. Leverage local data labor and build on existing skills. The key and most constraint ingredient to effective data solutions remains human skill/knowledge and needs to be retained locally. In doing so, consider cultural issues and language. Catalyze the next generation of data scientists and generate new required skills in the cities where the data is being collected…
Local Data Ownership – Use Creative Commons and licenses that state that data is not to be used for commercial purposes. The community directly owns the data it generates, along with the learning algorithms (machine learning classifiers) and derivatives. Strong data protection protocols need to be in place to protect identities and personally identifying information…
Ethical Data Sharing – Adopt existing data sharing protocols like the ICRC’s (2013). Permission for sharing is essential. How the data will be used should be clearly articulated. An opt in approach should be the preference wherever possible, and the ability for individuals to remove themselves from a data set after it has been collected must always be an option. Projects should always explicitly state which third parties will get access to data, if any, so that it is clear who will be able to access and use the data…
Right Not To Be Sensed – Local communities have a right not to be sensed. Large scale city sensing projects must have a clear framework for how people are able to be involved or choose not to participate. All too often, sensing projects are established without any ethical framework or any commitment to informed consent. It is essential that the collection of any sensitive data, from social and mobile data to video and photographic records of houses, streets and individuals, is done with full public knowledge, community discussion, and the ability to opt out…
Learning from Mistakes – Big Data and Resilience projects need to be open to face, report, and discuss failures. Big Data technology is still very much in a learning phase. Failure and the learning and insights resulting from it should be accepted and appreciated. Without admitting what does not work we are not learning effectively as a community. Quality control and assessment for data-driven solutions is notably harder than comparable efforts in other technology fields. The uncertainty about quality of the solution is created by the uncertainty inherent in data…”

Five Ways to Make Government Procurement Better


Mark Headd at Civic Innovations:  “Nothing in recent memory has focused attention on the need for wholesale reform of the government IT procurement system more than the troubled launch of healthcare.gov.
There has been a myriad of blog posts, stories and articles written in the last few weeks detailing all of the problems that led to the ignominious launch of the website meant to allow people to sign up for health care coverage.
Though the details of this high profile flop are in the latest headlines, the underlying cause has been talked about many times before – the process by which governments contract with outside parties to obtain IT services is broken…
With all of this in mind, here are – in no particular order – five suggested changes that can be adopted to improve the government procurement process.
Raise the threshold on simplified / streamlined procurement
Many governments use a separate, more streamlined process for smaller projects that do not require a full RFP (in the City of Philadelphia, professional services projects that do not exceed $32,000 annually go through this more streamlined bidding process). In Philadelphia, we’ve had great success in using these smaller projects to test new ideas and strategies for partnering with IT vendors. There is much we can learn from these experiments, and a modest increase to enable more experimentation would allow governments to gain valuable new insights.
Narrowing the focus of any enhanced thresholds for streamlined budding to web-based projects would help mitigate risk and foster a quicker process for testing new ideas.
Identify clear standards for projects
Having a clear set of vendor-agnostic IT standards to use when developing RFPs and in performing work can make a huge difference in how a project turns out. Clearly articulating standards for:

  • The various components that a system will use.
  • The environment in which it will be housed.
  • The testing it must undergo prior to final acceptance.

…can go a long way to reduce the risk an uncertainly inherent in IT projects.
It’s worth noting that most governments probably already have a set of IT standards that are usually made part of any IT solicitation. But these standards documents can quickly become out of date – they must undergo constant review and refinement. In addition, many of the people writing these standards may confuse a specific vendor product or platform with a true standard.
Require open source
Requiring that IT projects be open source during development or after completion can be an effective way to reduce risk on an IT project and enhance transparency. This is particularly true of web-based projects.
In addition, government RFPs should encourage the use of existing open source tools – leveraging existing software components that are in use in similar projects and maintained by an active community – to foster external participation by vendors and volunteers alike. When governments make the code behind their project open source, they enable anyone that understands software development to help make them better.
Develop a more robust internal capacity for IT project management and implementation
Governments must find ways to develop the internal capacity for developing, implementing and managing technology projects.
Part of the reason that governments make use of a variety of different risk mitigation provisions in public bidding is that there is a lack of people in government with hands on experience building or maintaining technology. There is a dearth of makers in government, and there is a direct relationship between the perceived risk that governments take on with new technology projects and the lack of experienced technologists working in government.
Governments need to find ways to develop a maker culture within their workforces and should prioritize recruitment from the local technology and civic hacking communities.
Make contracting, lobbying and campaign contribution data public as open data
One of the more disheartening revelations to come out of the analysis of healthcare.gov implementation is that some of the firms that were awarded work as part of the project also spent non-trivial amounts of money on lobbying. It’s a good bet that this kind of thing also happens at the state and local level as well.
This can seriously undermine confidence in the bidding process, and may cause many smaller firms – who lack funds or interest in lobbying elected officials – to simply throw up their hands and walk away.
In the absence of statutory or regulatory changes to prevent this from happening, governments can enhance the transparency around the bidding process by working to ensure that all contracting data as well as data listing publicly registered lobbyists and contributions to political campaigns is open.
Ensuring that all prospective participants in the public bidding process have confidence that the process will be fair and transparent is essential to getting as many firms to participate as possible – including small firms more adept at agile software development methodologies. More bids typically equates to higher quality proposals and lower prices.
None of the changes list above will be easy, and governments are positioned differently in how well they may achieve any one of them. Nor do they represent the entire universe of things we can do to improve the system in the near term – these are items that I personally think are important and very achievable.
One thing that could help speed the adoption of these and other changes is the development of robust communication framework between government contracting and IT professionals in different cities and different states. I think a “Municipal Procurement Academy” could go a long way toward achieving this.”