Zillow is a prime example of how open data creates economic value. The Seattle-based company has grown rapidly since its launch in 2006, generating more than $78 million in revenue in its last financial quarter and employing more than 500 workers. But real estate firms aren’t the only businesses benefiting from data collected and published by government.
GovLab, a research laboratory run by New York University, publishes the Open Data 500, a list of companies that benefit from open data produced by the federal government. The list contains more than 15 categories of businesses, ranging from health care and education to energy, finance, legal and the environment. And the data flows from all the major agencies, including NASA, Defense, Transportation, Homeland Security and Labor….
Zillow’s road to success underscores the challenges that lie ahead if local government is going to grab its share of open data’s economic bonanza. One of the company’s biggest hurdles was to create a system that could integrate government data from thousands of databases in county government. “There’s no standard format, which is very frustrating,” Stan Humphries, Zillow’s chief economist, told Computerworld.com. “It’s up to us to figure out 3,000 different ways to ingest data and make sense of it…. More at GovTech”
The Free 'Big Data' Sources Everyone Should Know
Bernard Marr at Linkedin Pulse: “…The moves by companies and governments to put large amounts of information into the public domain have made large volumes of data accessible to everyone….here’s my rundown of some of the best free big data sources available today.
Data.gov
The US Government pledged last year to make all government data available freely online. This site is the first stage and acts as a portal to all sorts of amazing information on everything from climate to crime. To check it out, click here.
US Census Bureau
A wealth of information on the lives of US citizens covering population data, geographic data and education. To check it out, click here. To check it out, click here.
European Union Open Data Portal
As the above, but based on data from European Union institutions. To check it out, click here.
Data.gov.uk
Data from the UK Government, including the British National Bibliography – metadata on all UK books and publications since 1950. To check it out, click here.
The CIA World Factbook
Information on history, population, economy, government, infrastructure and military of 267 countries. To check it out, click here.
Healthdata.gov
125 years of US healthcare data including claim-level Medicare data, epidemiology and population statistics. To check it out, click here.
NHS Health and Social Care Information Centre
Health data sets from the UK National Health Service. To check it out, click here.
Amazon Web Services public datasets
Huge resource of public data, including the 1000 Genome Project, an attempt to build the most comprehensive database of human genetic information and NASA’s database of satellite imagery of Earth. To check it out, click here.
Facebook Graph
Although much of the information on users’ Facebook profile is private, a lot isn’t – Facebook provide the Graph API as a way of querying the huge amount of information that its users are happy to share with the world (or can’t hide because they haven’t worked out how the privacy settings work). To check it out, click here.
Gapminder
Compilation of data from sources including the World Health Organization and World Bank covering economic, medical and social statistics from around the world. To check it out, click here.
Google Trends
Statistics on search volume (as a proportion of total search) for any given term, since 2004. To check it out, click here.
Google Finance
40 years’ worth of stock market data, updated in real time. To check it out, click here.
Google Books Ngrams
Search and analyze the full text of any of the millions of books digitised as part of the Google Books project. To check it out, click here.
National Climatic Data Center
Huge collection of environmental, meteorological and climate data sets from the US National Climatic Data Center. The world’s largest archive of weather data. To check it out, click here.
DBPedia
Wikipedia is comprised of millions of pieces of data, structured and unstructured on every subject under the sun. DBPedia is an ambitious project to catalogue and create a public, freely distributable database allowing anyone to analyze this data. To check it out, click here.
Topsy
Free, comprehensive social media data is hard to come by – after all their data is what generates profits for the big players (Facebook, Twitter etc) so they don’t want to give it away. However Topsy provides a searchable database of public tweets going back to 2006 as well as several tools to analyze the conversations. To check it out, click here.
Likebutton
Mines Facebook’s public data – globally and from your own network – to give an overview of what people “Like” at the moment. To check it out, click here.
New York Times
Searchable, indexed archive of news articles going back to 1851. To check it out, click here.
Freebase
A community-compiled database of structured data about people, places and things, with over 45 million entries. To check it out, click here.
Million Song Data Set
Metadata on over a million songs and pieces of music. Part of Amazon Web Services. To check it out, click here.”
See also Bernard Marr‘s blog at Big Data Guru
Opening Government: Designing Open Innovation Processes to Collaborate With External Problem Solvers
New paper by Ines Mergel in Social Science Computer Review: “Open government initiatives in the U.S. government focus on three main aspects: transparency, participation, and collaboration. Especially the collaboration mandate is relatively unexplored in the literature. In practice, government organizations recognize the need to include external problem solvers into their internal innovation creation processes. This is partly derived from a sense of urgency to improve the efficiency and quality of government service delivery. Another formal driver is the America Competes Act that instructs agencies to search for opportunities to meaningfully promote excellence in technology, education, and science. Government agencies are responding to these requirements by using open innovation (OI) approaches to invite citizens to crowdsource and peer produce solutions to public management problems. These distributed innovation processes occur at all levels of the U.S. government and it is important to understand what design elements are used to create innovative public management ideas. This article systematically reviews existing government crowdsourcing and peer production initiatives and shows that after agencies have defined their public management problem, they go through four different phases of the OI process: (1) idea generation through crowdsourcing, (2) incubation of submitted ideas with peer voting and collaborative improvements of favorite solutions, (3) validation with a proof of concept of implementation possibilities, and (4) reveal of the selected solution and the (internal) implementation of the winning idea. Participation and engagement are incentivized both with monetary and nonmonetary rewards, which lead to tangible solutions as well as intangible innovation outcomes, such as increased public awareness.”
Big video data could change how we do everything — from catching bad guys to tracking shoppers
Sean Varah at VentureBeat: “Everyone takes pictures and video with their devices. Parents record their kids’ soccer games, companies record employee training, police surveillance cameras at busy intersections run 24/7, and drones monitor pipelines in the desert.
With vast amounts of video growing vaster at a rate faster than the day before, and the hottest devices like drones decreasing in price and size until everyone has one (OK, not in their pocket quite yet) it’s time to start talking about mining this mass of valuable video data for useful purposes.
Julian Mann, the cofounder of Skybox Imaging — a company in the business of commercial satellite imagery and the developer advocate for Google Earth outreach — says that the new “Skybox for Good” program will provide “a constantly updated model of change of the entire planet” with the potential to “save lives, protect the environment, promote education, and positively impact humanity.”…
Mining video data through “man + machine” artificial intelligence is new technology in search of unsolved problems. Could this be the next chapter in the ever-evolving technology revolution?
For the past 50 years, satellite imagery has only been available to the U.S. intelligence community and those countries with technology to launch their own. Digital Globe was one of the first companies to make satellite imagery available commercially, and now Skybox and a few others have joined them. Drones are even newer, having been used by the U.S. military since the ‘90s for surveillance over battlefields or, in this age of counter-terrorism, playing the role of aerial detectives finding bad guys in the middle of nowhere. Before drones, the same tasks required thousands of troops on the ground, putting many young men and women in harm’s way. Today, hundreds of trained “eyes” safely located here in the U.S. watch hours of video from a single drone to assess current situations in countries far away….”
Bringing the data revolution to education, and education to the data revolution
Pauline Rose at Post2015.org: “Calls for a data revolution are putting the spotlight on the importance of more and better data as a means to hold policymakers to account for post-2015 goals. In many ways, education has been at the forefront of approaches to measuring progress over the past 15 years. The influence of the Education for All Global Monitoring Report (GMR) and the efforts of the UNESCO Institute for Statistics (UIS) in improving the availability of education data provide important lessons for tracking progress post-2015. This experience should play an important contribution to informing the practical next steps for the data revolution.
Building on this experience, a roundtable held at the Overseas Development Institute on 17 November brought together over 40 technical experts, who debated approaches to measuring progress towards post-2015 education targets, with a focus on learning and equity. The meeting coincided with the launch of consultation on post-2015 education indicators by the Technical Advisory Group (TAG) to the EFA Steering Committee. As noted in the opening remarks on the data revolution by Neil Jackson, Chief Statistician at DFID, in many ways the education sector is leading the way in thinking about how to monitor post-2015 progress in concrete ways.
One of the problems that the GMR and UIS faced in tracking progress over the past 15 years was that indicators were not set at the time of deciding on education for all goals in 2000, hence the importance of the current consultation process. Another was that data have not been available a sufficiently disaggregated form to track progress on the most disadvantaged subgroups within each country, that is those most likely to be left behind. The GMR’s World Inequality Database on Education (WIDE), drawing on internationally-comparable household survey data, has been one step forward in presenting data in an accessible format to show that the poorest children living in rural areas, and often girls, are still far from completing primary school in many countries, and that many are also not learning the basics in reading and mathematics even if they have spent time in school….”
Show Me the Evidence
New book by Ron Haskins: “This book tells the story of how the Obama administration planned and enacted several initiatives to fund social programs based on rigorous evidence of success and thereby created a fundamental change in the role of evidence in federal policymaking.
Using interviews with the major players from the White House, the Office of Management and Budget, federal agencies, Congress, and the child advocacy community, the authors detail the development and implementation of six evidence-based social policy initiatives by the Obama administration.
The initiatives range widely over fundamental issues in the nation’s social policy including preschool and K-12 education, teen pregnancy, employment and training, health, and community-based programs. These initiatives constitute a revolution in the use of social science evidence to guide federal policymaking and the operation of federal grant programs.
A fascinating story for everyone interested in politics and policy, this book also provides a blueprint for policymakers worldwide who are interested in expanding the use of evidence in policy.
Read David Wessel’s review in the Wall Street Journal.”
White House: Help Shape Public Participation
at the White House Blog: “Public participation — where citizens help shape and implement government programs — is a foundation of open, transparent, and engaging government services. From emergency management and regulatory development to science and education, better and more meaningful engagement with those who use public services can measurably improve government for everyone.
A team across the government is now working side-by-side with civil society organizations to deliver the first U.S. Public Participation Playbook, dedicated to providing best practices for how agencies can better design public participation programs, and suggested performance metrics for evaluating their effectiveness.
Developing a U.S. Public Participation Playbook has been an open government priority, and was included in both the first and second U.S. Open Government National Action Plans as part of the United States effort to increase public integrity in government programs. This resource reflects the commitment of the government and civic partners to measurably improve participation programs, and is designed using the same inclusive principles that it champions.
More than 30 Federal leaders from across diverse missions in public service have collaborated on draft best practices, or “plays,” lead by the General Services Administration’s inter-agency SocialGov Community. The playbook is not limited to digital participation, and is designed to address needs from the full spectrum of public participation programs.
The plays are structured to provide best practices, tangible examples, and suggested performance metrics for government activities that already exist or are under development. Some categories included in the plays include encouraging community development and outreach, empowering participants through public/private partnerships, using data to drive decisions, and designing for inclusiveness and accessibility.
In developing this new resource, the team has been reaching out to more than a dozen civil society organizations and stakeholders, asking them to contribute as the Playbook is created. The team would like your input as well! Over the next month, contribute your ideas to the playbook using Madison, an easy-to-use, open source platform that allows for accountable review of each contribution.
Through this process, the team will work together to ensure that the Playbook reflects the best ideas and examples for agencies to use in developing and implementing their programs with public participation in mind. This resource will be a living document, and stakeholders from inside or outside of government should continually offer new insights — whether new plays, the latest case studies, or the most current performance metrics — to the playbook.
We look forward to seeing the public participate in the creation and evolution of the Public Participation Playbook!”
Look to Government—Yes, Government—for New Social Innovations
Paper by Christian Bason and Philip Colligan: “If asked to identify the hotbed of social innovation right now, many people would likely point to the new philanthropy of Silicon Valley or the social entrepreneurship efforts supported by Ashoka, Echoing Green, and Skoll Foundation. Very few people, if any, would mention their state capital or Capitol Hill. While local and national governments may have promulgated some of the greatest advances in human history — from public education to putting a man on the moon — public bureaucracies are more commonly known to stifle innovation.
Yet, around the world, there are local, regional, and national government innovators who are challenging this paradigm. They are pioneering a new form of experimental government — bringing new knowledge and practices to the craft of governing and policy making; drawing on human-centered design, user engagement, open innovation, and cross-sector collaboration; and using data, evidence, and insights in new ways.
Earlier this year, Nesta, the UK’s innovation foundation (which Philip helps run), teamed up with Bloomberg Philanthropies to publish i-teams, the first global review of public innovation teams set up by national and city governments. The study profiled 20 of the most established i-teams from around the world, including:
- French Experimental Fund for Youth, which has supported more than 554 experimental projects (such as one that reduces school drop-out rates) that have benefited over 480,000 young people;
- Nesta’s Innovation Lab, which has run 70 open innovation challenges and programs supporting over 750 innovators working in fields as diverse as energy efficiency, healthcare, and digital education;
- New Orleans’ Innovation and Delivery team, which achieved a 19% reduction in the number of murders in the city in 2013 compared to the previous year.
How are i-teams achieving these results? The most effective ones are explicit about the goal they seek – be it creating a solution to a specific policy challenge, engaging citizenry in behaviors that help the commonweal, or transforming the way government behaves. Importantly, these teams are also able to deploy the right skills, capabilities, and methods for the job.
In addition, i-teams have a strong bias toward action. They apply academic research in behavioral economics and psychology to public policy and services, focusing on rapid experimentation and iteration. The approach stands in stark contrast to the normal routines of government.
Take for example, The UK’s Behavioural Insights Team (BIT), often called the Nudge Unit. It sets clear goals, engages the right expertise to prototype means to the end, and tests innovations rapidly in the field, to learn what’s not working and rapidly scales what is.
One of BIT’s most famous projects changed taxpayer behavior. BIT’s team of economists, behavioral psychologists, and seasoned government staffers came up with minor changes to tax letters, sent out by the UK Government, that subtlety introduced positive peer pressure. By simply altering the letters to say that most people in their local area had already paid their taxes, BIT was able to boost repayment rates by around 5%. This trial was part of a range of interventions, which have helped forward over £200 million in additional tax revenue to HM Revenue & Customs, the UK’s tax authority.
The Danish government’s internal i-team, MindLab (which Christian ran for 8 years) has likewise influenced citizen behavior….”
USDA Opens VIVO Research Networking Tool to Public
Sharon Durham at the USDA: “VIVO, a Web application used internally by U.S. Department of Agriculture (USDA) scientists since 2012 to allow better national networking across disciplines and locations, is now available to the public. USDA VIVO will be a “one-stop shop” for Federal agriculture expertise and research outcomes.”USDA employs over 5,000 researchers to ensure our programs are based on sound public policy and the best available science,” said USDA Chief Scientist and Undersecretary for Research, Education, and Economics Dr. Catherine Woteki. “USDA VIVO provides a powerful Web search tool for connecting interdisciplinary researchers, research projects and outcomes with others who might bring a different approach or scope to a research project. Inviting private citizens to use the system will increase the potential for collaboration to solve food- and agriculture-related problems.”
The idea behind USDA VIVO is to link researchers with peers and potential collaborators to ignite synergy among our nation’s best scientific minds and to spark unique approaches to some of our toughest agricultural problems. This efficient networking tool enables scientists to easily locate others with a particular expertise. VIVO also makes it possible to quickly identify scientific expertise and respond to emerging agricultural issues, like specific plant and animal disease or pests.
USDA’s Agricultural Research Service (ARS), Economic Research Service, National Institute of Food and Agriculture, National Agricultural Statistics Service and Forest Service are the first five USDA agencies to participate in VIVO. The National Agricultural Library, which is part of ARS, will host the Web application. USDA hopes to add other agencies in the future.
VIVO was in part developed under a $12.2 million grant from the National Center for Research Resources, part of the National Institutes of Health (NIH). The grant, made under the 2009 American Recovery and Reinvestment Act, was provided to the University of Florida and collaborators at Cornell University, Indiana University, Weill Cornell Medical College, Washington University in St. Louis, the Scripps Research Institute and the Ponce School of Medicine.
VIVO’s underlying database draws information about research being conducted by USDA scientists from official public systems of record and then makes it uniformly available for searching. The data can then be easily leveraged in other applications. In this way, USDA is also making its research projects and related impacts available to the Federal RePORTER tool, released by NIH on September 22, 2014. Federal RePORTER is part of a collaborative effort between Federal entities and other research institutions to create a repository that will be useful to assess the impact of Federal research and development investments.”
The idea behind USDA VIVO is to link researchers with peers and potential collaborators to ignite synergy among our nation’s best scientific minds and to spark unique approaches to some of our toughest agricultural problems. This efficient networking tool enables scientists to easily locate others with a particular expertise. VIVO also makes it possible to quickly identify scientific expertise and respond to emerging agricultural issues, like specific plant and animal disease or pests.
USDA’s Agricultural Research Service (ARS), Economic Research Service, National Institute of Food and Agriculture, National Agricultural Statistics Service and Forest Service are the first five USDA agencies to participate in VIVO. The National Agricultural Library, which is part of ARS, will host the Web application. USDA hopes to add other agencies in the future.
VIVO was in part developed under a $12.2 million grant from the National Center for Research Resources, part of the National Institutes of Health (NIH). The grant, made under the 2009 American Recovery and Reinvestment Act, was provided to the University of Florida and collaborators at Cornell University, Indiana University, Weill Cornell Medical College, Washington University in St. Louis, the Scripps Research Institute and the Ponce School of Medicine.
VIVO’s underlying database draws information about research being conducted by USDA scientists from official public systems of record and then makes it uniformly available for searching. The data can then be easily leveraged in other applications. In this way, USDA is also making its research projects and related impacts available to the Federal RePORTER tool, released by NIH on September 22, 2014. Federal RePORTER is part of a collaborative effort between Federal entities and other research institutions to create a repository that will be useful to assess the impact of Federal research and development investments.”
Digital Sociology
New book by Deborah Lupton: “We now live in a digital society. New digital technologies have had a profound influence on everyday life, social relations, government, commerce, the economy and the production and dissemination of knowledge. People’s movements in space, their purchasing habits and their online communication with others are now monitored in detail by digital technologies. We are increasingly becoming digital data subjects, whether we like it or not, and whether we choose this or not.
The sub-discipline of digital sociology provides a means by which the impact, development and use of these technologies and their incorporation into social worlds, social institutions and concepts of selfhood and embodiment may be investigated, analysed and understood. This book introduces a range of interesting social, cultural and political dimensions of digital society and discusses some of the important debates occurring in research and scholarship on these aspects. It covers the new knowledge economy and big data, reconceptualising research in the digital era, the digitisation of higher education, the diversity of digital use, digital politics and citizen digital engagement, the politics of surveillance, privacy issues, the contribution of digital devices to embodiment and concepts of selfhood and many other topics.”