Index: The Data Universe

Curated on Posted on by admin

The Living Library Index – inspired by the Harper’s Index – provides important statistics and highlights global trends in governance innovation. This installment focuses on the data universe and was originally published in 2013.

  • How much data exists in the digital universe as of 2012: 2.7 zetabytes*
  • Increase in the quantity of Internet data from 2005 to 2012: +1,696%
  • Percent of the world’s data created in the last two years: 90
  • Number of exabytes (=1 billion gigabytes) created every day in 2012: 2.5; that number doubles every month
  • Percent of the digital universe in 2005 created by the U.S. and western Europe vs. emerging markets: 48 vs. 20
  • Percent of the digital universe in 2012 created by emerging markets: 36
  • Percent of the digital universe in 2020 predicted to be created by China alone: 21
  • How much information in the digital universe is created and consumed by consumers (video, social media, photos, etc.) in 2012: 68%
  • Percent of which enterprises have liability or responsibility for (copyright, privacy, compliance with regulations, etc.): 80
  • Amount included in the Obama Administration’s 2-12 Big Data initiative: over $200 million
  • Amount the Department of Defense is investing annually on Big Data projects as of 2012: over $250 million
  • Data created per day in 2012: 2.5 quintillion bytes
  • How many terabytes* of data collected by the U.S. Library of Congress as of April 2011: 235
  • How many terabytes of data collected by Walmart per hour as of 2012: 2,560, or 2.5 petabytes*
  • Projected growth in global data generated per year, as of 2011: 40%
  • Number of IT jobs created globally by 2015 to support big data: 4.4 million (1.9 million in the U.S.)
  • Potential shortage of data scientists in the U.S. alone predicted for 2018: 140,000-190,000, in addition to 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions
  • Time needed to sequence the complete human genome (analyzing 3 billion base pairs) in 2003: ten years
  • Time needed in 2013: one week
  • The world’s annual effective capacity to exchange information through telecommunication networks in 1986, 2007, and (predicted) 2013: 281 petabytes, 65 exabytes, 667 exabytes
  • Projected amount of digital information created annually that will either live in or pass through the cloud: 1/3
  • Increase in data collection volume year-over-year in 2012: 400%
  • Increase in number of individual data collectors from 2011 to 2012: nearly double (over 300 data collection parties in 2012)

*1 zetabyte = 1 billion terabytes | 1 petabyte = 1,000 terabytes | 1 terabyte = 1,000 gigabytes | 1 gigabyte = 1 billion bytes

Sources

Civic Innovation Fellowships Go Global

Curated on Posted on by Stefaan Verhulst

Some thoughts from Panthea Lee from Reboot: “In recent years, civic innovation fellowships have shown great promise to improve the relationships between citizens and government. In the United States, Code for America and the Presidential Innovation Fellows have demonstrated the positive impact a small group of technologists can have working hand-in-hand with government. With the launch of Code for All, Code for Europe, Code4Kenya, and Code4Africa, among others, the model is going global.
But despite the increasing popularity of civic innovation fellowships, there are few templates for how a “Code for” program can be adapted to a different context. In the US, the success of Code for America has drawn from a wealth of tech talent eager to volunteer skills, public and private support, and the active participation of municipal governments. Elsewhere, new “Code for” programs are surely going to have to operate within a different set of capacities and constraints.”

Five myths about big data

Curated on Posted on by Stefaan Verhulst

Samuel Arbesman, senior scholar at the Ewing Marion Kauffman Foundation and the author of “The Half-Life of Facts” in the Washington Post: “Big data holds the promise of harnessing huge amounts of information to help us better understand the world. But when talking about big data, there’s a tendency to fall into hyperbole. It is what compels contrarians to write such tweets as “Big Data, n.: the belief that any sufficiently large pile of s— contains a pony.” Let’s deflate the hype.
1. “Big data” has a clear definition.
The term “big data” has been in circulation since at least the 1990s, when it is believed to have originated in Silicon Valley. IBM offers a seemingly simple definition: Big data is characterized by the four V’s of volume, variety, velocity and veracity. But the term is thrown around so often, in so many contexts — science, marketing, politics, sports — that its meaning has become vague and ambiguous….
2. Big data is new.
By many accounts, big data exploded onto the scene quite recently. “If wonks were fashionistas, big data would be this season’s hot new color,” a Reuters report quipped last year. In a May 2011 report, the McKinsey Global Institute declared big data “the next frontier for innovation, competition, and productivity.”
It’s true that today we can mine massive amounts of data — textual, social, scientific and otherwise — using complex algorithms and computer power. But big data has been around for a long time. It’s just that exhaustive datasets were more exhausting to compile and study in the days when “computer” meant a person who performed calculations….
3. Big data is revolutionary.
In their new book, “Big Data: A Revolution That Will Transform How We Live, Work, and Think,”Viktor Mayer-Schonberger and Kenneth Cukier compare “the current data deluge” to the transformation brought about by the Gutenberg printing press.
If you want more precise advertising directed toward you, then yes, big data is revolutionary. Generally, though, it’s likely to have a modest and gradual impact on our lives….
4. Bigger data is better.
In science, some admittedly mind-blowing big-data analyses are being done. In business, companies are being told to “embrace big data before your competitors do.” But big data is not automatically better.
Really big datasets can be a mess. Unless researchers and analysts can reduce the number of variables and make the data more manageable, they get quantity without a whole lot of quality. Give me some quality medium data over bad big data any day…
5. Big data means the end of scientific theories.
Chris Anderson argued in a 2008 Wired essay that big data renders the scientific method obsolete: Throw enough data at an advanced machine-learning technique, and all the correlations and relationships will simply jump out. We’ll understand everything.
But you can’t just go fishing for correlations and hope they will explain the world. If you’re not careful, you’ll end up with spurious correlations. Even more important, to contend with the “why” of things, we still need ideas, hypotheses and theories. If you don’t have good questions, your results can be silly and meaningless.
Having more data won’t substitute for thinking hard, recognizing anomalies and exploring deep truths.”

Announcing Project Open Data from Cloudant Labs

Curated on Posted on by Stefaan Verhulst

Yuriy Dybskiy from Cloudant: “There has been an emerging pattern over the last few years of more and more government datasets becoming available for public access. Earlier this year, the White House announced official policy on such data – Project Open Data.

Available resources

Here are four resources on the topic:

  1. Tim Berners-Lee: Open, Linked Data for a Global Community – [10 min video]
  2. Rufus Pollock: Open Data – How We Got Here and Where We’re Going – [24 min video]
  3. Open Knowledge Foundation Datasets – http://data.okfn.org/data
  4. Max Ogden: Project dat – collaborative data – [github repo]

One of the main challenges is access to the datasets. If only there were a database that had easy access to its data baked right in it.
Luckily, there is CouchDB and Cloudant, which share the same APIs to access data via HTTP. This makes for a really great option to store interesting datasets.

Cloudant Open Data

Today we are happy to announce a Cloudant Labs project – Cloudant Open Data!
Several datasets are available at the moment, for example, businesses_sf – data regarding businesses registered in San Francisco and sf_pd_incidents – a collection of incident reports (criminal and non-criminal) made by the San Francisco Police Department.
We’ll add more, but if you have one you’d like us to add faster – drop us a line at [email protected]
Create an account and play with these datasets yourself”

Internet Governance is Our Shared Responsibility

Curated on Posted on by Stefaan Verhulst

New paper by Vint Cerf, Patrick Ryan and Max Senges Senges: “This essay looks at the the different roles that multistakeholder institutions play in the Internet governance ecosystem. We propose a model for thinking of Internet governance within the context of the Internet’s layered model. We use the example of the negotiations in Dubai in 2102 at the World Conference on International Telecommunications as an illustration for why it is important for different institutions within the governance system to focus on their respective areas of expertise (e.g., the ITU, ICANN, and IGF). Several areas of conflict (a “tussle”) are reviewed, such as the desire to promote more broadband infrastructure, a topic that is in the remit of the International Telecommunications Union, but also the recurring desire of countries like Russia and China to use the ITU to regulate content and restrict free expression on the Internet through onerous cybersecurity and spam provisions. We conclude that it is folly to try and regulate all these areas through an international treaty, and encourage further development of mechanisms for global debate like the Internet Governance Forum (IGF).”

Citizen-Centered Governance: The Mayor's Office of New Urban Mechanics and the Evolution of CRM in Boston

Curated on Posted on by Stefaan Verhulst

New Paper by Susan P. Crawford and Dana Walters (Berkman Center Research Publication No. 17): “Over the last three years, the Boston Mayor’s Office of New Urban Mechanics, the innovative, collaborative ethos within City Hall fostered by Mayor Menino and his current chief of staff, Mitchell Weiss, and Boston’s launch of a CRM system and its associated Citizens Connect smartphone app have all attracted substantial media attention. In particular, the City of Boston’s strategy to put citizen engagement and participation at the center of its efforts, implemented by Chris Osgood and Nigel Jacob as co-chairs of the Mayor’s Office of New Urban Mechanics, has drawn attention to the potential power of collaboration and technology to transform citizens’ connections to their government and to each other. Several global developments have combined to make Boston’s collaborative efforts interesting: First, city managers around the world confront shrinking budgets and diminishing trust in the role of government; second, civic entrepreneurs and technology innovators are pressuring local governments to adopt new forms of engagement with citizens; and third, new digital tools are emerging that can help make city services both more visible and more effective. Boston’s experience in pursuing partnerships that facilitate opportunities for engaging citizens may provide scalable (and disruptive) lessons for other cities.

During the summer of 2013, in anticipation of Mayor Menino’s retirement in January 2014, Prof. Susan Crawford and Project Assistant Dana Walters carried out a case study examining the ongoing evolution of the Boston Mayor’s Hotline into a platform for civic engagement. We chose this CRM focus because the initial development of the system provides a concrete example of how leaders in government can connect to local partners and citizens. In the course of this research, we interviewed 21 city employees and several of their partners outside government, and gathered data about the use of the system.

We found a traditional technology story—selection and integration of CRM software, initial performance management using that software, development of ancillary channels of communication, initial patterns of adoption and use—that reflects the commitment of Mayor Menino to personalized constituent service. We also found that that commitment, his long tenure, and the particular personalities of the people on the New Urban Mechanics team make this both a cultural story as well as a technology story. Here are the highlights…”

Searching Big Data for ‘Digital Smoke Signals’

Curated on Posted on by Stefaan Verhulst

Steve Lohr in the New York Times: “It is the base camp of the United Nations Global Pulse team — a tiny unit inside an institution known for its sprawling bureaucracy, not its entrepreneurial hustle. Still, the focus is on harnessing technology in new ways — using data from social networks, blogs, cellphones and online commerce to transform economic development and humanitarian aid in poorer nations….

The efforts by Global Pulse and a growing collection of scientists at universities, companies and nonprofit groups have been given the label “Big Data for development.” It is a field of great opportunity and challenge. The goal, the scientists involved agree, is to bring real-time monitoring and prediction to development and aid programs. Projects and policies, they say, can move faster, adapt to changing circumstances and be more effective, helping to lift more communities out of poverty and even save lives.

Research by Global Pulse and other groups, for example, has found that analyzing Twitter messages can give an early warning of a spike in unemployment, price rises and disease. Such “digital smoke signals of distress,” Mr. Kirkpatrick said, usually come months before official statistics — and in many developing countries today, there are no reliable statistics.

Finding the signals requires data, though, and much of the most valuable data is held by private companies, especially mobile phone operators, whose networks carry text messages, digital-cash transactions and location data. So persuading telecommunications operators, and the governments that regulate and sometimes own them, to release some of the data is a top task for the group. To analyze the data, the groups apply tools now most widely used for pinpointing customers with online advertising.”

The New Innovation Battlegrounds Are City Hall And The State House

Curated on Posted on by Stefaan Verhulst

Julius Genachowski in Forbes: “What’s going on here? In cities and states across the country, two forces are engaged in battles with major consequences for the future of the Internet and the U.S. innovation economy.
The first force is new ventures harnessing technology—particularly the Internet and mobile—to challenge incumbents in a growing number of industries: From hotels (Airbnb) to rental cars (ZipCar, RelayRides, Car2Go) to taxis (SideCar, Lyft, Uber) to car dealerships (Tesla) to parking lots (Parking Panda) to textbooks (Chegg) to lending and fundraising (Lending Club, Kickstarter) to restaurants (food trucks) to boating (Boatband, GetMyBoat) to errand running services (TaskRabbit) to Internet service (Chattanooga, TN; Lafayette, LA; Google Fiber).
Many of these ventures are part of the new “sharing economy.” ….
The second force in these battles is city and state governments, which typically have long and deep relationships with established industries. Not surprisingly, and acting rationally from their perspective, existing businesses have been lobbying state and local officials to restrict new entrants.
And across the country, new laws are being proposed and enacted—and existing but out-of-date laws are being enforced—to protect incumbents from new Internet- and mobile-based competitors….
There are lessons here for the current battles in city halls and state houses. We suggest four simple principles for every state and local official considering regulatory decisions affecting the sharing economy and other disruptive Internet- and mobile-based businesses:

  • Stand with innovation. The benefits of innovation can be hard to appreciate fully early on, but we know from our history that innovation drives consumer benefits and economic growth. Give innovative new services the benefit of the doubt. And where there are issues to address, take a tailored, technology-neutral approach.
  • Focus on consumers. Consider the full range of benefits new services provide consumers. Most innovators and their investors are putting up their time and money because they see a gap in the market—ways in which consumers are not fully served by existing businesses. The results of a fair-minded consumer-focused analysis might be different than some first instincts. For example, Airbnb and Uber provide insurance to protect consumers; grey-market home stays and unlicensed livery cabs may not. Weigh the benefits of moving grey-market activities out of the shadows.
  • Keep an open mind. Spend the time to understand new businesses and new technologies, including by speaking with new service providers and their users. Don’t just rely on opponents’ characterizations.
  • Use the service. Before deciding to regulate an innovative service, public officials should use the service. Because they’re new, innovative services can be hard to fully appreciate without experiencing them—and using them will provide hands-on insights on their benefits as well as tailored ways to address any issues. At the FCC we launched a Technology Experience Center so that agency staff could use cutting-edge communications devices and services potentially affected by agency rules.
  • Innovation is a core competitive advantage for the U.S. and a primary driver of economic growth and job creation across the country. In today’s fast-moving global economy, capital and talent can flow anywhere. Pro-innovation policies are critical to growing jobs and investment in U.S. cities.”

Why the world’s governments are interested in creating hubs for open data

Curated on Posted on by Stefaan Verhulst

in Gigaom: “Amid the tech giants and eager startups that have camped out in East London’s trendy Shoreditch neighborhood, the Open Data Institute is the rare nonprofit on the block that talks about feel-good sorts of things like “triple-bottom line” and “social and environmental value.” …Governments everywhere are embracing the idea that open data is the right way to manage services for citizens. The U.K. has been a leader on this — just check out the simplicity of gov.uk — which is one of the reasons why ODI is U.K. born….“Open data” is open access to the data that has exploded on the scene in recent years, some of it due to the rise of our connected, digital lifestyles from the internet, sensors, GPS, and cell phones, just to name a few resources. But ODI is particularly interested in working with data sets that can have big global and societal impacts, like health, financial, environmental and government data. For example, in conjunction with startup OpenCorporates, ODI recently helped launch a data visualization about Goldman Sachs’s insanely complex corporate structure.”

International Principles on the Application of Human Rights to Communications Surveillance

Curated on Posted on by Stefaan Verhulst

Final version, 10 July 2013:  “As technologies that facilitate State surveillance of communications advance, States are failing to ensure that laws and regulations related to communications surveillance adhere to international human rights and adequately protect the rights to privacy and freedom of expression. This document attempts to explain how international human rights law applies in the current digital environment, particularly in light of the increase in and changes to communications surveillance technologies and techniques. These principles can provide civil society groups, industry, States and others with a framework to evaluate whether current or proposed surveillance laws and practices are consistent with human rights.
These principles are the outcome of a global consultation with civil society groups, industry and international experts in communications surveillance law, policy and technology.”