Open Data is a Civil Right


Yo Yoshida, Founder & CEO, Appallicious in GovTech: “As Americans, we expect a certain standardization of basic services, infrastructure and laws — no matter where we call home. When you live in Seattle and take a business trip to New York, the electric outlet in the hotel you’re staying in is always compatible with your computer charger. When you drive from San Francisco to Los Angeles, I-5 doesn’t all-of-a-sudden turn into a dirt country road because some cities won’t cover maintenance costs. If you take a 10-minute bus ride from Boston to the city of Cambridge, you know the money in your wallet is still considered legal tender.

But what if these expectations of consistency were not always a given? What if cities, counties and states had absolutely zero coordination when it came to basic services? This is what it is like for us in the open data movement. There are so many important applications and products that have been built by civic startups and concerned citizens. However, all too often these efforts are confided to city limits, and unavailable to anyone outside of them. It’s time to start reimagining the way cities function and how local governments operate. There is a wealth of information housed in local governments that should be public by default to help fuel a new wave of civic participation.
Appallicious’ Neighborhood Score provides an overall health and sustainability score, block-by-block for every neighborhood in the city of San Francisco. The first time metrics have been applied to neighborhoods so we can judge how government allocates our resources, so we can better plan how to move forward. But, if you’re thinking about moving to Oakland, just a subway stop away from San Francisco and want to see the score for a neighborhood, our app can’t help you, because that city has yet to release the data sets we need.
In Contra Costa County, there is the lifesaving PulsePoint app, which notifies smartphone users who are trained in CPR when someone nearby may be in need of help. This is an amazing app—for residents of Contra Costa County. But if someone in neighboring Alameda County needs CPR, the app, unfortunately, is completely useless.
Buildingeye visualizes planning and building permit data to allow users to see what projects are being proposed in their area or city. However, buildingeye is only available in a handful of places, simply because most cities have yet to make permits publicly available. Think about what this could do for the construction sector — an industry that has millions of jobs for Americans. Buildingeye also gives concerned citizens access to public documents like never before, so they can see what might be built in their cities or on their streets.
Along with other open data advocates, I have been going from city-to-city, county-to-county and state-to-state, trying to get governments and departments to open up their massive amounts of valuable data. Each time one city, or one county, agrees to make their data publicly accessible, I can’t help but think it’s only a drop in the bucket. We need to think bigger.
Every government, every agency and every department in the country that has already released this information to the public is a case study that points to the success of open data — and why every public entity should follow their lead. There needs to be a national referendum that instructs that all government data should be open and accessible to the public.
Last May, President Obama issued an executive order requiring that going forward, any data generated by the federal government must be made available to the public in open, machine-readable formats. In the executive order, Obama stated that, “openness in government strengthens our democracy, promotes the delivery of efficient and effective services to the public, and contributes to economic growth.”
If this is truly the case, Washington has an obligation to compel local and state governments to release their data as well. Many have tried to spur this effort. California Lt. Gov. Gavin Newsom created the Citizenville Challenge to speed up adoption on the local level. The U.S. Conference of Mayors has also been vocal in promoting open data efforts. But none of these initiatives could have the same effect of a federal mandate.
What I am proposing is no small feat, and it won’t happen overnight. But there should be a concerted effort by those in the technology industry, specifically civic startups, to call on Congress to draft legislation that would require every city in the country to make their data open, free and machine readable. Passing federal legislation will not be an easy task — but creating a “universal open data” law is possible. It would require little to no funding, and it is completely nonpartisan. It’s actually not a political issue at all; it is, for lack of a better word, and administrative issue.
Often good legislation is blocked because lawmakers and citizens are concerned about project funding. While there should be support to help cities and towns achieve the capability of opening their data, a lot of the time, they don’t need it. In 2009, the city and county of San Francisco opened up its data with zero dollars. Many other cities have done the same. There will be cities and municipalities that will need financial assistance to accomplish this. But it is worth it, and it will not require a significant investment for a substantial return. There are free online open data portals, like ckan, dkan and a new effort from Accela, CivicData.com, to centralize open data efforts.
When the UK Government recently announced a £1.5 million investment to support open data initiatives, its Cabinet Office Minister said, “We know that it creates a more accountable, efficient and effective government. Open Data is a raw material for economic growth, supporting the creation of new markets, business and jobs and helping us compete in the global race.”
We should not fall behind these efforts. There is too much at stake for our citizens, not to mention our economy. A recent McKinsey report found that making open data has the potential to create $3 trillion in value worldwide.
Former Speaker Tip O’Neil famously said, “all politics are local.” But we in the civic startup space believe all data is local. Data is reporting potholes in your neighborhood and identifying high crime areas in your communities. It’s seeing how many farmers’ markets there are in your town compared to liquor stores. Data helps predict which areas of a city are most at risk during a heat wave and other natural disasters. A federal open data law would give the raw material needed to create tools to improve the lives of all Americans, not just those who are lucky enough to live in a city that has released this information on its own.
It’s a different way of thinking about how a government operates and the relationship it has with its citizens. Open data gives the public an amazing opportunity to be more involved with governmental decisions. We can increase accountability and transparency, but most importantly we can revolutionize the way local residents communicate and work with their government.
Access to this data is a civil right. If this is truly a government by, of and for the people, then its data needs to be available to all of us. By opening up this wealth of information, we will design a better government that takes advantage of the technology and skills of civic startups and innovative citizens….”

A Framework for Benchmarking Open Government Data Efforts


DS Sayogo, TA Pardo, M Cook in the HICSS ’14 Proceedings of the 2014 47th Hawaii International Conference on System Sciences: “This paper presents a preliminary exploration on the status of open government data worldwide as well as in-depth evaluation of selected open government data portals. Using web content analysis of the open government data portals from 35 countries, this study outlines the progress of open government data efforts at the national government level. This paper also conducted in-depth evaluation of selected cases to justify the application of a proposed framework for understanding the status of open government data initiatives. This paper suggest that findings of this exploration offer a new-level of understanding of the depth, breath, and impact of current open government data efforts. The review results also point to the different stages of open government data portal development in term of data content, data manipulation capability and participatory and engagement capability. This finding suggests that development of open government portal follows an incremental approach similar to those of e-government development stages in general. Subsequently, this paper offers several observations in terms of policy and practical implication of open government data portal development drawn from the application of the proposed framework”

New Research Network to Study and Design Innovative Ways of Solving Public Problems


Network

MacArthur Foundation Research Network on Opening Governance formed to gather evidence and develop new designs for governing 

NEW YORK, NY, March 4, 2014 The Governance Lab (The GovLab) at New York University today announced the formation of a Research Network on Opening Governance, which will seek to develop blueprints for more effective and legitimate democratic institutions to help improve people’s lives.
Convened and organized by the GovLab, the MacArthur Foundation Research Network on Opening Governance is made possible by a three-year grant of $5 million from the John D. and Catherine T. MacArthur Foundation as well as a gift from Google.org, which will allow the Network to tap the latest technological advances to further its work.
Combining empirical research with real-world experiments, the Research Network will study what happens when governments and institutions open themselves to diverse participation, pursue collaborative problem-solving, and seek input and expertise from a range of people. Network members include twelve experts (see below) in computer science, political science, policy informatics, social psychology and philosophy, law, and communications. This core group is supported by an advisory network of academics, technologists, and current and former government officials. Together, they will assess existing innovations in governing and experiment with new practices and how institutions make decisions at the local, national, and international levels.
Support for the Network from Google.org will be used to build technology platforms to solve problems more openly and to run agile, real-world, empirical experiments with institutional partners such as governments and NGOs to discover what can enhance collaboration and decision-making in the public interest.
The Network’s research will be complemented by theoretical writing and compelling storytelling designed to articulate and demonstrate clearly and concretely how governing agencies might work better than they do today. “We want to arm policymakers and practitioners with evidence of what works and what does not,” says Professor Beth Simone Noveck, Network Chair and author of Wiki Government: How Technology Can Make Government Better, Democracy Stronger and Citi More Powerful, “which is vital to drive innovation, re-establish legitimacy and more effectively target scarce resources to solve today’s problems.”
“From prize-backed challenges to spur creative thinking to the use of expert networks to get the smartest people focused on a problem no matter where they work, this shift from top-down, closed, and professional government to decentralized, open, and smarter governance may be the major social innovation of the 21st century,” says Noveck. “The MacArthur Research Network on Opening Governance is the ideal crucible for helping  transition from closed and centralized to open and collaborative institutions of governance in a way that is scientifically sound and yields new insights to inform future efforts, always with an eye toward real-world impacts.”
MacArthur Foundation President Robert Gallucci added, “Recognizing that we cannot solve today’s challenges with yesterday’s tools, this interdisciplinary group will bring fresh thinking to questions about how our governing institutions operate, and how they can develop better ways to help address seemingly intractable social problems for the common good.”
Members
The MacArthur Research Network on Opening Governance comprises:
Chair: Beth Simone Noveck
Network Coordinator: Andrew Young
Chief of Research: Stefaan Verhulst
Faculty Members:

  • Sir Tim Berners-Lee (Massachusetts Institute of Technology (MIT)/University of Southampton, UK)
  • Deborah Estrin (Cornell Tech/Weill Cornell Medical College)
  • Erik Johnston (Arizona State University)
  • Henry Farrell (George Washington University)
  • Sheena S. Iyengar (Columbia Business School/Jerome A. Chazen Institute of International Business)
  • Karim Lakhani (Harvard Business School)
  • Anita McGahan (University of Toronto)
  • Cosma Shalizi (Carnegie Mellon/Santa Fe Institute)

Institutional Members:

  • Christian Bason and Jesper Christiansen (MindLab, Denmark)
  • Geoff Mulgan (National Endowment for Science Technology and the Arts – NESTA, United Kingdom)
  • Lee Rainie (Pew Research Center)

The Network is eager to hear from and engage with the public as it undertakes its work. Please contact Stefaan Verhulst to share your ideas or identify opportunities to collaborate.”

The Economics of Access to Information


Article by Mariano Mosquera at Edmond J. Safra Research Lab: “There has been an important development in the study of the right of access to public information and the so-called economics of information: by combining these two premises, it is possible to outline an economics theory of access to public information.


Moral Hazard
The legal development of the right of access to public information has been remarkable. Many international conventions, laws and national regulations have been passed on this matter. In this regard, access to information has consolidated within the framework of international human rights law.
The Inter-American Court of Human Rights was the first international court to acknowledge that access to information is a human right that is part of the right to freedom of speech. The Court recognized this right in two parts, as the individual right of any person to search for information and as a positive obligation of the state to ensure the individual’s right to receive the requested information.
This right and obligation can also be seen as the demand and supply of information.
The so-called economics of information has focused on the issue of information asymmetry between the principal and the agent. The principal (society) and the agent (state) enter into a contract.This contract is based on the idea that the agent’s specialization and professionalism (or the politician’s, according to Weber) enables him to attend to the principal’s affairs, such as public affairs in this case. This representation contract does not provide for a complete delegation,but rather it involves the principal’s commitment to monitoring the agent.
When we study corruption, it is important to note that monitoring aims to ensure that the agent adjusts its behavior to comply with the contract, in order to pursue public goals, and not to serve private interests. Stiglitz4 describes moral hazard as a situation arising from information asymmetry between the principal and the agent. The principal takes a risk when acting without comprehensive information about the agent’s actions. The moral hazard means that the handling of closed, privileged information by the agent could bring about negative consequences for the principal.
In this case, it is a risk related to corrupt practices, since a public official could use the state’s power and information to achieve private benefits, and not to resolve public issues in accordance with the principal-agent contract. This creates negative social consequences.
In this model, there are a number of safeguards against moral hazard, such as monitoring institutions (with members of the opposition) and rewards for efficient and effective administration,5 among others. Access to public information could also serve as an effective means of monitoring the agent, so that the agent adjusts its behavior to comply with the contract.
The Economic Principle of Public Information
According to this principal-agent model, public information should be defined as:
information whose social interpretation enables the state to act in the best interests of society. This definition is based on the idea of information for monitoring purposes and uses a systematic approach to feedback. This definition also implies that the state is not entirely effective at adjusting its behavior by itself.
Technically, as an economic principle of public information, public information is:
information whose interpretation by the principal is useful for the agent, so that the latter adjusts its behavior to comply with the principal-agent contract. It should be noted that this is very different from the legal definition of public information, such as “any information produced or held by the state.” This type of legal definition is focused only on supply, but not on demand.
In this principal-agent model, public information stems from two different rationales: the principal’s interpretation and the usefulness for the agent. The measure of the principal’s interpretation is the likelihood of being useful for the agent. The measure of usefulness for the agent is the likelihood of adjusting the principal-agent contract.
Another totally different situation is the development of institutions that ensure the application of this principle. For example, the channels of supplied, and demanded, information, and the channels of feedback, could be strengthened so that the social interpretation that is useful for the state actually reaches the public authorities that are able to adjust policies….”

Open Government -Opportunities and Challenges for Public Governance


New volume of Public Administration and Information Technology series: “Given this global context, and taking into account both the need of academicians and practitioners, it is the intention of this book to shed light on the open government concept and, in particular:
• To provide comprehensive knowledge of recent major developments of open government around the world.
• To analyze the importance of open government efforts for public governance.
• To provide insightful analysis about those factors that are critical when designing, implementing and evaluating open government initiatives.
• To discuss how contextual factors affect open government initiatives’success or failure.
• To explore the existence of theoretical models of open government.
• To propose strategies to move forward and to address future challenges in an international context.”

Big Data, Big New Businesses


Nigel Shaboldt and Michael Chui: “Many people have long believed that if government and the private sector agreed to share their data more freely, and allow it to be processed using the right analytics, previously unimaginable solutions to countless social, economic, and commercial problems would emerge. They may have no idea how right they are.

Even the most vocal proponents of open data appear to have underestimated how many profitable ideas and businesses stand to be created. More than 40 governments worldwide have committed to opening up their electronic data – including weather records, crime statistics, transport information, and much more – to businesses, consumers, and the general public. The McKinsey Global Institute estimates that the annual value of open data in education, transportation, consumer products, electricity, oil and gas, health care, and consumer finance could reach $3 trillion.

These benefits come in the form of new and better goods and services, as well as efficiency savings for businesses, consumers, and citizens. The range is vast. For example, drawing on data from various government agencies, the Climate Corporation (recently bought for $1 billion) has taken 30 years of weather data, 60 years of data on crop yields, and 14 terabytes of information on soil types to create customized insurance products.

Similarly, real-time traffic and transit information can be accessed on smartphone apps to inform users when the next bus is coming or how to avoid traffic congestion. And, by analyzing online comments about their products, manufacturers can identify which features consumers are most willing to pay for, and develop their business and investment strategies accordingly.

Opportunities are everywhere. A raft of open-data start-ups are now being incubated at the London-based Open Data Institute (ODI), which focuses on improving our understanding of corporate ownership, health-care delivery, energy, finance, transport, and many other areas of public interest.

Consumers are the main beneficiaries, especially in the household-goods market. It is estimated that consumers making better-informed buying decisions across sectors could capture an estimated $1.1 trillion in value annually. Third-party data aggregators are already allowing customers to compare prices across online and brick-and-mortar shops. Many also permit customers to compare quality ratings, safety data (drawn, for example, from official injury reports), information about the provenance of food, and producers’ environmental and labor practices.

Consider the book industry. Bookstores once regarded their inventory as a trade secret. Customers, competitors, and even suppliers seldom knew what stock bookstores held. Nowadays, by contrast, bookstores not only report what stock they carry but also when customers’ orders will arrive. If they did not, they would be excluded from the product-aggregation sites that have come to determine so many buying decisions.

The health-care sector is a prime target for achieving new efficiencies. By sharing the treatment data of a large patient population, for example, care providers can better identify practices that could save $180 billion annually.

The Open Data Institute-backed start-up Mastodon C uses open data on doctors’ prescriptions to differentiate among expensive patent medicines and cheaper “off-patent” varieties; when applied to just one class of drug, that could save around $400 million in one year for the British National Health Service. Meanwhile, open data on acquired infections in British hospitals has led to the publication of hospital-performance tables, a major factor in the 85% drop in reported infections.

There are also opportunities to prevent lifestyle-related diseases and improve treatment by enabling patients to compare their own data with aggregated data on similar patients. This has been shown to motivate patients to improve their diet, exercise more often, and take their medicines regularly. Similarly, letting people compare their energy use with that of their peers could prompt them to save hundreds of billions of dollars in electricity costs each year, to say nothing of reducing carbon emissions.

Such benchmarking is even more valuable for businesses seeking to improve their operational efficiency. The oil and gas industry, for example, could save $450 billion annually by sharing anonymized and aggregated data on the management of upstream and downstream facilities.

Finally, the move toward open data serves a variety of socially desirable ends, ranging from the reuse of publicly funded research to support work on poverty, inclusion, or discrimination, to the disclosure by corporations such as Nike of their supply-chain data and environmental impact.

There are, of course, challenges arising from the proliferation and systematic use of open data. Companies fear for their intellectual property; ordinary citizens worry about how their private information might be used and abused. Last year, Telefónica, the world’s fifth-largest mobile-network provider, tried to allay such fears by launching a digital confidence program to reassure customers that innovations in transparency would be implemented responsibly and without compromising users’ personal information.

The sensitive handling of these issues will be essential if we are to reap the potential $3 trillion in value that usage of open data could deliver each year. Consumers, policymakers, and companies must work together, not just to agree on common standards of analysis, but also to set the ground rules for the protection of privacy and property.”

Four Threats to American Democracy


Jared Diamond in Governance: “The U.S. government has spent the last two years wrestling with a series of crises over the federal budget and debt ceiling. I do not deny that our national debt and the prospect of a government shutdown pose real problems. But they are not our fundamental problems, although they are symptoms of them. Instead, our fundamental problems are four interconnected issues combining to threaten a breakdown of effective democratic government in the United States.
Why should we care? Let’s remind ourselves of the oft-forgotten reasons why democracy is a superior form of government (provided that it works), and hence why its deterioration is very worrisome. (Of course, I acknowledge that there are many countries in which democracy does not work, because of the lack of a national identity, of an informed electorate, or of both). The advantages of democracy include the following:

  • In a democracy, one can propose and discuss virtually any idea, even if it is initially unpalatable to the government. Debate may reveal the idea to be the best solution, whereas in a dictatorship the idea would not have gotten debated, and its virtues would not have been discovered.
  • In a democracy, citizens and their ideas get heard. Hence, without democracy, people are more likely to feel unheard and frustrated and to resort to violence.
  • Compromise is essential to a democracy. It enables us to avoid tyranny by the majority or (conversely) paralysis of government through vetoes exercised by a frustrated minority.
  • In modern democracies, all citizens can vote. Hence, government is motivated to invest in all citizens, who thereby receive the opportunity to become productive, rather than just a small dictatorial elite receiving that opportunity.

Why should we Americans keep reminding ourselves of those fundamental advantages of democracies? I would answer: not only in order to motivate ourselves to defend our democratic processes, but also because increasing numbers of Americans today are falling into the trap of envying the supposed efficiency of China’s dictatorship. Yes, it is true that dictatorships, by closing debate, can sometimes implement good policies faster than can the United States, as has China in quickly converting to lead-free gasoline and building a high-speed rail network. But dictatorships suffer from a fatal disadvantage. No one, in the 5,400 years of history of centralized government on all the continents, has figured out how to ensure that a dictatorship will embrace only good policies. Dictatorships also prevent the public debate that helps to avert catastrophic policies unparalleled in any large modern First World democracy—such as China’s quickly abolishing its educational system, sending its teachers out into the fields, and creating the world’s worst air pollution.
That is why democracy, given the prerequisites of an informed electorate and a basic sense of common interest, is the best form of government—at least, better than all the alternatives that have been tried, as Winston Churchill quipped. Our form of government is a big part of the explanation why the United States has become the richest and most powerful country in the world. Hence, an undermining of democratic processes in the United States means throwing away one of our biggest advantages. Unfortunately, that is what we are now doing, in four ways.
First, political compromise has been deteriorating in recent decades, and especially in the last five years. That deterioration can be measured as the increase in Senate rejections of presidential nominees whose approvals used to be routine, the increasing use of filibusters by the minority party, the majority party’s response of abolishing filibusters for certain types of votes, and the decline in number of laws passed by Congress to the lowest level of recent history. The reasons for this breakdown in political compromise, which seems to parallel increasing levels of nastiness in other areas of American life, remain debated. Explanations offered include the growth of television and then of the Internet, replacing face-to-face communication, and the growth of many narrowly partisan TV channels at the expense of a few broad-public channels. Even if these reasons hold a germ of truth, they leave open the question why these same trends operating in Canada and in Europe have not led to similar deterioration of political compromise in those countries as well.
Second, there are increasing restrictions on the right to vote, weighing disproportionately on voters for one party and implemented at the state level by the other party. Those obstacles include making registration to vote difficult and demanding that registered voters show documentation of citizenship when they present themselves at the polls. Of course, the United States has had a long history of denying voting rights to blacks, women, and other groups. But access to voting had been increasing in the last 50 years, so the recent proliferation of restrictions reverses that long positive trend. In addition to those obstacles preventing voter registration, the United States has by far the lowest election turnout among large First World democracies: under 60% of registered voters in most presidential elections, 40% for congressional elections, and 20% for the recent election for mayor of my city of Los Angeles. (A source of numbers for this and other comparisons that I shall cite is an excellent recent book by Howard Steven Friedman, The Measure of a Nation). And, while we are talking about elections, let’s not forget the astronomical recent increase in costs and durations of election campaigns, their funding by wealthy interests, and the shift in campaign pitches to sound bites. Those trends, unparalled in other large First World democracies, undermine the democratic prerequisite of a well-informed electorate.
A third contributor to the growing breakdown of democracy is our growing gap between rich and poor. Among our most cherished core values is our belief that the United States is a “land of opportunity,” and that we uniquely offer to our citizens the potential for rising from “rags to riches”—provided that citizens have the necessary ability and work hard. This is a myth. Income and wealth disparity in the United States (as measured by the Gini index of equality/inequality, and in other ways) is much higher in the United States than in any other large First World democracy. So is hereditary socioeconomic immobility, that is, the probability that a son’s relative income will just mirror his father’s relative income, and that sons of poor fathers will not become wealthy. Part of the reason for those depressing facts is inequality of educational opportunities. Children of rich Americans tend to receive much better educations than children of poor Americans.
That is bad for our economy, because it means that we are failing to develop a large fraction of our intellectual capital. It is also bad for our political stability, because poor parents who correctly perceive that their children are not being given the opportunity to succeed may express their resulting frustration in violence. Twice during my 47 years of residence in Los Angeles, in 1964 and 1993, frustration in poor areas of Los Angeles erupted into violence, lootings, and killings. In the 1993 riots, when police feared that rioters would spill into the wealthy suburb of Beverly Hills, all that the outnumbered police could do to protect Beverly Hills was to string yellow plastic police tape across major streets. As it turned out, the rioters did not try to invade Beverly Hills in 1993. But if present trends causing frustration continue, there will be more riots in Los Angeles and other American cities, and yellow plastic police tape will not suffice to contain the rioters.
The remaining contributor to the decline of American democracy is the decline of government investment in public purposes, such as education, infrastructure, and nonmilitary research and development. Large segments of the American populace deride government investment as “socialism.” But it is not socialism. On the contrary, it is one of the longest established functions of government. Ever since the rise of the first governments 5,400 years ago, governments have served two main functions: to maintain internal peace by monopolizing force, settling disputes, and forbidding citizens to resort to violence in order to settle disputes themselves; and to redistribute individual wealth for investing in larger aims—in the worst cases, enriching the elite; in the best cases, promoting the good of society as a whole. Of course, some investment is private, by wealthy individuals and companies expecting to profit from their investments. But many potential payoffs cannot attract private investment, either because the payoff is so far off in the future (such as the payoff from universal primary school education), or because the payoff is diffused over all of society rather than concentrated in areas profitable to the private investor (such as diffused benefits of municipal fire departments, roads, and broad education). Even the most passionate American supporters of small government do not decry as socialism the funding of fire departments, interstate highways, and public schools.

Can Twitter Predict Major Events Such As Mass Protests?


Emerging Technology From the arXiv : “The idea that social media sites such as Twitter can predict the future has a controversial history. In the last few years, various groups have claimed to be able to predict everything from the outcome of elections to the box office takings for new movies.
It’s fair to say that these claims have generated their fair share of criticism. So it’s interesting to see a new claim come to light.
Today, Nathan Kallus at the Massachusetts Institute of Technology in Cambridge says he has developed a way to predict crowd behaviour using statements made on Twitter. In particular, he has analysed the tweets associated with the 2013 coup d’état in Egypt and says that the civil unrest associated with this event was clearly predictable days in advance.
It’s not hard to imagine how the future behaviour of crowds might be embedded in the Twitter stream. People often signal their intent to meet in advance and even coordinate their behaviour using social media. So this social media activity is a leading indicator of future crowd behaviour.
That makes it seem clear that predicting future crowd behaviour is simply a matter of picking this leading indicator out of the noise.
Kallus says this is possible by mining tweets for any mention of future events and then analysing trends associated with them. “The gathering of crowds into a single action can often be seen through trends appearing in this data far in advance,” he says.
It turns out that exactly this kind of analysis is available from a company called Recorded Future based in Cambridge, which scans 300,000 different web sources in seven different languages from all over the world. It then extracts mentions of future events for later analysis….
The bigger question is whether it’s possible to pick out this evidence in advance. In other words, is possible to make predictions before the events actually occur?
That’s not so clear but there are good reasons to be cautious. First of all, while it’s possible to correlate Twitter activity to real protests, it’s also necessary to rule out false positives. There may be significant Twitter trends that do not lead to significant protests in the streets. Kallus does not adequately address the question of how to tell these things apart.
Then there is the question of whether tweets are trustworthy. It’s not hard to imagine that when it comes to issues of great national consequence, propaganda, rumor and irony may play a significant role. So how to deal with this?
There is also the question of demographics and whether tweets truly represent the intentions and activity of the population as a whole. People who tweet are overwhelmingly likely to be young but there is another silent majority that plays hugely important role. So can the Twitter firehose really represent the intentions of this part of the population too?
The final challenge is in the nature of prediction. If the Twitter feed is predictive, then what’s needed is evidence that it can be used to make real predictions about the future and not just historical predictions about the past.
We’ve looked at some of these problems with the predictive power of social media before and the challenge is clear: if there is a claim to be able to predict the future, then this claim must be accompanied by convincing evidence of an actual prediction about an event before it happens.
Until then, it would surely be wise to be circumspect about the predictive powers of Twitter and other forms of social media.
Ref: arxiv.org/abs/1402.2308: Predicting Crowd Behavior with Big Public Data”

Developing an open government plan in the open


Tim Hughes at OGP: “New laws, standards, policies, processes and technologies are critical for opening up government, but arguably just as (if not more) important are new cultures, behaviours and ways of working within government and civil society.
The development of an OGP National Action Plan, therefore, presents a twofold opportunity for opening up government: On the one hand it should be used to deliver a set of robust and ambitious commitments to greater transparency, participation and accountability. But just as importantly, the process of developing a NAP should also be used to model new forms of open and collaborative working within government and civil society. These two purposes of a NAP should be mutually reinforcing. An open and collaborative process can – as was the case in the UK – help to deliver a more robust and ambitious action plan, which in turn can demonstrate the efficacy of working in the open.
You could even go one step further to say that the development of an National Action Plan should present an (almost) “ideal” vision of what open government in a country could look like. If governments aren’t being open as they’re developing an open government action plan, then there’s arguably little hope that they’ll be open elsewhere.
As coordinators of the UK OGP civil society network, this was on our mind at the beginning and throughout the development of the UK’s 2013-15 National Action Plan. Crucially, it was also on the minds of our counterparts in the UK Government. From the start, therefore, the process was developed with the intention that it should itself model the principles of open government. Members of the UK OGP civil society network met with policy officials from the UK Government on a regular basis to scope out and develop the action plan, and we published regular updates of our discussions and progress for others to follow and engage with. The process wasn’t without its challenges – and there’s still much more we can do to open it up further in the future – but it was successful in moving far beyond the typical model of government deciding, announcing and defending its intentions and in delivering an action plan with some strong and ambitious commitments.
One of the benefits of working in an open and collaborative way is that it enabled us to conduct and publish a full – warts and all – review of what went well and what didn’t. So, consider this is an invitation to delve into our successes and failures, a challenge to do it better and a request to help us to do so too. Head over to the UK OGP civil society network blog to read about what we did, and tell us what you think: http://www.opengovernment.org.uk/national-action-plan/story-of-the-uk-national-action-plan-2013-15/

The newsonomics of measuring the real impact of news


Ken Doctor at Nieman Journalism Lab: “Hello there! It’s me, your friendly neighborhood Tweet Button. What if you could tap me and unlock a brand new source of funding for startup news sources of all kinds? What if, even better, you the reader could tap that money loose with a single click?
That’s the delightfully simple conceit behind a little widget, Impaq.me, you may have seen popping up as you traverse the news web. It’s social. It’s viral. It uses OPM (Other People’s Money) — and maybe a little bit of your own. It makes a new case to funders and maybe commercial sponsors. And it spits out metrics around the clock. It aims to be a convergence widget, acting on that now-aging idea that our attention is as important as our wallet. Consider it a new digital Swiss Army knife for the attention economy. TWEET
It’s impossible to tell how much of an impact Impaq.me may have. It’s still in its second round of testing at six of the U.S.’s most successful independent nonprofit startups — MinnPost, Center for Investigative Reporting, The Texas Tribune, Voice of San Diego, ProPublica, and the Center for Public Integrity — but as in all things digital, timing is everything. And that timing seems right.
First, let’s consider that spate of new news sites that have sprouted with the winter rains — Bill Keller’s and Neil Barsky’s Marshall Project being only the latest. It’s been quite a run — from Ezra Klein’s Project X to Pierre Omidyar’s First Look (and just launched The Intercept) to the reimagining of FiveThirtyEight. While they encompass a broad range of business models and goals (“The newsonomics of why everyone seems to be starting a news site”), they all need two things: money and engagement. Or, maybe better ordered, engagement and money. The dance between the two is still in the early stages of Internet choreography. Get the sequences right and you win.
Second, and related, is the big question of “social” and how our sharing of news is changing the old publishing dynamic of editors deciding what we’re going to read. Just this week, two pieces here at the Lab — one on Upworthy’s influence and one on the social/search tango — highlighted the still-being-understood role of social in our news-reading lives.
Third, funders of news sites, especially Knight and other lead foundations, are looking for harder evidence of the value generated by their early grants. Millions have been poured into creating new news sites. Now they’re asking: What has our funding really done? Within that big question, Impaq.me is only one of several new attempts to demonstrably measure real impact in new ways. We’ll take a brief look at those impact initiatives below….
If Impaq.me is all about impact and money, then it’s got good company. There are at least two other noteworthy impact-measuring projects going on.

  • The Center for Investigative Reporting’s Impact Tracker effort impact-tracking initiative launched last fall. The big idea: getting beyond the traditional metrics like unique visitors and pageviews to track the value of investigative and enterprise work. To that end, CIR has hired Lindsay Green-Barber, a CUNY-trained social scientist, and given her a perhaps first-ever title: media impact analyst.We can see the fruits of the work around CIR’s impressive Returning Home to Battle veterans series. On that series, CIR is tracking such impacts as change and rise in the public discourse around veterans’ issues and related allocation of government resources. The notion of good journalism intended to shine a light in dark places has been embedded in the CIR DNA for a long time; this new effort is intended to provide data — and words — to describe progress toward solutions. CIR is working with The Seattle Times on the impact of that paper’s education reporting, and CIR may soon look at more partnerships as well. Related: CIR is holding two “Dissection” events in New York and Washington in April, bringing together journalists, funders, and social scientists to widen the media impact movement.
  • Chalkbeat, a growing national education news site, too, is moving on impact analysis. It’s called MORI (Measures of our Reporting’s Influence), and it’s a WordPress plugin. Says Chalkbeat cofounder Elizabeth Green: “We built MORI to solve for a problem that I guess you could call ‘impact loss.’ We knew that our stories were having all kinds of impacts, but we had no way of keeping track of these impacts or making sense of them. That meant that we couldn’t easily compile what we had done in the last year to share with the outside world (board, donors, foundations, readers, our moms) but also — just as important — we couldn’t look back on what we’d done and learn from it.”Sound familiar?
    After much inquiry, Chalkbeat settled on technology. “Within each story’s back end,” Green said, “we can enter inputs — qualitative data about the type of story, topic, and target audience — as well as outcomes — impacts on policy and practice (what we call ‘informed action’) as well as impacts on what we call ‘civic deliberation.’”