Recent progress in Open Data production and consumption


Examples from a Governmental institute (SMHI) and a collaborative EU research project (SWITCH-ON) by Arheimer, Berit; and Falkenroth, Esa: “The Swedish Meteorological and Hydrological Institute (SMHI) has a long tradition both in producing and consuming open data on a national, European and global scale. It is also promoting community building among water scientists in Europe by participating in and initiating collaborative projects. This presentation will exemplify the contemporary European movement imposed by the INSPIRE directive and the Open Data Strategy, by showing the progress in openness and shift in attitudes during the last decade when handling Research Data and Public Sector Information at a national European institute. Moreover, the presentation will inform about a recently started collaborative project (EU FP7 project No 603587) coordinated by SMHI and called SWITCH-ON http://water-switch-on.eu/. The project addresses water concerns and currently untapped potential of open data for improved water management across the EU. The overall goal of the project is to make use of open data, and add value to society by repurposing and refining data from various sources. SWITCH-ON will establish new forms of water research and facilitate the development of new products and services based on principles of sharing and community building in the water society. The SWITCH-ON objectives are to use open data for implementing: 1) an innovative spatial information platform with open data tailored for direct water assessments, 2) an entirely new form of collaborative research for water-related sciences, 3) fourteen new operational products and services dedicated to appointed end-users, 4) new business and knowledge to inform individual and collective decisions in line with the Europe’s smart growth and environmental objectives. The presentation will discuss challenges, progress and opportunities with the open data strategy, based on the experiences from working both at a Governmental institute and being part of the global research community.”

Indonesian techies crowdsource election results


Ben Bland in the Financial Times: “Three Indonesian tech experts say they have used crowdsourcing to calculate an accurate result for the country’s contested presidential election in six days, while 4m officials have been beavering away for nearly two weeks counting the votes by hand.

The Indonesian techies, who work for multinational companies, were spurred into action after both presidential candidates claimed victory and accused each other of trying to rig the convoluted counting process, raising fears that the country’s young democracy was under threat.

“We did this to prevent the nation being ripped apart because of two claims to victory that nobody can verify,” said Ainun Najib, who is based in Singapore. “This solution was only possible because all the polling station data were openly available for public scrutiny and verification.”

Mr Najib and two friends took advantage of the decision by the national election commission (KPU) to upload the individual results from Indonesia’s 480,000 polling stations to its website for the first time, in an attempt to counter widespread fears about electoral fraud.

The three Indonesians scraped the voting data from the KPU website on to a database and then recruited 700 friends and acquaintances through Facebook to type in the results and check them. They uploaded the data to a website called kawalpemilu.org, which means “guard the election” in Indonesian.

Throughout the process, Mr Najib said he had to fend off hacking attacks, forcing him to shift data storage to a cloud-based service. The whole exercise cost $10 for a domain name and $0.10 for the data storage….”

The Skeleton Crew


Book Review by Edward Jay Epstein in the Wall Street Journal: “…Even in an age when we are tracked electronically by our phone companies at every single moment, about 4,000 unidentified corpses turn up in the U.S. every year, of which about half have been murdered. In 2007 no fewer than 13,500 sets of unidentified human remains were languishing in the evidence rooms of medical examiners, according to an analysis published in the National Institute of Justice Journal.
In her brilliant book “The Skeleton Crew,” Deborah Halber explains why local law enforcement often fails to investigate such deaths:”Unidentified corpses are like obtuse, financially strapped houseguests: they turn up uninvited, take up space reserved for more obliging visitors, require care and attention, and then, when you are ready for them to move on, they don’t have anywhere to go.” The result is that many of these remains are consigned to oblivion.
While the population of the anonymous dead receives only scant attention from the police or the media, it has given rise to a macabre subculture of Internet sleuthing. Ms. Halber chronicles with lucidity and wit how amateur investigators troll websites, such as the Doe Network, Official Cold Case Investigations and Websleuths Crime Sleuthing Community, and check online databases looking for matches between the reported missing and the unidentified dead. It is a grisly pursuit involving linking the images of dead bodies to the descriptions posted by people trying to find someone.
Ms. Halber devotes most of “The Skeleton Crew” to describing a handful of cases that have given rise to this bizarre avocation….”

European Commission encourages re-use of public sector data


Press Release: “Today, the European Commission is publishing guidelines to help Member States benefit from the revised Directive on the re-use of public sector information (PSI Directive). These guidelines explain for example how to give access to weather data, traffic data, property asset data and maps. Open data can be used as the basis for innovative value-added services and products, such as mobile apps, which encourage investment in data-driven sectors. The guidelines published today are based on a detailed consultation and cover issues such as:

  1. Licencing: guidelines on when public bodies can allow the re-use of documents without conditions or licences; gives conditions under which the re-use of personal data is possible. For example:

  • Public sector bodies should not impose licences when a simple notice is sufficient;

  • Open licences available on the web, such as several “Creative Commons” licences can facilitate the re-use of public sector data without the need to develop custom-made licences;

  • Attribution requirement is sufficient in most cases of PSI re-use.

  1. Datasets: presents five thematic dataset categories that businesses and other potential re-users are mostly interested in and could thus be given priority for being made available for re-use. For example:

  • Postcodes, national and local maps;

  • Weather, land and water quality, energy consumption, emission levels and other environmental and earth data;

  • Transport data: public transport timetables, road works, traffic information;

  • Statistics: GDP, age, health, unemployment, income, education etc.;

  • Company and business registers.

  1. Cost: gives an overview on how public sector bodies, including libraries, museums and archives, should calculate the amount they should charge re-users for data. For example:

  • Where digital documents are downloaded electronically a no‑cost policy is recommended;

  • For cost-recovery charging, any income generated in the process of collecting or producing documents, e.g. from registration fees or taxes, should be subtracted from the total costs incurred so as to establish the ‘net cost’ of collection, production, reproduction and dissemination.

European Commission Vice President @NeelieKroesEU said: “This guidance will help all of us benefit from the wealth of information public bodies hold. Opening and re-using this data will lead to many new businesses and convenient services.

An independent report carried out by the consultants McKinsey in 2013 claimed that open data re-use could boost the global economy hugely; and a 2013 Spanish studyfound that commercial re-users in Spain could employ around 10,000 people and reach a business volume of €900 million….”

See also Speech by Neelie Kroes: Embracing the open opportunity

The Quiet Movement to Make Government Fail Less Often


in The New York Times: “If you wanted to bestow the grandiose title of “most successful organization in modern history,” you would struggle to find a more obviously worthy nominee than the federal government of the United States.

In its earliest stirrings, it established a lasting and influential democracy. Since then, it has helped defeat totalitarianism (more than once), established the world’s currency of choice, sent men to the moon, built the Internet, nurtured the world’s largest economy, financed medical research that saved millions of lives and welcomed eager immigrants from around the world.

Of course, most Americans don’t think of their government as particularly successful. Only 19 percent say they trust the government to do the right thing most of the time, according to Gallup. Some of this mistrust reflects a healthy skepticism that Americans have always had toward centralized authority. And the disappointing economic growth of recent decades has made Americans less enamored of nearly every national institution.

But much of the mistrust really does reflect the federal government’s frequent failures – and progressives in particular will need to grapple with these failures if they want to persuade Americans to support an active government.

When the federal government is good, it’s very, very good. When it’s bad (or at least deeply inefficient), it’s the norm.

The evidence is abundant. Of the 11 large programs for low- and moderate-income people that have been subject to rigorous, randomized evaluation, only one or two show strong evidence of improving most beneficiaries’ lives. “Less than 1 percent of government spending is backed by even the most basic evidence of cost-effectiveness,” writes Peter Schuck, a Yale law professor, in his new book, “Why Government Fails So Often,” a sweeping history of policy disappointments.

As Mr. Schuck puts it, “the government has largely ignored the ‘moneyball’ revolution in which private-sector decisions are increasingly based on hard data.”

And yet there is some good news in this area, too. The explosion of available data has made evaluating success – in the government and the private sector – easier and less expensive than it used to be. At the same time, a generation of data-savvy policy makers and researchers has entered government and begun pushing it to do better. They have built on earlier efforts by the Bush and Clinton administrations.

The result is a flowering of experiments to figure out what works and what doesn’t.

New York City, Salt Lake City, New York State and Massachusetts have all begun programs to link funding for programs to their success: The more effective they are, the more money they and their backers receive. The programs span child care, job training and juvenile recidivism.

The approach is known as “pay for success,” and it’s likely to spread to Cleveland, Denver and California soon. David Cameron’s conservative government in Britain is also using it. The Obama administration likes the idea, and two House members – Todd Young, an Indiana Republican, and John Delaney, a Maryland Democrat – have introduced a modest bill to pay for a version known as “social impact bonds.”

The White House is also pushing for an expansion of randomized controlled trials to evaluate government programs. Such trials, Mr. Schuck notes, are “the gold standard” for any kind of evaluation. Using science as a model, researchers randomly select some people to enroll in a government program and others not to enroll. The researchers then study the outcomes of the two groups….”

Networks and Hierarchies


on whether political hierarchy in the form of the state has met its match in today’s networked world in the American Interest: “…To all the world’s states, democratic and undemocratic alike, the new informational, commercial, and social networks of the internet age pose a profound challenge, the scale of which is only gradually becoming apparent. First email achieved a dramatic improvement in the ability of ordinary citizens to communicate with one another. Then the internet came to have an even greater impact on the ability of citizens to access information. The emergence of search engines marked a quantum leap in this process. The advent of laptops, smartphones, and other portable devices then emancipated electronic communication from the desktop. With the explosive growth of social networks came another great leap, this time in the ability of citizens to share information and ideas.
It was not immediately obvious how big a challenge all this posed to the established state. There was a great deal of cheerful talk about the ways in which the information technology revolution would promote “smart” or “joined-up” government, enhancing the state’s ability to interact with citizens. However, the efforts of Anonymous, Wikileaks and Edward Snowden to disrupt the system of official secrecy, directed mainly against the U.S. government, have changed everything. In particular, Snowden’s revelations have exposed the extent to which Washington was seeking to establish a parasitical relationship with the key firms that operate the various electronic networks, acquiring not only metadata but sometimes also the actual content of vast numbers of phone calls and messages. Techniques of big-data mining, developed initially for commercial purposes, have been adapted to the needs of the National Security Agency.
The most recent, and perhaps most important, network challenge to hierarchy comes with the advent of virtual currencies and payment systems like Bitcoin. Since ancient times, states have reaped considerable benefits from monopolizing or at least regulating the money created within their borders. It remains to be seen how big a challenge Bitcoin poses to the system of national fiat currencies that has evolved since the 1970s and, in particular, how big a challenge it poses to the “exorbitant privilege” enjoyed by the United States as the issuer of the world’s dominant reserve (and transaction) currency. But it would be unwise to assume, as some do, that it poses no challenge at all….”

Do We Choose Our Friends Because They Share Our Genes?


Rob Stein at NPR: “People often talk about how their friends feel like family. Well, there’s some new research out that suggests there’s more to that than just a feeling. People appear to be more like their friends genetically than they are to strangers, the research found.
“The striking thing here is that friends are actually significantly more similar to one another than we were expecting,” says  James Fowler, a professor of medical genetics at the University of California, San Diego, who conducted the study with Nicholas A. Christakis, a social scientist at Yale University.
In fact, the study in Monday’s issue of the Proceedings of the National Academy of Sciences found that friends are as genetically similar as fourth cousins.
“It’s as if they shared a great- great- great-grandparent in common,” Fowler told Shots.
Some of the genes that friends were most likely to have in common involve smell. “We tend to smell things the same way that our friends do,” Fowler says. The study involved nearly 2,000 adults.
This suggests that as humans evolved, the ability to tolerate and be drawn to certain smells may have influenced where people hung out. Today we might call this the Starbucks effect.
“You may really love the smell of coffee. And you’re drawn to a place where other people have been drawn to who also love the smell of coffee,” Fowler says. “And so that might be the opportunity space for you to make friends. You’re all there together because you love coffee and you make friends because you all love coffee.”…”

i-teams


New Report and Site from NESTA: “Last year we were aware of the growing trend for governments to set up innovation teams, funds, and labs. Yet who are they? What do they do? And crucially, are they making any difference for their host and partner governments? Together Nesta and Bloomberg Philanthropies set out to answer these questions.
Drawing on an in-depth literature review, over 80 interviews, and  surveys, i-teams tells the stories of 20 teams, units and funds, all are established by government, and all are charged with making innovation happen. The i-teams case studied are based in city, regional and national governments across six continents, and work across the spectrum of innovation – from focusing on incremental improvements to aiming for radical transformations.
The i-teams were all created in recognition that governments need dedicated structures, capabilities and space to allow innovation to happen. Beyond this, the i-teams work in different ways, drawing on a mix of methods, approaches, skills, resources, and tackling challenges as diverse as reducing murder rates to improving education attainment.
The i-teams report details the different ways in which these twenty i-teams operate, but to highlight a few:

  • The Behavioural Insights Team designs trials to test policy ideas, and achieved government savings of around 22 times the cost of the team in the first two years of operation.
  • MindLab is a Danish unit using human centred design as a way to identify problems and develop policy recommendations. One project helped businesses to find the right industry code for registrations and demonstrated a 21:1 return on investment in savings to government and businesses.
  • New Orleans Innovation Delivery Team is based in city hall and is tasked with solving mayoral challenges. Their public safety efforts led to a 20% reduction in the number of murders in 2013 compared to the previous year.
  • PS21 encourages staff to find better ways of improving Singaporean public services. An evaluation of PS21 estimated that over a year it generated 520,000 suggestions from staff, of which approximately 60 per cent were implemented, leading to savings of around £55 million.

Alongside the report we have launched theiteams.org a living map to keep track of i-teams developing and emerging around the world, and to create a network of global government innovators. As James Anderson from Bloomberg Philanthropies says, “There’s no reason for every government to start its innovation efforts from scratch.” There is much we can learn from what is underway, what’s working and what’s not, to ensure all i-teams are using the most cutting edge techniques, methods and approaches….”

Meet the UK start-ups changing the world with open data


Sophie Curtis in The Telegraph: “Data is more accessible today than anyone could have imagined 10 or 20 years ago. From corporate databases to social media and embedded sensors, data is exploding, with total worldwide volume expected to reach 6.6 zettabytes by 2020.
Open data is information that is available for anyone to use, for any purpose, at no cost. For example, the Department for Education publishes open data about the performance of schools in England, so that companies can create league tables and citizens can find the best-performing schools in their catchment area.
Governments worldwide are working to open up more of their data. Since January 2010, more than 18,500 UK government data sets have been released via the data.gov.uk web portal, creating new opportunities for organisations to build innovative digital services.
Businesses are also starting to realise the value of making their non-personal data freely available, with open innovation leading to the creation products and services that they can benefit from….

Now a range of UK start-ups are working with the ODI to build businesses using open data, and have already unlocked a total of £2.5 million worth of investments and contracts.
Mastodon C joined the ODI start-up programme at its inception in December 2012. Shortly after joining, the company teamed up with Ben Goldacre and Open Healthcare UK, and embarked on a project investigating the use of branded statins over the far cheaper generic versions.
The data analysis identified potential efficiency savings to the NHS of £200 million. The company is now also working with the Technology Strategy Board and Nesta to help them gain better insight into their data.
Another start-up, CarbonCulture is a community platform designed to help people use resources more efficiently. The company uses high-tech metering to monitor carbon use in the workplace and help clients save money.
Organisations such as 10 Downing Street, Tate, Cardiff Council, the GLA and the UK Parliament are using the company’s digital tools to monitor and improve their energy consumption. CarbonCulture has also helped the Department of Energy and Climate Change reduce its gas use by 10 per cent.
Spend Network’s business is built on collecting the spend statements and tender documents published by government in the UK and Europe and then publishing this data openly so that anyone can use it. The company currently hosts over £1.2 trillion of transactions from the UK and over 1.8 million tenders from across Europe.
One of the company’s major breakthroughs was creating the first national, open spend analysis for central and local government. This was used to uncover a 45 per cent delay in the UK’s tendering process, holding up £22 billion of government funds to the economy.
Meanwhile, TransportAPI uses open data feeds from Traveline, Network Rail and Transport for London to provide nationwide timetables, departure and infrastructure information across all modes of public transport.
TransportAPI currently has 700 developers and organisations signed up to its platform, including individual taxpayers and public sector organisations like universities and local authorities. Travel portals, hyperlocal sites and business analytics are also integrating features, such as the ‘nearest transport’ widget, into their websites.
These are just four examples of how start-ups are using open data to create new digital services. The ODI this week announced seven new open data start-ups joining the programme, covering 3D printed learning materials, helping disabled communities, renewable energy markets, and smart cities….”

Facebook tinkered with users’ feeds for a massive psychology experiment


William Hughes in AVClub: “Scientists at Facebook have published a paper showing that they manipulated the content seen by more than 600,000 users in an attempt to determine whether this would affect their emotional state. The paper, “Experimental evidence of massive-scale emotional contagion through social networks,” was published in The Proceedings Of The National Academy Of Sciences. It shows how Facebook data scientists tweaked the algorithm that determines which posts appear on users’ news feeds—specifically, researchers skewed the number of positive or negative terms seen by randomly selected users. Facebook then analyzed the future postings of those users over the course of a week to see if people responded with increased positivity or negativity of their own, thus answering the question of whether emotional states can be transmitted across a social network. Result: They can! Which is great news for Facebook data scientists hoping to prove a point about modern psychology. It’s less great for the people having their emotions secretly manipulated.

In order to sign up for Facebook, users must click a box saying they agree to the Facebook Data Use Policy, giving the company the right to access and use the information posted on the site. The policy lists a variety of potential uses for your data, most of them related to advertising, but there’s also a bit about “internal operations, including troubleshooting, data analysis, testing, research and service improvement.” In the study, the authors point out that they stayed within the data policy’s liberal constraints by using machine analysis to pick out positive and negative posts, meaning no user data containing personal information was actually viewed by human researchers. And there was no need to ask study “participants” for consent, as they’d already given it by agreeing to Facebook’s terms of service in the first place.

Facebook data scientist Adam Kramer is listed as the study’s lead author. In an interview the company released a few years ago, Kramer is quoted as saying he joined Facebook because “Facebook data constitutes the largest field study in the history of the world.”

See also:
Facebook Experiments Had Few Limits, Data Science Lab Conducted Tests on Users With Little Oversight, Wall Street Journal.
Stop complaining about the Facebook study. It’s a golden age for research, Duncan Watts