Paper by Kevin J. Boudreau and Karim Lakhani: “This paper discusses several challenges in designing field experiments to better understand how organizational and institutional design shapes innovation outcomes and the production of knowledge. We proceed to describe the field experimental research program carried out by our Crowd Innovation Laboratory at Harvard University to clarify how we have attempted to address these research design challenges. This program has simultaneously solved important practical innovation problems for partner organizations, like NASA and Harvard Medical School, while contributing research advances, particularly in relation to innovation contests and tournaments….(More)“
The digital revolution liberating Latin American people
Luis Alberto Moreno in the Financial Times: “Imagine a place where citizens can deal with the state entirely online, where all health records are electronic and the wait for emergency care is just seven minutes. Singapore? Switzerland? Try Colima, Mexico.
Pessimists fear the digital revolution will only widen social and economic disparities in the developing world — particularly in Latin America, the world’s most unequal region. But Colima, though small and relatively prosperous, shows how some of the region’s governments are harnessing these tools to modernise services, improve quality of life and share the benefits of technology more equitably.
In the past 10 years, this state of about 600,000 people has transformed the way government works, going completely digital. Its citizens can carry out 62 procedures online, from applying for permits to filing crime reports. No internet at home? Colima offers hundreds of free WiFi hotspots.
Colombia and Peru are taking broadband to remote corners of their rugged territories. Bogotá has subsidised the expansion of its fibre optic network, which now links virtually every town in the country. Peru is expanding a programme that aims to bring WiFi to schools, hospitals and other public buildings in each of its 25 regions. The Colombian plan, Vive Digital, fosters internet adoption among all its citizens. Taxes on computers, tablets and smartphones have been scrapped. Low-income families have been given vouchers to sign up for broadband. In five years, the percentage of households connected to the internet jumped from 16 per cent to 50 per cent. Among small businesses it soared from 7 per cent to 61 per cent .
Inexpensive devices and ubiquitous WiFi, however, do not guarantee widespread usage. Diego Molano Vega, an architect of Vive Digital, found that many programs designed for customers in developed countries were ill suited to most Colombians. “There are no poor people in Silicon Valley,” he says. Latin American governments should use their purchasing power to push for development of digital services easily adopted by their citizens and businesses. Chile is a leader: it has digitised hundreds of trámites — bureaucratic procedures involving endless forms and queues. In a 4,300km-longcountry of mountains, deserts and forests, this enables access to all sorts of services through the internet. Entrepreneurs can now register businesses online for free in a single day.
In Chile, entrepreneurs can now register new businesses online for free in a single day…
Technology can be harnessed to boost equity in education. Brazil’s Mato Grosso do Sul state launched a free online service to prepare high school students for a tough national exam in which a good grade is a prerequisite for admission to federal universities. On average the results of the students who used the service were 31 per cent higher than those of their peers, prompting 10 other states to adopt the system.
Digital tools can also help raise competitiveness in business. Uruguay’s livestock information system keeps track of the country’s cattle. The publicly financed electronic registry ensures every beast can be traced, making it easier to monitor outbreaks of diseases….(More)”
The data or the hunch
Ian Leslie at Intelligent Life: “THE GIFT FOR talent-spotting is mysterious, highly prized and celebrated. We love to hear stories about the baseball coach who can spot the raw ability of an erratic young pitcher, the boss who sees potential in the guy in the post room, the director who picks a soloist out of the chorus line. Talent shows are a staple of the TV schedules. We like to believe that certain people—sometimes ourselves—can just sense when a person has something special. But there is another method of spotting talent which doesn’t rely on hunches. In place of intuition, it offers data and analysis. Rather than relying on the gut, it invites us to use our heads. It tends not to make for such romantic stories, but it is effective—which is why, despite our affection, the hunch is everywhere in retreat.
Strike one against the hunch was the publication of “Moneyball” by Michael Lewis (2003), which has attained the status of a management manual for many in sport and beyond. Lewis reported on a cash-strapped major-league baseball team, the Oakland A’s, who enjoyed unlikely success against bigger and better-funded competitors. Their secret sauce was data. Their general manager, Billy Beane, had realised that when it came to evaluating players, the gut instincts of experienced baseball scouts were unreliable, and he employed statisticians to identify talent overlooked by the big clubs…..
These days, when a football club is interested in a player, it considers the average distance he runs in a game, the number of passes and tackles or blocks he makes, his shots on goal, the ratio of goals to shots, and many other details nobody thought to measure a generation ago. Sport is far from the only industry in which talent-spotting is becoming a matter of measurement. Prithwijit Mukerji, a postgraduate at the University of Westminster in London, recently published a paper on the way the music industry is being transformed by “the Moneyball approach”. By harvesting data from Facebook and Twitter and music services like Spotify and Shazam, executives can track what we are listening to in far more detail than ever before, and use it as a guide to what we will listen to next….
This is the day of the analyst. In education, academics are working their way towards a reliable method of evaluating teachers, by running data on test scores of pupils, controlled for factors such as prior achievement and raw ability. The methodology is imperfect, but research suggests that it’s not as bad as just watching someone teach. A 2011 study led by Michael Strong at the University of California identified a group of teachers who had raised student achievement and a group who had not. They showed videos of the teachers’ lessons to observers and asked them to guess which were in which group. The judges tended to agree on who was effective and ineffective, but, 60% of the time, they were wrong. They would have been better off flipping a coin. This applies even to experts: the Gates Foundation funded a vast study of lesson observations, and found that the judgments of trained inspectors were highly inconsistent.
THE LAST STRONGHOLD of the hunch is the interview. Most employers and some universities use interviews when deciding whom to hire or admit. In a conventional, unstructured interview, the candidate spends half an hour or so in a conversation directed at the whim of the interviewer. If you’re the one deciding, this is a reassuring practice: you feel as if you get a richer impression of the person than from the bare facts on their résumé, and that this enables you to make a better decision. The first theory may be true; the second is not.
Decades of scientific evidence suggest that the interview is close to useless as a tool for predicting how someone will do a job. Study after study has found that organisations make better decisions when they go by objective data, like the candidate’s qualifications, track record and performance in tests. “The assumption is, ‘if I meet them, I’ll know’,” says Jason Dana, of Yale School of Management, one of many scholars who have looked into the interview’s effectiveness. “People are wildly over-confident in their ability to do this, from a short meeting.” When employers adopt a holistic approach, combining the data with hunches formed in interviews, they make worse decisions than they do going on facts alone….” (More)
Democratising the Data Revolution
Jonathan Gray at Open Knowledge: “What will the “data revolution” do? What will it be about? What will it count? What kinds of risks and harms might it bring? Whom and what will it serve? And who will get to decide?
Today we are launching a new discussion paper on “Democratising the Data Revolution”, which is intended to advance thinking and action around civil society engagement with the data revolution. It looks beyond the disclosure of existing information, towards more ambitious and substantive forms of democratic engagement with data infrastructures.1
It concludes with a series of questions about what practical steps institutions and civil society organisations might take to change what is measured and how, and how these measurements are put to work.
You can download the full PDF report here, or continue to read on in this blog post.
What Counts?
How might civil society actors shape the data revolution? In particular, how might they go beyond the question of what data is disclosed towards looking at what is measured in the first place? To kickstart discussion around this topic, we will look at three kinds of intervention: changing existing forms of measurement, advocating new forms of measurement and undertaking new forms of measurement.
Changing Existing Forms of Measurement
Rather than just focusing on the transparency, disclosure and openness of public information, civil society groups can argue for changing what is measured with existing data infrastructures. One example of this is recent campaigning around company ownership in the UK. Advocacy groups wanted to unpick networks of corporate ownership and control in order to support their campaigning and investigations around tax avoidance, tax evasion and illicit financial flows.
While the UK company register recorded information about “nominal ownership”, it did not include information about so-called “beneficial ownership”, or who ultimately benefits from the ownership and control of companies. Campaigners undertook an extensive programme of activities to advocate for changes and extensions to existing data infrastructures – including via legislation, software systems, and administrative protocols.2
Advocating New Forms of Measurement
As well as changing or recalibrating existing forms of measurement, campaigners and civil society organisations can make the case for the measurement of things which were not previously measured. For example, over the past several decades social and political campaigning has resulted in new indicators about many different issues – such as gender inequality, health, work, disability, pollution or education.3 In such cases activists aimed to establish a given indicator as important and relevant for public institutions, decision makers, and broader publics – in order to, for example, inform policy development or resource allocation.
Undertaking New Forms of Measurement
Historically, many civil society organisations and advocacy groups have collected their own data to make the case for action on issues that they work on – from human rights abuses to endangered species….(More)”
Government at a Glance 2015
New report and dataset by the OECD: “Government at a Glance provides readers with a dashboard of key indicators assembled with the goal of contributing to the analysis and international comparison of public sector performance. Indicators on government revenues, expenditures, and employment are provided alongside key output and outcome data in the sectors of education, health and justice. Government at a Glance also includes indicators on key governance and public management issues, such as transparency in governance, regulatory governance, public procurement and the implementation of employment and remuneration reforms since 2008. While measuring government performance has long been recognized as playing an important role in increasing the effectiveness and efficiency of the public administration, following the economic crisis and fiscal tightening in many member countries, good indicators are needed more than ever to help governments make informed decisions regarding tough choices and help restore confidence in government institutions… (More)”
The science prize that’s making waves
Gillian Tett at the Financial Times: “The Ocean Health XPrize reveals a new fashion among philanthropists’…There is another reason why the Ocean Health XPrize fascinates me: what it reveals about the new fashion among philanthropists for handing out big scientific prizes. The idea is not a new one: wealthy people and governments have been giving prizes for centuries. In 1714, for example, the British government passed the Longitude Act, establishing a board to offer reward money for innovation in navigation — the most money was won by John Harrison, a clockmaker who invented the marine chronometer.
But a fascinating shift has taken place in the prize-giving game. In previous decades, governments or philanthropists usually bestowed money to recognise past achievements, often in relation to the arts. In 2012, McKinsey, the management consultants, estimated that before 1991, 97 per cent of prize money was a “recognition” award — for example, the Nobel Prizes. Today, however, four-fifths of all prize money is “incentive” or “inducement” awards. This is because many philanthropists and government agencies have started staging competitions to spur innovation in different fields, particularly science.
The best known of these is the XPrize Foundation, initiated two decades ago by Peter Diamandis, the entrepreneur. The original award, the Ansari XPrize, offered $10m to the first privately financed team to put a vehicle into space. Since then, the XPrize has spread its wings into numerous different fields, including education and life sciences. Indeed, having given $30m in prize money so far, it has another $70m of competitions running, including the Google Lunar XPrize, which is offering $30m to land a privately funded robot on the moon.
McKinsey estimates that if you look across the field of prize-giving around the world, “total funds available from large prizes have more than tripled over the last decade to reach $350m”, while the “total prize sector could already be worth as much as $1bn to $2bn”. The Ocean Health XPrize, in other words, is barely a drop in the prize-giving ocean.
Is this a good thing? Not always, it might seem. As the prizes proliferate, they can sometimes overlap. The money being awarded tends — inevitably — to reflect the pet obsessions of philanthropists, rather than what scientists themselves would like to explore. And even the people running the prizes admit that these only work when there is a clear problem to be solved….(More)”
This App Lets You See The Tough Choices Needed To Balance Your City’s Budget
Balancing Act is a web app that displays a straightforward pie chart of a city’s budget, broken down into categories like pensions, parks & recreations, police, and education. But it doesn’t just display the current budget breakdown. It invites users to tweak it, expressing their own priorities, all while keeping the city in the black. Do you want your libraries to be better funded? Fine—but you’re going to have to raise property taxes to do it.
“Balancing Act provides a way for people to both understand what public entities are doing and then to weight that against the other possible things that government can do,” says Chris Adams, president of Engaged Public, a Colorado-based consulting firm that develops technology for government and non-profits. “Especially in this era of information, all of us have a responsibility to spend a bit of time understanding how our government is spending money on our behalf.”
Hartford, Connecticut is the first city in the country that is using Balancing Act. The city was facing a $49 million budget deficit this spring, and Mayor Pedro Segarra says he took input from citizens using Balancing Act. Meanwhile, in Engaged Public’s home state, residents can input their income to generate an itemized tax receipt and then tweak the Colorado state budget as they see fit.
Engaged Public hopes that by making budgets more interactive and accessible, more people will take an interest in them.
“Budget information almost universally exists, but it’s not in accessible formats—mostly they’re in PDF files,” says Adams. “So citizens are invited to pour through tens of thousands of pages of PDFs. But that really doesn’t give you a high-level understanding of what’s at stake in a reasonable amount of time.”
If widely used, Balancing Act could be a useful tool for politicians to check the pulse of their constituents. For example, decreasing funding to parks draws a negative public reaction. But if enough people on Balancing Act experimented with the budget, saw the necessity of it, and submitted their recommendations, then an elected might be willing to make a decision that would otherwise seem politically risky….(More)”
When Guarding Student Data Endangers Valuable Research
Susan M. Dynarski in the New York Times: “There is widespread concern over threats to privacy posed by the extensive personal data collected by private companies and public agencies.
Some of the potential danger comes from the government: The National Security Agency has swept up the telephone records of millions of people, in what it describes as a search for terrorists. Other threats are posed by hackers, who have exploited security gaps to steal data from retail giantslike Target and from the federal Office of Personnel Management.
Resistance to data collection was inevitable — and it has been particularly intense in education.
Privacy laws have already been strengthened in some states, and multiple bills now pending in state legislatures and in Congress would tighten the security and privacy of student data. Some of this proposed legislation is so broadly written, however, that it could unintentionally choke off the use of student data for its original purpose: assessing and improving education. This data has already exposed inequities, allowing researchers and advocates to pinpoint where poor, nonwhite and non-English-speaking children have been educated inadequately by their schools.
Data gathering in education is indeed extensive: Across the United States, large, comprehensive administrative data sets now track the academic progress of tens of millions of students. Educators parse this data to understand what is working in their schools. Advocates plumb the data to expose unfair disparities in test scores and graduation rates, building cases to target more resources for the poor. Researchers rely on this data when measuring the effectiveness of education interventions.
To my knowledge there has been no large-scale, Target-like theft of private student records — probably because students’ test scores don’t have the market value of consumers’ credit card numbers. Parents’ concerns have mainly centered not on theft, but on the sharing of student data with third parties, including education technology companies. Last year, parentsresisted efforts by the tech start-up InBloom to draw data on millions of students into the cloud and return it to schools as teacher-friendly “data dashboards.” Parents were deeply uncomfortable with a third party receiving and analyzing data about their children.
In response to such concerns, some pending legislation would scale back the authority of schools, districts and states to share student data with third parties, including researchers. Perhaps the most stringent of these proposals, sponsored by Senator David Vitter, a Louisiana Republican, would effectively end the analysis of student data by outside social scientists. This legislation would have banned recent prominent research documenting the benefits of smaller classes, the value of excellent teachersand the varied performance of charter schools.
Under current law, education agencies can share data with outside researchers only to benefit students and improve education. Collaborations with researchers allow districts and states to tap specialized expertise that they otherwise couldn’t afford. The Boston public school district, for example, has teamed up with early-childhood experts at Harvard to plan and evaluate its universal prekindergarten program.
In one of the longest-standing research partnerships, the University of Chicago works with the Chicago Public Schools to improve education. Partnerships like Chicago’s exist across the nation, funded by foundations and the United States Department of Education. In one initiative, a Chicago research consortium compiled reports showing high school principals that many of the seniors they had sent off to college swiftly dropped out without earning a degree. This information spurred efforts to improve high school counseling and college placement.
Specific, tailored information in the hands of teachers, principals or superintendents empowers them to do better by their students. No national survey could have told Chicago’s principals how their students were doing in college. Administrative data can provide this information, cheaply and accurately…(More)”
What cybersecurity can learn from citizen science
But as anyone who has observed an online forum thread dissecting the minutiae of geek culture can attest, hobbyists can be remarkably thorough in their exploration of topics they are passionate about. And it is often a point of pride to pick the subject that is the least conventional or popular.
The idea of citizen science is to include amateur science enthusiasts in the collection and processing of data. Thanks to the Internet, we’ve seen a surge in the number of self-taught experts in a variety of subjects. New participation platforms are social and gamified – utilizing people’s desire to compete or collaborate with others who share their passion.
How this process plays out differs from one app to the next, according to their needs: StarDust@Home asks volunteers to help sort through samples captured by the Stardust spacecraft when it flew through the coma of comet Wild 2 in 2004. They do this by viewing movies of the contents of the aerogel tiles that were used as collectors.
The security community is ripe for using the citizen science in similar ways to these. Most antimalware vendors make use of customer samples for adding detection and cleaning to their products. Many security companies use customers’ reports to gather file reputation, telemetry and prevalence data. And bug reports come from researchers of all ages and education levels – not just professional security researchers. “Month of Bug” events are a more controversial way that security is gamified. Could security companies or organizations be doing more to engage enthusiasts to help improve our response to security issues?
It could be argued that the stuff of security research – especially malware research – is potentially harmful in the hands of amateurs and should be handled only by seasoned professionals. Not only that, security is an adversarial system where the criminals would likely try to game the system to improve their profits. These are important concerns that would need to be addressed.
But the citizen science approach provides good lessons…(More)”
Did Performance Measurement Cause America’s Police Problem?
Katherine Barrett and Richard Greene in Governing: “You’ve doubtless heard the maxim “what gets measured, gets managed.” Sometimes it’s attributed to management guru Peter Drucker, though others also get credit for it. But whoever actually coined the phrase, we remember the first time we became aware of it, about a quarter of a century ago.
It seemed like a purely positive sentiment to us back in the days when we naively believed that performance measurement could cure most governmental ills. If gathering data about inputs, outputs and outcomes could solve all management problems, then cities and states had access to a golden key to a more effective and efficient future. Then reality intervened and we recognized that even good measurements don’t necessarily result in the right policy or practice changes.
But, somewhat more ominously, we’ve become aware of a troubling question that lurks in the field of performance measurement: What happens if we’re not measuring the right things in the first place? If Drucker — or whoever — was right, doesn’t that mean that we may manage government programs in a way that leads to more problems? Sometimes, for example, states and localities focus their measurements on the speed with which a service is delivered. Faster always seems better. But often delivering a service quickly means doing so less effectively.
For fire departments, response times are a commonly used measure of service quality. But “the requirement for low response times may incentivize firefighters to drive fast,” said Amy Donahue, professor and vice-provost for academic operations at the University of Connecticut. “And it has been shown that while speeding saves very little in terms of total driving time, it is much more dangerous — both to those in the emergency vehicle and other innocents who might get in their way. The potential for accidents is high, and when they happen, the consequences can be very tragic.”
As the field has become aware of these dangers, many agencies are trying to mitigate them by improving education, prohibiting responders from exceeding speed limits, and requiring responders to participate in emergency vehicle operators programs.
Examples like this one are everywhere. But we just came across something in the March 2015 edition of New Perspectives in Policing that had never occurred to us before and that seems to be widely ignored by public safety organizations around the country. It was written by Malcolm K. Sparrow, professor of practice of public management at the John F. Kennedy School of Government at Harvard University
As violent incidents in several of America’s cities show the underlying tensions between police and the public they serve, Sparrow argues that some of this dissonance has actually been encouraged by the fact that most police departments are pushed to measure crime clearance and enforcement. These are important factors, but they have little to do with community satisfaction. Meanwhile, he points out that “a few departments now use citizen satisfaction surveys on a regular basis, but most do not.”…(More)”