DrivenData Blog: “As we begin launching our first competitions, we thought it would be a good idea to lay out what exactly we’re trying to do and why….
At DrivenData, we want to bring cutting-edge practices in data science and crowdsourcing to some of the world’s biggest social challenges and the organizations taking them on. We host online challenges, usually lasting 2-3 months, where a global community of data scientists competes to come up with the best statistical model for difficult predictive problems that make a difference.
Just like every major corporation today, nonprofits and NGOs have more data than ever before. And just like those corporations, they are trying to figure out how to make the best use of their data. We work with mission-driven organizations to identify specific predictive questions that they care about answering and can use their data to tackle.
Then we host the online competitions, where experts from around the world vie to come up with the best solution. Some competitors are experienced data scientists in the private sector, analyzing corporate data by day, saving the world by night, and testing their mettle on complex questions of impact. Others are smart, sophisticated students and researchers looking to hone their skills on real-world datasets and real-world problems. Still more have extensive experience with social sector data and want to bring their expertise to bear on new, meaningful challenges – with immediate feedback on how well their solution performs.
Like any data competition platform, we want to harness the power of crowds combined with the increasing prevalence of large, relevant datasets. Unlike other data competition platforms, our primary goal is to create actual, measurable, lasting positive change in the world with our competitions. At the end of each challenge, we work with the sponsoring organization to integrate the winning solutions, giving them the tools to drive real improvements in their impact….
We are launching soon and we want you to join us!
If you want to get updates about our launch this fall with exciting, real competitions, please sign up for our mailing list here and follow us on Twitter: @drivendataorg.
If you are a data scientist, feel free to create an account and start playing with our first sandbox competitions.
If you are a nonprofit or public sector organization, and want to squeeze every drop of mission effectiveness out of your data, check out the info on our site and let us know! “
The Stasi, casinos and the Big Data rush
Book Review by Hannah Kuchler of “What Stays in Vegas” (by Adam Tanner) in the Financial Times: “Books with sexy titles and decidedly unsexy topics – like, say, data – have a tendency to disappoint. But What Stays in Vegas is an engrossing, story-packed takedown of the data industry.
It begins, far from America’s gambling capital, in communist East Germany. The author, Adam Tanner, now a fellow at Harvard’s Institute for Quantitative Social Science, was in the late 1980s a travel writer taking notes on Dresden. What he did not realise was that the Stasi was busy taking notes on him – 50 pages in all – which he found when the files were opened after reunification. The secret police knew where he had stopped to consult a map, to whom he asked questions and when he looked in on a hotel.
Today, Tanner explains: “Thanks to meticulous data gathering from both public documents and commercial records, companies . . . know far more about typical consumers than the feared East German secret police recorded about me.”
Shining a light on how businesses outside the tech sector have become data addicts, Tanner focuses on Las Vegas casinos, which spotted the value in data decades ago. He was given access to Caesar’s Entertainment, one of the world’s largest casino operators. When chief executive Gary Loveman joined in the late 1990s, the former Harvard Business School professor bet the company’s future on harvesting personal data from its loyalty scheme. Rather than wooing the “whales” who spent the most, the company would use the data to decide which freebies were worth giving away to lure in mid-spenders who came back often – a strategy credited with helping the business grow.
The real revelations come when Tanner examines the data brokers’ “Cheez Whiz”. Like the maker of a popular processed dairy spread, he argues, data brokers blend ingredients from a range of sources, such as public records, marketing lists and commercial records, to create a detailed picture of your identity – and you will never quite be able to pin down the origin of any component…
The Big Data rush has gone into overdrive since the global economic crisis as marketers from different industries have sought new methods to grab the limited consumer spending available. Tanner argues that while users have in theory given permission for much of this information to be made public in bits and pieces, increasingly industrial-scale aggregation often feels like an invasion of privacy.
Privacy policies are so long and obtuse (one study Tanner quotes found that it would take a person more than a month, working full-time, to read all the privacy statements they come across in a year), people are unwittingly littering their data all over the internet. Anyway, marketers can intuit what we are like from the people we are connected to online. And as the data brokers’ lists are usually private, there is no way to check the compilers have got their facts right…”
Goodbye, Organization Man
David Brooks in the New York Times:”…The result, right now, is unnecessary deaths from the Ebola virus in Africa. …. At root, this is a governance failure. The disease spreads fastest in places where the health care infrastructure is lacking or nonexistent. Liberia, for example, is being overrun while Ivory Coast has put in a series of policies to prevent an outbreak. The few doctors and nurses in the affected places have trouble acquiring the safety basics: gloves and body bags. More than 100, so far, have died fighting the outbreak.
But it’s not just a failure of governance in Africa. It’s a failure of governance around the world. I wonder if we are looking at the results of a cultural shift.
A few generations ago, people grew up in and were comfortable with big organizations — the army, corporations and agencies. They organized huge construction projects in the 1930s, gigantic industrial mobilization during World War II, highway construction and corporate growth during the 1950s. Institutional stewardship, the care and reform of big organizations, was more prestigious.
Now nobody wants to be an Organization Man. We like start-ups, disrupters and rebels. Creativity is honored more than the administrative execution. Post-Internet, many people assume that big problems can be solved by swarms of small, loosely networked nonprofits and social entrepreneurs. Big hierarchical organizations are dinosaurs.
The Ebola crisis is another example that shows that this is misguided. The big, stolid agencies — the health ministries, the infrastructure builders, the procurement agencies — are the bulwarks of the civil and global order. Public and nonprofit management, the stuff that gets derided as “overhead,” really matters. It’s as important to attract talent to health ministries as it is to spend money on specific medicines.
As recent books by Francis Fukuyama and Philip Howard have detailed, this is an era of general institutional decay. New, mobile institutions languish on the drawing broad, while old ones are not reformed and tended. Executives at public agencies are robbed of discretionary power. Their hands are bound by court judgments and regulations.
When the boring tasks of governance are not performed, infrastructures don’t get built. Then, when epidemics strike, people die.”
EU: GLOW (Global Legislative Openness Week)
GLOW is a celebration of open, participatory legislative processes around the world as well as an opportunity for diverse stakeholders to collaborate with one another and make progress toward adopting and implementing open-government commitments. The week is being led by the Legislative Openness Working Group of the Open Government Partnership, which is co-anchored by the National Democratic Institute and the Congress of Chile.
The campaign kicks off with the International Day of Democracy on September 15, and throughout the 10 days you are invited to share your ideas and experiences, kickstart new transparency tools and engage members of your community in dialogue. Learn more about the global open government movement at OGP, and stay tuned into GLOW events by following this site and #OpenParl2014.
Where will GLOW be happening?
GLOW will connect a range of legislative openness activities, organized independently by civil society organizations and parliaments around the world. You can follow the action on Twitter by using the hashtag #OpenParl2014. We hope the GLOW campaign will inspire you to design and organize your own event or activity during this week. If you’d like to share your event and collaborate with others during GLOW, please send us a note.
The week’s festivities will be anchored by two Working Group meetings of civil society and parliamentary members. Beginning on the International Day of Democracy, September 15, the Working Group will host a regional meeting on expanding civic engagement through parliamentary openness in Podgorica, Montenegro, hosted in partnership with the Parliament of Montenegro. The week will conclude with the Working Group’s annual meeting in Chile, on September 25 and 26, 2014, where members will discuss progress made in the year since the Working Group’s launch. This meeting coincides with the 11th Plenary Assembly of ParlAmericas, an independent network composed of the national legislatures of the 35 independent states of the Americas, which will also consider issues of legislative openness as part of its meeting….” (More)
Rethinking Democracy
Dani Rodrik at Project Syndicate: “By many measures, the world has never been more democratic. Virtually every government at least pays lip service to democracy and human rights. Though elections may not be free and fair, massive electoral manipulation is rare and the days when only males, whites, or the rich could vote are long gone. Freedom House’s global surveys show a steady increase from the 1970s in the share of countries that are “free” – a trend that the late Harvard political scientist Samuel Huntington dubbed the “third wave” of democratization….
A true democracy, one that combines majority rule with respect for minority rights, requires two sets of institutions. First, institutions of representation, such as political parties, parliaments, and electoral systems, are needed to elicit popular preferences and turn them into policy action. Second, democracy requires institutions of restraint, such as an independent judiciary and media, to uphold fundamental rights like freedom of speech and prevent governments from abusing their power. Representation without restraint – elections without the rule of law – is a recipe for the tyranny of the majority.
Democracy in this sense – what many call “liberal democracy” – flourished only after the emergence of the nation-state and the popular upheaval and mobilization produced by the Industrial Revolution. So it should come as no surprise that the crisis of liberal democracy that many of its oldest practitioners currently are experiencing is a reflection of the stress under which the nation-state finds itself….
In developing countries, it is more often the institutions of restraint that are failing. Governments that come to power through the ballot box often become corrupt and power-hungry. They replicate the practices of the elitist regimes they replaced, clamping down on the press and civil liberties and emasculating (or capturing) the judiciary. The result has been called “illiberal democracy” or “competitive authoritarianism.” Venezuela, Turkey, Egypt, and Thailand are some of the better-known recent examples.
When democracy fails to deliver economically or politically, perhaps it is to be expected that some people will look for authoritarian solutions. And, for many economists, delegating economic policy to technocratic bodies in order to insulate them from the “folly of the masses” almost always is the preferred approach.
…
Effective institutions of restraint do not emerge overnight; and it might seem like those in power would never want to create them. But if there is some likelihood that I will be voted out of office and that the opposition will take over, such institutions will protect me from others’ abuses tomorrow as much as they protect others from my abuses today. So strong prospects for sustained political competition are a key prerequisite for illiberal democracies to turn into liberal ones over time.
Optimists believe that new technologies and modes of governance will resolve all problems and send democracies centered on the nation-state the way of the horse-drawn carriage. Pessimists fear that today’s liberal democracies will be no match for the external challenges mounted by illiberal states like China and Russia, which are guided only by hardnosed realpolitik. Either way, if democracy is to have a future, it will need to be rethought.”
Crowd-Sourced, Gamified Solutions to Geopolitical Issues
Gamification Corp: “Daniel Green, co-founder and CTO of Wikistrat, spoke at GSummit 2014 on an intriguing topic: How Gamification Motivates All Age Groups: Or How to Get Retired Generals to Play Games Alongside Students and Interns.
Wikistrat, a crowdsourced consulting company, leverages a worldwide network of experts from various industries to solve some of the world’s geopolitical problems through the power of gamification. Wikistrat also leverages fun, training, mentorship, and networking as core concepts in their company.
Dan (@wsdan) spoke with TechnologyAdvice host Clark Buckner about Wikistrat’s work, origins, what clients can expect from working with Wikistrat, and how gamification correlates with big data and business intelligence. Listen to the podcast and read the summary below:
Wikistrat aims to solve a common problem faced by most governments and organizations when generating strategies: “groupthink.” Such entities can devise a diverse set of strategies, but they always seem to find their resolution in the most popular answer.
In order to break group thinking, Wikistrat carries out geopolitical simulations that work around “collaborative competition.” The process involves:
-
Securing analysts: Wikistrat recruits a diverse group of analysts who are experts in certain fields and located in different strategic places.
-
Competing with ideas: These analysts are placed in an online environment where, instead of competing with each other, one analyst contributes an idea, then other analysts create 2-3 more ideas based on the initial idea.
-
Breaking group thinking: Now the competition becomes only about ideas. People champion the ideas they care about rather than arguing with other analysts. That’s when Wikistrat breaks group thinking and helps their clients discover ideas they may have never considered before.
Gamification occurs when analysts create different scenarios for a specific angle or question the client raises. Plus, Wikistrat’s global analyst coverage is so good that they tout having at least one expert in every country. They accomplished this by allowing anyone—not just four-star generals—to register as an analyst. However, applicants must submit a resume and a writing sample, as well as pass a face-to-face interview….”
Twitter Analytics Project HealthMap Outperforming WHO in Ebola Tracking
HIS Talk: “HealthMap, a collaborative data analytics project launched in 2006 between Harvard Medical School and Boston Children’s Hospital, has been quietly tracking the recent Ebola outbreak in Western Africa with notable accuracy, beating the World Health Organization’s own tracking efforts by two weeks in some instances.
HealthMap aggregates information from a variety of online sources to plot real-time disease outbreaks. Currently, the platform analyzes data from the World Health Organization, Google News, and GeoSentinel, a global disease tracking platform that tracks major geography changes in diseases carried through travelers, foreign visitors, and immigrants. The analytics project also got a new source of feeder-data this February when Twitter announced that the HealthMap project had been selected as a Twitter Data Grant recipient, which gives the 45 epidemiologists working on the project access to the “fire hose” of unfiltered data generated from Twitter’s 500 million daily tweets….”
Open Data: Going Beyond Solving Problems to Making the Impossible Possible
Lisa Abeyta at the Huffington Post: “As a global community, we are producing data at an astounding rate. The pace was recently described as a “new Google every four days” by the highly respected Andreesen Horowitz partner, Peter Levine, in a thought-provoking post addressing the challenge of making sense of this mountain of data.
“… we are now collecting more data each day, so much that 90 percent of the data in the world today has been created in the last two years alone. In fact, every day, we create 2.5 quintillion bytes of data — by some estimates that’s one new Google every four days, and the rate is only increasing. Our desire to use, interact, and learn from this data will become increasingly important and strategic to businesses and society as a whole.”
For the past year and a half, my cofounders and team have focused on what it will take to use, interact and learn from data being produced within the civic sector. It’s one thing to be able to build an app for civic; it’s quite another to build a platform that can manage multiple apps across multiple platforms while addressing the challenges plaguing the “wild west” nature of growth in the quickly emerging market of open data. I was recently invited to share our lessons learned and the promise of the future in mobile open data at the TEDxABQ Technology Salon. Here is a bit of what I shared:
Beyond Solving Problems to New Possibilities
When we first looked at the opportunities for creating apps built on open data, our priority was finding pain points for both cities and the people who lived there. We focused on solving real problems, and it led to some early success. We worked with the City of Albuquerque to deploy their ABQ RIDE app on iOS and Android platforms, and the app not only solved real problems for riders, it also saved real money for the city. The app has grown to over 20,000 regular users and continues to be one of the highest downloaded apps on our platform.
But recently, we’ve started asking questions that go beyond the basic, that change the experience or make things possible in ways that never were before. Here are two I’m incredibly proud to be a part of…”
Knowledge is Beautiful
New book by David McCandless: “In this mind-blowing follow-up to the bestselling Information is Beautiful, the undisputed king of infographics David McCandless uses stunning and unique visuals to reveal unexpected insights into how the world really works. Every minute of every hour of every day we are bombarded with information – be it on television, in print or online. How can we relate to this mind-numbing overload? Enter David McCandless and his amazing infographics: simple, elegant ways to understand information too complex or abstract to grasp any way but visually. McCandless creates dazzling displays that blend the facts with their connections, contexts and relationships, making information meaningful, entertaining – and beautiful. Knowledge is Beautiful is an endlessly fascinating spin through the world of visualized data, all of it bearing the hallmark of David McCandless’s ground-breaking signature style. Taking infographics to the next level, Knowledge is Beautiful offers a deeper, more wide-ranging look at the world and its history. Covering everything from dog breeds and movie plots to the most commonly used passwords and crazy global warming solutions, Knowledge is Beautiful is guaranteed to enrich your understanding of the world.”
What Cars Did for Today’s World, Data May Do for Tomorrow’s
Quentin Hardy in the New York Times: “New technology products head at us constantly. There’s the latest smartphone, the shiny new app, the hot social network, even the smarter thermostat.
As great (or not) as all these may be, each thing is a small part of a much bigger process that’s rarely admired. They all belong inside a world-changing ecosystem of digital hardware and software, spreading into every area of our lives.
Thinking about what is going on behind the scenes is easier if we consider the automobile, also known as “the machine that changed the world.” Cars succeeded through the widespread construction of highways and gas stations. Those things created a global supply chain of steel plants and refineries. Seemingly unrelated things, including suburbs, fast food and drive-time talk radio, arose in the success.
Today’s dominant industrial ecosystem is relentlessly acquiring and processing digital information. It demands newer and better ways of collecting, shipping, and processing data, much the way cars needed better road building. And it’s spinning out its own unseen businesses.
A few recent developments illustrate the new ecosystem. General Electric plans to announce Monday that it has created a “data lake” method of analyzing sensor information from industrial machinery in places like railroads, airlines, hospitals and utilities. G.E. has been putting sensors on everything it can for a couple of years, and now it is out to read all that information quickly.
The company, working with an outfit called Pivotal, said that in the last three months it has looked at information from 3.4 million miles of flights by 24 airlines using G.E. jet engines. G.E. said it figured out things like possible defects 2,000 times as fast as it could before.
The company has to, since it’s getting so much more data. “In 10 years, 17 billion pieces of equipment will have sensors,” said William Ruh, vice president of G.E. software. “We’re only one-tenth of the way there.”
It hardly matters if Mr. Ruh is off by five billion or so. Billions of humans are already augmenting that number with their own packages of sensors, called smartphones, fitness bands and wearable computers. Almost all of that will get uploaded someplace too.
Shipping that data creates challenges. In June, researchers at the University of California, San Diego announced a method of engineering fiber optic cable that could make digital networks run 10 times faster. The idea is to get more parts of the system working closer to the speed of light, without involving the “slow” processing of electronic semiconductors.
“We’re going from millions of personal computers and billions of smartphones to tens of billions of devices, with and without people, and that is the early phase of all this,” said Larry Smarr, drector of the California Institute for Telecommunications and Information Technology, located inside U.C.S.D. “A gigabit a second was fast in commercial networks, now we’re at 100 gigabits a second. A terabit a second will come and go. A petabit a second will come and go.”
In other words, Mr. Smarr thinks commercial networks will eventually be 10,000 times as fast as today’s best systems. “It will have to grow, if we’re going to continue what has become our primary basis of wealth creation,” he said.
Add computation to collection and transport. Last month, U.C. Berkeley’s AMP Lab, created two years ago for research into new kinds of large-scale computing, spun out a company called Databricks, that uses new kinds of software for fast data analysis on a rental basis. Databricks plugs into the one million-plus computer servers inside the global system of Amazon Web Services, and will soon work inside similar-size megacomputing systems from Google and Microsoft.
It was the second company out of the AMP Lab this year. The first, called Mesosphere, enables a kind of pooling of computing services, building the efficiency of even million-computer systems….”