Stefaan Verhulst

Research in the Crowdsourcing Age, a Case Study

Curated on July 12, 2016October 9, 2018 by Stefaan Verhulst

Report by Paul Hitlin (Pew): “How scholars, companies and workers are using Mechanical Turk, a ‘gig economy’ platform, for tasks computers can’t handle

Digital age platforms are providing researchers the ability to outsource portions of their work – not just to increasingly intelligent machines, but also to a relatively low-cost online labor force comprised of humans. These so-called “online outsourcing” services help employers connect with a global pool of free-agent workers who are willing to complete a variety of specialized or repetitive tasks.

Because it provides access to large numbers of workers at relatively low cost, online outsourcing holds a particular appeal for academics and nonprofit research organizations – many of whom have limited resources compared with corporate America. For instance, Pew Research Center has experimented with using these services to perform tasks such as classifying documents and collecting website URLs. And a Google search of scholarly academic literature shows that more than 800 studies – ranging from medical research to social science – were published using data from one such platform, Amazon’s Mechanical Turk, in 2015 alone.¹

The rise of these platforms has also generated considerable commentary about the so-called “gig economy” and the possible impact it will have on traditional notions about the nature of work, the structure of compensation and the “social contract” between firms and workers. Pew Research Center recently explored some of the policy and employment implications of these new platforms in a national survey of Americans.

Proponents say this technology-driven innovation can offer employers – whether companies or academics – the ability to control costs by relying on a global workforce that is available 24 hours a day to perform relatively inexpensive tasks. They also argue that these arrangements offer workers the flexibility to work when and where they want to. On the other hand, some critics worry this type of arrangement does not give employees the same type of protections offered in more traditional work environments – while others have raised concerns about the quality and consistency of data collected in this manner.

A recent report from the World Bank found that the online outsourcing industry generated roughly $2 billion in 2013 and involved 48 million registered workers (though only 10% of them were considered “active”). By 2020, the report predicted, the industry will generate between $15 billion and $25 billion.

Amazon’s Mechanical Turk is one of the largest outsourcing platforms in the United States and has become particularly popular in the social science research community as a way to conduct inexpensive surveys and experiments. The platform has also become an emblem of the way that the internet enables new businesses and social structures to arise.

In light of its widespread use by the research community and overall prominence within the emerging world of online outsourcing, Pew Research Center conducted a detailed case study examining the Mechanical Turk platform in late 2015 and early 2016. The study utilizes three different research methodologies to examine various aspects of the Mechanical Turk ecosystem. These include human content analysis of the platform, a canvassing of Mechanical Turk workers and an analysis of third party data.

The first goal of this research was to understand who uses the Mechanical Turk platform for research or business purposes, why they use it and who completes the work assignments posted there. To evaluate these issues, Pew Research Center performed a content analysis of the tasks posted on the site during the week of Dec. 7-11, 2015.

A second goal was to examine the demographics and experiences of the workers who complete the tasks appearing on the site. This is relevant not just to fellow researchers that might be interested in using the platform, but as a snapshot of one set of “gig economy” workers. To address these questions, Pew Research Center administered a nonprobability online survey of Turkers from Feb. 9-25, 2016, by posting a task on Mechanical Turk that rewarded workers for answering questions about their demographics and work habits. The sample of 3,370 workers contains any number of interesting findings, but it has its limits. This canvassing emerges from an opt-in sample of those who were active on MTurk during this particular period, who saw our survey and who had the time and interest to respond. It does not represent all active Turkers in this period or, more broadly, all workers on MTurk.

Finally, this report uses data collected by the online tool mturk-tracker, which is run by Dr. Panagiotis G. Ipeirotis of the New York University Stern School of Business, to examine the amount of activity occurring on the site. The mturk-tracker data are publically available online, though the insights presented here have not been previously published elsewhere….(More)”

What is Artificial Intelligence?

Curated on July 12, 2016August 3, 2018 by Stefaan Verhulst

Report by Mike Loukides and Ben Lorica: “Defining artificial intelligence isn’t just difficult; it’s impossible, not the least because we don’t really understand human intelligence. Paradoxically, advances in AI will help more to define what human intelligence isn’t than what artificial intelligence is.

But whatever AI is, we’ve clearly made a lot of progress in the past few years, in areas ranging from computer vision to game playing. AI is making the transition from a research topic to the early stages of enterprise adoption. Companies such as Google and Facebook have placed huge bets on AI and are already using it in their products. But Google and Facebook are only the beginning: over the next decade, we’ll see AI steadily creep into one product after another. We’ll be communicating with bots, rather than scripted robo-dialers, and not realizing that they aren’t human. We’ll be relying on cars to plan routes and respond to road hazards. It’s a good bet that in the next decades, some features of AI will be incorporated into every application that we touch and that we won’t be able to do anything without touching an application.

Given that our future will inevitably be tied up with AI, it’s imperative that we ask: Where are we now? What is the state of AI? And where are we heading?

Capabilities and Limitations Today

Descriptions of AI span several axes: strength (how intelligent is it?), breadth (does it solve a narrowly defined problem, or is it general?), training (how does it learn?), capabilities (what kinds of problems are we asking it to solve?), and autonomy (are AIs assistive technologies, or do they act on their own?). Each of these axes is a spectrum, and each point in this many-dimensional space represents a different way of understanding the goals and capabilities of an AI system.

On the strength axis, it’s very easy to look at the results of the last 20 years and realize that we’ve made some extremely powerful programs. Deep Blue beat Garry Kasparov in chess; Watson beat the best Jeopardy champions of all time; AlphaGo beat Lee Sedol, arguably the world’s best Go player. But all of these successes are limited. Deep Blue, Watson, and AlphaGo were all highly specialized, single-purpose machines that did one thing extremely well. Deep Blue and Watson can’t play Go, and AlphaGo can’t play chess or Jeopardy, even on a basic level. Their intelligence is very narrow, and can’t be generalized. A lot of work has gone into usingWatson for applications such as medical diagnosis, but it’s still fundamentally a question-and-answer machine that must be tuned for a specific domain. Deep Blue has a lot of specialized knowledge about chess strategy and an encyclopedic knowledge of openings. AlphaGo was built with a more general architecture, but a lot of hand-crafted knowledge still made its way into the code. I don’t mean to trivialize or undervalue their accomplishments, but it’s important to realize what they haven’t done.

We haven’t yet created an artificial general intelligence that can solve a multiplicity of different kinds of problems. We still don’t have a machine that can listen to recordings of humans for a year or two, and start speaking. While AlphaGo “learned” to play Go by analyzing thousands of games, and then playing thousands more against itself, the same software couldn’t be used to master chess. The same general approach? Probably. But our best current efforts are far from a general intelligence that is flexible enough to learn without supervision, or flexible enough to choose what it wants to learn, whether that’s playing board games or designing PC boards.

Toward General Intelligence

How do we get from narrow, domain-specific intelligence to more general intelligence? By “general intelligence,” we don’t necessarily mean human intelligence; but we do want machines that can solve different kinds of problems without being programmed with domain-specific knowledge. We want machines that can make human judgments and decisions. That doesn’t necessarily mean that AI systems will implement concepts like creativity, intuition, or instinct, which may have no digital analogs. A general intelligence would have the ability to follow multiple pursuits and to adapt to unexpected situations. And a general AI would undoubtedly implement concepts like “justice” and “fairness”: we’re already talking about the impact of AI on the legal system….

It’s easier to think of super-intelligence as a matter of scale. If we can create “general intelligence,” it’s easy to assume that it could quickly become thousands of times more powerful than human intelligence. Or, more precisely: either general intelligence will be significantly slower than human thought, and it will be difficult to speed it up either through hardware or software; or it will speed up quickly, through massive parallelism and hardware improvements. We’ll go from thousand-core GPUs to trillions of cores on thousands of chips, with data streaming in from billions of sensors. In the first case, when speedups are slow, general intelligence might not be all that interesting (though it will have been a great ride for the researchers). In the second case, the ramp-up will be very steep and very fast….(More) (Full Report)”

iStreetWatch

Curated on July 12, 2016August 15, 2018 by Stefaan Verhulst

Welcome to iStreetWatch: “We track racist and xenophobic harassment in public spaces. Incidents of racist and anti-migrant abuse are becoming ever more public and ever more prolific. If you have witnessed or experienced racist or xenophobic harassment, please submit your experience here.

We aim to:

Make these now everyday incidents visible to a wider community.
Help people at risk map which areas are safer to be in.
Collect data over time to help monitor the correlation between these incidents and inflammatory speech from the media and politicians.

Everyone has the right to feel safe on the street. For many people, leaving their house means risking verbal and sometimes physical abuse. This site has been created in response to the rise in hate crime following the referendum result. We all have a role to play in making our streets safe for everyone.

If you witness racist or xenophobic harassment, there are other things you can do:

Say something: Show you won’t accept this behaviour on our streets.
Film it: Providing evidence can be key to making an arrest or charge.
Report abuse to the police: There are laws to protect people from abuse and harassment which should be enforced
Be safe: Don’t intervene if you think you will escalate the situation….(More)”

Postal big data: Global flows as proxy indicators for national wellbeing

Curated on July 11, 2016October 9, 2018 by Stefaan Verhulst

Data Driven Journalism: “A new project has developed an innovative means to approximate socioeconomic indicators by analyzing the network of international postal flows.

The project used 14 million aggregated electronic postal records from 187 countries collected by the Universal Postal Union over a four-year period (2010-2014) to create an international network showing the way post flows around the world.

In addition, the project builds upon previous research efforts using global flow networks, derived from the five following open data sources:

For each network, a country’s degree of connectivity for incoming and outgoing flows was quantified using the Jaccard coefficient and Spearman’s rank correlation coefficient….

To understand these connections in the context of socioeconomic indicators, the researchers then compared these positions to the values of GDP, Life expectancy, Corruption Perception Index, Internet penetration rate, Happiness index, Gini index, Economic Complexity Index, Literacy, Poverty, CO2 emissions, Fixed phone line penetration, Mobile phone users, and the Human Development Index.

Image: Spearman rank correlations between global flow network degrees and socioeconomic indicators (CC BY 4.0).

From this analysis, the researchers revealed that:

The best-performing degree, in terms of consistently high performance across indicators is the global degree, suggesting that looking at how well connected a country is in the global multiplex can be more indicative of its socioeconomic profile as a whole than looking at single networks.
GDP per capita and life expectancy are most closely correlated with the global degree, closely followed by the postal, trade and IP weighed degrees – indicative of a relationship between national wealth and the flow of goods and information.
Similarly to GDP, the rate of poverty of a country is best represented by the global degree, followed by the postal degree. The negative correlation indicates that the more impoverished a country is, the less well connected it is to the rest of the world.
Low human development (high rank) is most highly negatively correlated with the global degree, followed by the postal, trade and IP degrees. This shows that high human development (low rank) is associated with high global connectivity and activity in terms of incoming and outgoing flows of information and goods. ….Read the fully study here.”

Human Factors in Big Data

Curated on July 10, 2016August 3, 2018 by Stefaan Verhulst

Dossier by Jenny de Boer, Marc Steen, Maurice van Beurden, Alexander Toet, Susanne Tak, Jan van Erp en Ward Venrooij: “Much attention in Big Data is given to the technical challenges that arise. However, Human Factor challenges and opportunities are also relevant. Big Data provides human factor specialists with a new and rich way of data collection and data analysis of human factors patterns. Big Data also has implications on how human factors knowledge can be applied (and developed) to develop Big Data services that people are willing to use. How people are critical to the success of big data: This first article highlights the role of people in developing Data Driven Innovations. Visualising Uncertainty: In order to interpret Big Data, visualization of data is of great importance. This article gives insight in how to visualize predictive information based on Big Data. Measuring dashboard performance: In the third article, the development of a framework is described, that can be used to develop and test the quality of data presentation in dashboards….(More) “

Solving All the Wrong Problems

Curated on July 10, 2016May 29, 2019 by Stefaan Verhulst

Allison Arieff in the New York Times: “Every day, innovative companies promise to make the world a better place. Are they succeeding? Here is just a sampling of the products, apps and services that have come across my radar in the last few weeks:

A service that sends someone to fill your car with gas.

A service that sends a valet on a scooter to you, wherever you are, to park your car.

A service that will film anything you desire with a drone….

We are overloaded daily with new discoveries, patents and inventions all promising a better life, but that better life has not been forthcoming for most. In fact, the bulk of the above list targets a very specific (and tiny!) slice of the population. As one colleague in tech explained it to me recently, for most people working on such projects, the goal is basically to provide for themselves everything that their mothers no longer do….When everything is characterized as “world-changing,” is anything?

Clay Tarver, a writer and producer for the painfully on-point HBO comedy “Silicon Valley,” said in a recent New Yorker article: “I’ve been told that, at some of the big companies, the P.R. departments have ordered their employees to stop saying ‘We’re making the world a better place,’ specifically because we have made fun of that phrase so mercilessly. So I guess, at the very least, we’re making the world a better place by making these people stop saying they’re making the world a better place.”

O.K., that’s a start. But the impulse to conflate toothbrush delivery with Nobel Prize-worthy good works is not just a bit cultish, it’s currently a wildfire burning through the so-called innovation sector. Products and services are designed to “disrupt” market sectors (a.k.a. bringing to market things no one really needs) more than to solve actual problems, especially those problems experienced by what the writer C. Z. Nnaemeka has described as “the unexotic underclass” — single mothers, the white rural poor, veterans, out-of-work Americans over 50 — who, she explains, have the “misfortune of being insufficiently interesting.”

If the most fundamental definition of design is to solve problems, why are so many people devoting so much energy to solving problems that don’t really exist? How can we get more people to look beyond their own lived experience?

In “Design: The Invention of Desire,” a thoughtful and necessary new book by the designer and theorist Jessica Helfand, the author brings to light an amazing kernel: “hack,” a term so beloved in Silicon Valley that it’s painted on the courtyard of the Facebook campus and is visible from planes flying overhead, is also prison slang for “horse’s ass carrying keys.”

To “hack” is to cut, to gash, to break. It proceeds from the belief that nothing is worth saving, that everything needs fixing. But is that really the case? Are we fixing the right things? Are we breaking the wrong ones? Is it necessary to start from scratch every time?…

Ms. Helfand calls for a deeper embrace of personal vigilance: “Design may provide the map,” she writes, “but the moral compass that guides our personal choices resides permanently within us all.”

Can we reset that moral compass? Maybe we can start by not being a bunch of hacks….(More)”

Smart Cities – International Case Studies

Curated on July 10, 2016August 15, 2018 by Stefaan Verhulst

“These case studies were developed by the Inter-American Development Bank (IDB), in association with the Korea Research Institute for Human Settlements (KRIHS).

Anyang, Korea Anyang, a 600,000 population city near Seoul is developing international recognition on its smart city project that has been implemented incrementally since 2003. This initiative began with the Bus Information System to enhance citizen’s convenience at first, and has been expanding its domain into wider Intelligent Transport System as well as crime and disaster prevention in an integrated manner. Anyang is considered a benchmark for smart city with a 2012 Presidential Award in Korea and receives large number of international visits. Anyang’s Integrated Operation and Control Center (IOCC) acts as the platform that gathers, analyzes and distributes information for mobility, disasters management and crime. Anyang is currently utilizing big data for policy development and is continuing its endeavor to expand its smart city services into areas such as waste and air quality management. Download Anyang case study

Medellín, Colombia Medellin is a city that went from being known for its security problems to being an international referent of technological and social innovation, urban transformation, equity, and citizen participation. This report shows how Medellin has implemented a series of strategies that have made it a smart city that is developing capacity and organic structure in the entities that control mobility, the environment, and security. In addition, these initiatives have created mechanisms to communicate and interact with citizens in order to promote continuous improvement of smart services.

Through the Program “MDE: Medellin Smart City,” Medellin is implementing projects to create free Internet access zones, community centers, a Mi-Medellin co-creation portal, open data, online transactions, and other services. Another strategy is the creation of the Smart Mobility System which, through the use of technology, has achieved a reduction in the number of accidents, improvement in mobility, and a reduction in incident response time. Download Medellin case study

Namyangju, Korea…

Orlando, U.S.…

Pangyo, Korea…

Rio de Janeiro, Brazil…

Santander, España…

Singapore…

Songdo, Korea…

Tel Aviv, Israel… (More)”

Bridging data gaps for policymaking: crowdsourcing and big data for development

Curated on July 8, 2016August 3, 2018 by Stefaan Verhulst

Anthony Swan for the DevPolicyBlog: “…By far the biggest innovation in data collection is the ability to access and analyse (in a meaningful way) user-generated data. This is data that is generated from forums, blogs, and social networking sites, where users purposefully contribute information and content in a public way, but also from everyday activities that inadvertently or passively provide data to those that are able to collect it.

User-generated data can help identify user views and behaviour to inform policy in a timely way rather than just relying on traditional data collection techniques (census, household surveys, stakeholder forums, focus groups, etc.), which are often cumbersome, very costly, untimely, and in many cases require some form of approval or support by government.

It might seem at first that user-generated data has limited usefulness in a development context due to the importance of the internet in generating this data combined with limited internet availability in many places. However, U-Report is one example of being able to access user-generated data independent of the internet.

U-Report was initiated by UNICEF Uganda in 2011 and is a free SMS based platform where Ugandans are able to register as “U-Reporters” and on a weekly basis give their views on topical issues (mostly related to health, education, and access to social services) or participate in opinion polls. As an example, Figure 1 shows the result from a U-Report poll on whether polio vaccinators came to U-Reporter houses to immunise all children under 5 in Uganda, broken down by districts. Presently, there are more than 300,000 U-Reporters in Uganda and more than one million U-Reporters across 24 countries that now have U-Report. As an indication of its potential impact on policymaking,UNICEF claims that every Member of Parliament in Uganda is signed up to receive U-Report statistics.

Figure 1: U-Report Uganda poll results

U-Report and other platforms such as Ushahidi (which supports, for example, I PAID A BRIBE, Watertracker, election monitoring, and crowdmapping) facilitate crowdsourcing of data where users contribute data for a specific purpose. In contrast, “big data” is a broader concept because the purpose of using the data is generally independent of the reasons why the data was generated in the first place.

Big data for development is a new phrase that we will probably hear a lot more (see here [pdf] and here). The United Nations Global Pulse, for example, supports a number of innovation labs which work on projects that aim to discover new ways in which data can help better decision-making. Many forms of “big data” are unstructured (free-form and text-based rather than table- or spreadsheet-based) and so a number of analytical techniques are required to make sense of the data before it can be used.

Measures of Twitter activity, for example, can be a real-time indicator of food price crises in Indonesia [pdf] (see Figure 2 below which shows the relationship between food-related tweet volume and food inflation: note that the large volume of tweets in the grey highlighted area is associated with policy debate on cutting the fuel subsidy rate) or provide a better understanding of the drivers of immunisation awareness. In these examples, researchers “text-mine” Twitter feeds by extracting tweets related to topics of interest and categorising text based on measures of sentiment (positive, negative, anger, joy, confusion, etc.) to better understand opinions and how they relate to the topic of interest. For example, Figure 3 shows the sentiment of tweets related to vaccination in Kenya over time and the dates of important vaccination related events.

Figure 2: Plot of monthly food-related tweet volume and official food price statistics

Figure 3: Sentiment of vaccine related tweets in Kenya

Another big data example is the use of mobile phone usage to monitor the movement of populations in Senegal in 2013. The data can help to identify changes in the mobility patterns of vulnerable population groups and thereby provide an early warning system to inform humanitarian response effort.

The development of mobile banking too offers the potential for the generation of a staggering amount of data relevant for development research and informing policy decisions. However, it also highlights the public good nature of data collected by public and private sector institutions and the reliance that researchers have on them to access the data. Building trust and a reputation for being able to manage privacy and commercial issues will be a major challenge for researchers in this regard….(More)”

Mapping and Comparing Responsible Data Approaches

Curated on July 6, 2016July 19, 2019 by Stefaan Verhulst

New report by Jos Berens, Ulrich Mans and Stefaan Verhulst: “Recent years have witnessed something of a sea-change in the way humanitarian organizations consider and use data. Growing awareness of the potential of data has led to new enthusiasm and new, innovative applications that seek to respond to and mitigate crises in fresh ways. At the same time, it has become apparent that the potential benefits are accompanied by risks. A new framework is needed that can help balance the benefits and risks, and that can aid humanitarian organizations and others (e.g., policymakers) develop a more responsible approach to data collection and use in their efforts to combat natural and man-made crises around the world. …

The report we are releasing today, “Mapping and Comparing Responsible Data Approaches”, attempts to guide the first steps toward such a framework by learning from current approaches and principles. It is the outcome of a joint research project commissioned by UNOCHA and conducted in collaboration between the GovLab at NYU and Leiden University. In an effort to better understand the landscape, we have considered existing data use policies and principles from 17 organizations. These include 7 UN agencies, 7 International Organizations, 2 government agencies and 1 research institute. Our study of these organizations’ policies allowed us to extract a number of key takeaways that, together, amount to something like a roadmap for responsible data use for any humanitarian organization considering using data in new ways.

We began our research by closely mapping the existing responsible data use policies. To do this, we developed a template with eight broad themes that determines the key ingredients of responsible data framework. This use of a consistent template across organizations permits us to study and compare the 17 data use policies in a structured and systematic manner. Based on this template, we were able to extract 7 key takeaways for what works best when using data in a humanitarian context – presented in the conclusion to the paper being released today. They are designed to be broad enough to be broadly applicable, yet specific enough to be operational and actually usable….(More)”