Crowdsourcing hypothesis tests: making transparent how design choices shape research results

Curated on Posted on by Stefaan Verhulst

Paper by J.F. Landy and Leonid Tiokhin: “To what extent are research results influenced by subjective decisions that scientists make as they design studies?

Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams rendered statistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses.

Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim….(More)”.

Crowdsourcing Geographic Information

Curated on Posted on by Stefaan Verhulst

Paper by Dieter Pfoser: “The crowdsourcing of geographic information addresses the collection of geospatial data contributed by non-expert users and the aggregation of these data into meaningful geospatial datasets. While crowdsourcing generally implies a coordinated bottom-up
grass-roots effort to contribute information, in the context of geospatial data the term volunteered geographic information (VGI) specifically refers to a dedicated collection effort inviting non-expert users to contribute. A prominent example here is the OpenStreetMap effort focusing on map datasets. Crowdsourcing geospatial data is an evolving research area that covers efforts ranging from mining GPS tracking data to using social media content to profile population dynamics…(More)”.

Mapping Wikipedia

Curated on Posted on by Stefaan Verhulst

Michael Mandiberg at The Atlantic: “Wikipedia matters. In a time of extreme political polarization, algorithmically enforced filter bubbles, and fact patterns dismissed as fake news, Wikipedia has become one of the few places where we can meet to write a shared reality. We treat it like a utility, and the U.S. and U.K. trust it about as much as the news.

But we know very little about who is writing the world’s encyclopedia. We do know that just because anyone can edit, doesn’t mean that everyone does: The site’s editors are disproportionately cis white men from the global North. We also know that, as with most of the internet, a small number of the editors do a large amount of the editing. But that’s basically it: In the interest of improving retention, the Wikimedia Foundation’s own research focuses on the motivations of people who do edit, not on those who don’t. The media, meanwhile, frequently focus on Wikipedia’s personality stories, even when covering the bigger questions. And Wikipedia’s own culture pushes back against granular data harvesting: The Wikimedia Foundation’s strong data-privacy rules guarantee users’ anonymity and limit the modes and duration of their own use of editor data.

But as part of my research in producing Print Wikipedia, I discovered a data set that can offer an entry point into the geography of Wikipedia’s contributors. Every time anyone edits Wikipedia, the software records the text added or removed, the time of the edit, and the username of the editor. (This edit history is part of Wikipedia’s ethos of radical transparency: Everyone is anonymous, and you can see what everyone is doing.) When an editor isn’t logged in with a username, the software records that user’s IP address. I parsed all of the 884 million edits to English Wikipedia to collect and geolocate the 43 million IP addresses that have edited English Wikipedia. I also counted 8.6 million username editors who have made at least one edit to an article.

The result is a set of maps that offer, for the first time, insight into where the millions of volunteer editors who build and maintain English Wikipedia’s 5 million pages are—and, maybe more important, where they aren’t….

Like the Enlightenment itself, the modern encyclopedia has a history entwined with colonialism. Encyclopédie aimed to collect and disseminate all the world’s knowledge—but in the end, it could not escape the biases of its colonial context. Likewise, Napoleon’s Description de l’Égypte augmented an imperial military campaign with a purportedly objective study of the nation, which was itself an additional form of conquest. If Wikipedia wants to break from the past and truly live up to its goal to compile the sum of all human knowledge, it requires the whole world’s participation….(More)”.

Wisdom or Madness? Comparing Crowds with Expert Evaluation in Funding the Arts

Curated on Posted on by Stefaan Verhulst

Paper by Ethan R. Mollick and Ramana Nanda: “In fields as diverse as technology entrepreneurship and the arts, crowds of interested stakeholders are increasingly responsible for deciding which innovations to fund, a privilege that was previously reserved for a few experts, such as venture capitalists and grant‐making bodies. Little is known about the degree to which the crowd differs from experts in judging which ideas to fund, and, indeed, whether the crowd is even rational in making funding decisions. Drawing on a panel of national experts and comprehensive data from the largest crowdfunding site, we examine funding decisions for proposed theater projects, a category where expert and crowd preferences might be expected to differ greatly.

We instead find significant agreement between the funding decisions of crowds and experts. Where crowds and experts disagree, it is far more likely to be a case where the crowd is willing to fund projects that experts may not. Examining the outcomes of these projects, we find no quantitative or qualitative differences between projects funded by the crowd alone, and those that were selected by both the crowd and experts. Our findings suggest that crowdfunding can play an important role in complementing expert decisions, particularly in sectors where the crowds are end users, by allowing projects the option to receive multiple evaluations and thereby lowering the incidence of “false negatives.”…(More)”.

Hyperconnected, receptive and do-it-yourself city. An investigation into the European imaginary of crowdsourcing for urban governance

Curated on Posted on by Stefaan Verhulst

Paper by Chiara Certoma, Filippo Corsini and MarcoFrey: “This paper critically explores the construction and diffusion of the socio-technical imaginary of crowdsourcing for public governance in Europe via a quali-quantitative analysis of academic publications, research and innovation projects funded by the European Commission (EC) and local initiatives. Building upon the increasing narrative of digital social participation that describes crowdsourcing processes as short ways towards democratisation of public decision-making processes, our research describes the trends and threats associated with the “hyperconnected city” imaginary advanced by (part of) scholarly research and EC policy documents and projects.

We show how, while these last describe digital-supported participation processes as (at least potentially) able to bootstrap an open governance agenda, local urban initiatives suggest the need to question this technology-optimistic imaginary.

A critical analysis of crowdsourcing for public governance prototyped and piloted in some European cities makes it evident that at local level, alternative imaginaries are emerging. We describe them in this paper as the “receptive city” (often adopted by public institutions and administration), and the “do-it-yourself city” (referring to the critical perspective of (digital) social activists) imaginaries, both emerging from local-based experiences and debates; and clarify their convergence and divergence how these differs from the above-mentioned “hyperconnected city” imaginary prefigured by EC guidelines.

The conclusive section further expands the analysis prefiguring future research possibilities promises in terms of local experiences influencing the future internet for society and digital agenda for Europe….(More)”.

Incentive Competitions and the Challenge of Space Exploration

Curated on Posted on by Stefaan Verhulst

Article by Matthew S. Williams: “Bill Joy, the famed computer engineer who co-founded Sun Microsystems in 1982, once said, “No matter who you are, most of the smartest people work for someone else.” This has come to be known as “Joy’s Law” and is one of the inspirations for concepts such as “crowdsourcing”.

Increasingly, government agencies, research institutions, and private companies are looking to the power of the crowd to find solutions to problems. Challenges are created and prizes offered – that, in basic terms, is an “incentive competition.”

The basic idea of an incentive competition is pretty straightforward. When confronted with a particularly daunting problem, you appeal to the general public to provide possible solutions and offer a reward for the best one. Sounds simple, doesn’t it?

But in fact, this concept flies in the face of conventional problem-solving, which is for companies to recruit people with knowledge and expertise and solve all problems in-house. This kind of thinking underlies most of our government and business models, but has some significant limitations….

Another benefit to crowdsourcing is the way it takes advantage of the exponential growth in human population in the past few centuries. Between 1650 and 1800, the global population doubled, to reach about 1 billion. It took another one-hundred and twenty years (1927) before it doubled again to reach 2 billion.

However, it only took fifty-seven years for the population to double again and reach 4 billion (1974), and just fifteen more for it to reach 6 billion. As of 2020, the global population has reached 7.8 billion, and the growth trend is expected to continue for some time.

This growth has paralleled another trend, the rapid development of new ideas in science and technology. Between 1650 and 2020, humanity has experienced multiple technological revolutions, in what is a comparatively very short space of time….(More)”.

The Role of Crowdsourcing in the Healthcare Industry

Curated on Posted on by Stefaan Verhulst

Chapter by Kabir C. Sen: “The twenty first century has seen the advent of technical advances in storage, transmission and analysis of information. This has had a profound impact on the field of medicine. However, notwithstanding these advances, various obstacles remain in the world regarding the improvement of human lives through the provision of better health care. The obstacles emanate from the demand (i.e., the problem) as well as the supply (i.e., the solution) side. In some cases, the nature of the problems might not have been correctly identified. In others, a solution to a problem could be known only to a small niche of the global population. Thus, from the demand perspective, the variety of health care issues can range from the quest for a cure for a rare illness to the inability to successfully implement verifiable preventive measures for a disease that affects pockets of the global population. Alternatively, from the supply perspective, the approach to a host of health issues might vary because of fundamental differences in both medical philosophies and organizational policies.

In many instances, effective solutions to health care problems are lacking because of inadequate global knowledge about the particular disease. Alternatively, in other cases, a solution might exist but the relevant knowledge about it might only be available to selected pockets of the global medical community. Sometimes, the barriers to the transfer of knowledge might have their root causes in ignorance or prejudice about the initiator of the cure or solution. However, the advent of information technology has now provided an opportunity for individuals located at different geographical locations to collaborate on solutions to various problems. These crowdsourcing projects now have the potential to extract the “wisdom of crowds” for tackling problems which previously could not be solved by a group of experts (Surowiecki, 2014). Anecdotal evidence suggests that crowdsourcing has achieved some success in providing solutions for a rare medical disease (Arnold, 2014). This chapter discusses crowdsourcing’s potential to solve medical problems by designing a framework to evaluate its promises and suggest recommended future paths of actions….(More)”.

Unregulated Health Research Using Mobile Devices: Ethical Considerations and Policy Recommendations

Curated on Posted on by Stefaan Verhulst

Paper by Mark A. Rothstein et al: “Mobile devices with health apps, direct-to-consumer genetic testing, crowd-sourced information, and other data sources have enabled research by new classes of researchers. Independent researchers, citizen scientists, patient-directed researchers, self-experimenters, and others are not covered by federal research regulations because they are not recipients of federal financial assistance or conducting research in anticipation of a submission to the FDA for approval of a new drug or medical device. This article addresses the difficult policy challenge of promoting the welfare and interests of research participants, as well as the public, in the absence of regulatory requirements and without discouraging independent, innovative scientific inquiry. The article recommends a series of measures, including education, consultation, transparency, self-governance, and regulation to strike the appropriate balance….(More)”.

Unleashing the Crowd: Collaborative Solutions to Wicked Business and Societal Problems

Curated on Posted on by Stefaan Verhulst

Book by Ann Majchrzak and Arvind Malhotra: “This book disrupts the way practitioners and academic scholars think about crowds, crowdsourcing, innovation, and new organizational forms in this emerging period of ubiquitous access to the internet. The authors argue that the current approach to crowdsourcing unnecessarily limits the crowd to offering ideas, locking out those of us with knowledge about a problem.  They use data from 25 case studies of flash crowds — anonymous strangers answering online announcements to participate in a 7-10 day innovation challenge — half of whom were unleashed from the limitations of focusing on ideas.  Yet, these crowds were able to develop new business models, new product lines, and offer useful solutions to global problems in fields as diverse as health care insurance, software development, and societal change. This book, which offers a theory of collective production of innovative solutions explaining the practices that the crowds organically followed, will revolutionize current assumptions about how innovation and crowdsourcing should be managed for commercial as well as societal purposes….(More)”.

CROWD4EMS: a Crowdsourcing Platform for Gathering and Geolocating Social Media Content in Disaster Response

Curated on Posted on by Stefaan Verhulst

Paper by Ravi Shankar et al” Increase in access to mobile phone devices and social media networks has changed the way people report and respond to disasters. Community-driven initiatives such as Stand By Task Force (SBTF) or GISCorps have shown great potential by crowdsourcing the acquisition, analysis, and geolocation of social media data for disaster responders. These initiatives face two main challenges: (1) most of social media content such as photos and videos are not geolocated, thus preventing the information to be used by emergency responders, and (2) they lack tools to manage volunteers contributions and aggregate them in order to ensure high quality and reliable results. This paper illustrates the use of a crowdsourcing platform that combines automatic methods for gathering information from social media and crowdsourcing techniques, in order to manage and aggregate volunteers contributions. High precision geolocation is achieved by combining data mining techniques for estimating the location of photos and videos from social media, and crowdsourcing for the validation and/or improvement of the estimated location.

The evaluation of the proposed approach is carried out using data related to the Amatrice Earthquake in 2016, coming from Flickr, Twitter and Youtube. A common data set is analyzed and geolocated by both the volunteers using the proposed platform and a group of experts. Data quality and data reliability is assessed by comparing volunteers versus experts results. Final results are shown in a web map service providing a global view of the information social media provided about the Amatrice Earthquake event…(More)”.