Topics: crowdsourcing

Paper by Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, and Philip E. Tetlock: “Human forecasting accuracy in practice relies on the ‘wisdom of the crowd’ effect, in which predictions about future events are significantly improved by aggregating across a crowd of individual forecasters. Past work on the forecasting ability of large language models (LLMs) suggests that frontier LLMs, as individual forecasters, underperform compared to the gold standard of a human crowd forecasting tournament aggregate. In Study 1, we expand this research by using an LLM ensemble approach consisting of a crowd of twelve LLMs. We compare the aggregated LLM predictions on 31 binary questions to that of a crowd of 925 human forecasters from a three-month forecasting tournament. Our preregistered main analysis shows that the LLM crowd outperforms a simple no-information benchmark and is not statistically different from the human crowd. In exploratory analyses, we find that these two approaches are equivalent with respect to medium-effect-size equivalence bounds. We also observe an acquiescence effect, with mean model predictions being significantly above 50%, despite an almost even split of positive and negative resolutions. Moreover, in Study 2, we test whether LLM predictions (of GPT-4 and Claude 2) can be improved by drawing on human cognitive output. We find that both models’ forecasting accuracy benefits from exposure to the median human prediction as information, improving accuracy by between 17% and 28%: though this leads to less accurate predictions than simply averaging human and machine forecasts. Our results suggest that LLMs can achieve forecasting accuracy rivaling that of human crowd forecasting tournaments: via the simple, practically applicable method of forecast aggregation. This replicates the ‘wisdom of the crowd’ effect for LLMs, and opens up their use for a variety of applications throughout society…(More)”.

Paper by Leonard Boussioux, Jacqueline Lane, Miaomiao Zhang, Vladimir Jacimovic, and Karim Lakhani: “This study investigates the capability of generative artificial intelligence (AI) in creating innovative business solutions compared to human crowdsourcing methods. We initiated a crowdsourcing challenge focused on sustainable, circular economy business opportunities. The challenge attracted a diverse range of solvers from a myriad of countries and industries. Simultaneously, we employed GPT-4 to generate AI solutions using three different prompt levels, each calibrated to simulate distinct human crowd and expert personas. 145 evaluators assessed a randomized selection of 10 out of 234 human and AI solutions, a total of 1,885 evaluator-solution pairs. Results showed comparable quality between human and AI-generated solutions. However, human ideas were perceived as more novel, whereas AI solutions delivered better environmental and financial value. We use natural language processing techniques on the rich solution text to show that although human solvers and GPT-4 cover a similar range of industries of application, human solutions exhibit greater semantic diversity. The connection between semantic diversity and novelty is stronger in human solutions, suggesting differences in how novelty is created by humans and AI or detected by human evaluators. This study illuminates the potential and limitations of both human and AI crowdsourcing to solve complex organizational problems and sets the groundwork for a possible integrative human-AI approach to problem-solving…(More)”.