Book edited by Alain Samson. Introduction by Colin Camerer: “The goal of science is to accumulate knowledge, full stop. In my opinion, there is a lot of leakage in how we currently do this. The reproducibility “upgrade” (a term I prefer to “crisis”) going on in many areas of science is an example of trying to minimize leakage. Solid accumulation depends on not getting led too far or frequently astray by false positives which do not reproduce. A good infrastructure for rapidly evaluating and cumulating results is of special use for “hurry-up” social science. For example, as I write this there are probably hundreds of social science studies being done about COVID-19. It is essentially impossible for all those scientists to know what the other scientists are doing. There will be duplication and poorly designed
studies. (It is often said in design that everyone wants cheap, fast, and good. But you can only have two.)
When studies are written and circulated in preprints, a lot of null effects won’t be written up. Which studies will get the most attention? It will be a scrum of social media, presenting at seminars, slow and fast reviewing paces. The one thing that would undoubtedly be most useful—a giant dashboard summarizing weekly progress on each of those hundreds of studies—does not exist. This is a failure of good informatics.
Behavioral economics is accumulating knowledge about how different kinds of nudges influence behavior at a rapid pace. The challenge is that carefully assessing what an entire body of knowledge is telling us is actually quite difficult and is under-rewarded (by academic incentives). A lot of academic publishing, and similar career concerns within government or NGOs, depend on creativity and doing something new. This creates an incentive to exaggerate the novelty of one’s contribution compared to what is known from past studies….(More)”.