Excerpt by By Beth Simone Noveck: “Social scientists who either run experiments or conduct systematic reviews tend to be fervent proponents of the value of RCTs. But that evidentiary hierarchy—what some people call the “RCT industrial complex”—may actually lead us to discount workable solutions just because there is no accompanying RCT.
A trawl of the solution space shows that successful interventions developed by entrepreneurs in business, philanthropy, civil society, social enterprise, or business schools who promote and study open innovation, often by developing and designing competitions to source ideas, often come from more varied places. Uncovering these exciting social innovations lays bare the limitations of confining a definition of what works only to RCTs.
Many more entrepreneurial and innovative solutions are simply not tested with an RCT and are not the subject of academic study. As one public official said to me, you cannot saddle an entrepreneur with having to do a randomized controlled trial (RCT), which they do not have the time or know-how to do. They are busy helping real people, and we have to allow them “to get on with it.”
For example, MIT Solve, which describes itself as a marketplace for socially impactful innovation designed to identify lasting solutions to the world’s most pressing problems. It catalogs hundreds of innovations in use around the world, like Faircap, a chemical-free water filter used in Mozambique, or WheeLog!, an application that enables individuals and local governments to share accessibility information in Tokyo.
Research funding is also too limited (and too slow) for RCTs to assess every innovation in every domain. Many effective innovators do not have the time, resources, or know-how to partner with academic researchers to conduct a study, or they evaluate projects by some other means.
There are also significant limitations to RCTs. For a start, systematic evidence reviews are quite slow, frequently taking upward of two years, and despite published standards for review, there is a lack of transparency. Faster approaches are important. In addition, many solutions that have been tested with an RCT clearly do not work. Interestingly, the first RCT in an area tends to produce an inflated effect size….(More)”.