Index: Measuring Impact with Evidence

Curated on Posted on by Shruti Sannon

The Living Library Index – inspired by the Harper’s Index – provides important statistics and highlights global trends in governance innovation. This installment focuses on measuring impact with evidence and was originally published in 2013.

United States

  • Amount per $100 of government spending that is backed by evidence that the money is being spent wisely: less than $1
  • Number of healthcare treatments delivered in the U.S. that lack evidence of effectiveness: more than half
  • How much of total U.S. healthcare expenditure is spent to determine what works: less than 0.1 percent
  • Number of major U.S. federal social programs evaluated since 1990 using randomized experiments and found to have “weak or no positive effects”: 9 out of 10
  • Year the Coalition for Evidence-Based Policy was set up to work with federal policymakers to advance evidence-based reforms in major U.S. social programs: 2001
  • Year the Program Assessment Rating Tool (PART) was introduced by President Bush’s Office of Management and Budget (OMB): 2002
    • Out of about 1,000 programs assessed, number found to be effective in 2008: 19%
    • Percentage of programs that could not be assessed due to insufficient data: 17%
    • Amount spent on the Even Start Family Literacy Program, rated ineffective by PART, over the life of the Bush administration: more than $1 billion
  •  Year Washington State legislature began using Washington State Institute for Public Policy’s estimates on how “a portfolio of evidence-based and economically sound programs . . . could affect the state’s crime rate, the need to build more prisons, and total criminal-justice spending”: 2007
    • Amount invested by legislature in these programs: $48 million
    • Amount saved by the legislature: $250 million
  • Number of U.S. States in a pilot group working to adapt The Pew-MacArthur Results First Initiative, based on the Washington State model, to make performance-based policy decisions: 14
  • Net savings in health care expenditure by using the Transitional Care Model, which meets the Congressionally-based Top Tier Evidence Standard: $4,000 per patient
  • Number of states that conducted “at least some studies that evaluated multiple program or policy options for making smarter investments of public dollars” between 2008-2011: 29
  • Number of states that reported that their cost-benefit analysis influenced policy decisions or debate: 36
  • Date the Office of Management and Budget issued a memorandum proposing new evaluations and advising agencies to include details on determining effectiveness of their programs, link disbursement to evidence, and support evidence-based initiatives: 2007
  • Percentage increase in resources for innovation funds that use a tiered model for evidence, according to the President’s FY14 budget: 44% increase
  • Amount President Obama proposed in his FY 2013 budget to allocate in existing funding to Performance Partnerships “in which states and localities would be given the flexibility to propose better ways to combine federal resources in exchange for greater accountability for results”:  $200 million
  • Amount of U.S. federal program funding that Harvard economist Jeffrey Liebman suggests be directed towards evaluations of outcomes: 1%
  • Amount of funding the City of New York has committed for evidence-based research and development initiatives through its Center for Economic Opportunity: $100 million a year

Internationally

  • How many of the 30 OECD countries in 2005-6 have a formal requirement by law that the benefits of regulation justify the costs: half
    • Number of 30 OECD member countries in 2008 that reported quantifying benefits to regulations: 16
    • Those who reported quantifying costs: 24
  • How many members make up the Alliance for Useful Evidence, a network that “champion[s]  evidence, the opening up of government data for interrogation and use, alongside the sophistication in research methods and their applications”: over 1,000
  • Date the UK government, the ESRC and the Big Lottery Fund announced plans to create a network of ‘What Works’ evidence centres: March 2013
  • Core funding for the What Works Centre for Local Economic Growth: £1m p.a. over an initial three year term
  • How many SOLACE Summit members in 2012 were “very satisfied” with how Research and Intelligence resources support evidence-based decision-making: 4%
    • Number of areas they identified for improving evidence-based decision-making: 5
    • Evaluation of the impact of past decisions: 46% of respondents
    • Benchmarking data with other areas: 39%
    • assessment of options available: 33% 
    • how evidence is presented: 29% 
    • Feedback on public engagement and consultation: 25%
  •  Number of areas for improvement for Research and Intelligence staff development identified at the SOLACE Summit: 6
    • Strengthening customer insight and data analysis: 49%
    • Impact evaluation: 48%
    • Strategic/corporate thinking/awareness: 48%
    • Political acumen: 46%
    • Raising profile/reputation of the council for evidence-based decisions: 37%
    • Guidance/mentoring on use of research for other officers: 25%

Sources