1. Are well-designed and implemented; and
   
2. Have significant policy implications-- because they show, for example, that (a) a social intervention has an important effect on life outcomes and, preferably, can be readily replicated at modest cost; or (b) a widely-used intervention has little or no effect.

This site's criteria

To determine whether an RCT is well-designed and implemented, we use the criteria in the U.S. Office of Management and Budget (OMB) document What Constitutes Strong Evidence of a Program's Effectiveness (see appendix, pp 14-16), including such items as the following.

(Regarding the overall study design:)

• Adequate sample size -- one large enough to detect meaningful effects of the intervention.
  
• Random assignment conducted at the appropriate level -- either groups (e.g., classrooms, housing projects), or individuals (e.g., students, housing tenants), or both.
  
• Preferably, evaluation of the intervention in the real-world community settings and conditions where it would normally be implemented.

(Regarding whether the intervention and control groups remained equivalent during the study:)

• Few or no systematic differences between the intervention and control groups prior to the intervention (e.g., in age, sex, income, education).
  
• Few or no control group members who participated in the intervention, or otherwise benefited from it (i.e., there is minimal “cross-over” or “contamination” of controls).
  
• Outcome data collected in the same way, and at the same time, from the intervention and control groups.
  
• Outcome data obtained for a high proportion of sample members originally randomized (i.e., there is low sample “attrition”).
  
• Sample members retained in the original group to which they were randomly assigned, when analyzing study outcomes (i.e., an “intention-to-treat” analysis). This is done even for intervention group members who fail to participate in or complete the intervention.
  
• If needed, “placebo” treatment for the control group (i.e., an ineffectual but harmless treatment, to ensure all sample members believe they are receiving treatment).

(Regarding the study's outcome measures:)

• “Valid” outcome measures (i.e., measures that are highly correlated with the true outcomes that the intervention seeks to affect) -- preferably well-established tests and/or objective, real-world measures (e.g., arrest rates for a crime intervention).
  
• Outcome measures that are of policy or practical importance (e.g., for a pregnancy prevention program, actual pregnancies or unprotected sex, not just attitudes toward sex).
  
• Where appropriate, “blinding” of the study team members who collected outcome data (i.e., they are kept unaware of who is in the intervention versus control group).
  
• Preferably long-term follow-up (e.g., a year after the intervention ended, preferably longer).

(Regarding the study's reporting of the intervention's effects:)

• Reporting of the intervention's effects on all outcomes that the study measured, not just those for which there are positive effects.
  
• For each claim of a positive effect, a reporting of (i) the size of the effect, and whether it is of policy or practical importance; and (ii) tests showing that the effect is statistically significant (i.e., unlikely to be due to chance). These tests should take into account key features of the study design, such as whether individuals or groups were randomized.
  
• If possible, corroboration of reported effects in more than one implementation site and/or population.

Each of our study summaries includes a short section -- “Discussion of Study Quality” -- detailing how the study measures up to the above criteria, including any flaws or limitations.

How this site relates to other “what works” sites:

Our site focuses on the few studies across the spectrum of social policy that meet the top-level criteria above. We estimate that only 40-50 such studies exist. The site thereby seeks to complement the excellent existing “what works“ sites that provide in-depth coverage of specific policy areas, or systematic evidence reviews. Our site links to such sites.

How we seek to ensure, for each intervention on our site, that we've identified all well-designed RCTs:
For each intervention we summarize, we--

1. Do a comprehensive search of the academic literature and world wide web for such RCTs (using PsycINFO, ProQuest, PubMed, ERIC, C2-SPECTR, Google, Social SciSearch, Dissertation Abstracts, Wilson Social Sciences Abstracts, and similar resources); and
   
2. Specifically ask researchers and practitioners with expertise in the relevant literature if they know of any such RCTs that we are missing.

Through this process, we seek to identify all well-designed RCTs on the intervention, including those showing null effects that might not otherwise have come to our attention. If you are aware of studies we should include, please let us know (by email or phone at 202-239-1248).

This site was developed by the following Coalition staff, with input from the Coalition's Advisory Board and others:

David Anderson, Assistant Director

Deborah Gorman-Smith, Professor of Psychology, University of Illinois at Chicago, and William T. Grant Foundation Distinguished Fellow with the Coalition

Jon Baron, Executive Director