Review and Synthesis of Existing Court Reform Evaluations

The National Evaluation of the Court Improvement Program builds on previous evaluations of court reform. Existing evaluations are being reviewed and synthesized to provide a broader context for understanding and interpreting findings gained through the outcome evaluations of the study sites.

Project staff are reviewing evaluations performed on:

• CIP-funded initiatives
• Adoption Opportunity initiatives
• Model Courts (funded by the National Council of Juvenile and Family Court Judges)
• Family Drug Courts
• Other specialized courts

These evaluations assess a broad range of reforms and activities and use a range of evaluation designs and methodologies to assess performance. Therefore, evaluations are being assessed based on three characteristics: 1) reform type 2) evaluation design components 3) and rigor of design components.

The goal of this analysis is three fold. First, it will allow us to present findings from multiple evaluations of a given reform category (for instance, alternative dispute resolution models). Second, it will allow us to present these findings based on the rigor of the analysis used. Third, this approach will allow us to highlight stronger evaluations and present methodologies to the field for future use and replication.

Reform Type

The 60 evaluations we analyzed will be presented in the following framework:

• Alternative dispute resolution: This includes reforms involving mediation, family group decision making, and pre-hearing conferences.



• Child and parent legal representation: This includes reforms designed to change the availability, professional standards/qualifications, and/or payment systems of legal representation.



• Case tracking and management: This includes reforms involving state and local initiatives to improve case tracking and information sharing systems.



• Specialized hearings: This includes reforms involving efforts to specify hearing timeframes, develop specialized hearings for judicial review, standardize the content of court hearings, and define processes for third party reviews.



• Parent education and support: This includes reforms involving outreach and educational efforts to support parent and caregiver court participation.



• Systemic reforms: This includes reforms undertaken by sites that have implemented multiple and interrelated reform efforts such as model courts or family court systems.



• Training and educational materials: This includes reforms involving a wide range of curricula and training modules for judges and other court staff.

Types of Evaluation Components and Levels of Rigor

Our analysis will focus on examining each evaluation's design to assess its methodology and rigor. We will consider whether evaluations include a process evaluation component, an outcome evaluation component or both of these components.

• Process Component- This component focuses on understanding and documenting the implementation aspect of the reform and provides descriptive findings. In addition, this component may also pay particular attention to stakeholder perceptions, attitudes and beliefs.



• Outcome Component- This component is defined by its specific assessment of client and system level outcomes achieved as a result of the reform or activity such as reduced time to permanency or change in hearing quality.

Two levels of rigor were defined for both process and outcome evaluation components to distinguish the strength of the methodologies utilized.

Evaluation Tiers

Once each evaluation is assessed for its methodological quality and rigor, each will be classified into one of four tiers:

• Tier 1 Evaluations - These include both a strong outcome evaluation component and a strong process evaluation component. These evaluations use strong designs, rigorous and structured methodologies, and appropriate quantitative analyses to better understand and interpret findings.



• Tier 2 Evaluations - These include evaluations that have both process and outcome components and one of the two components is methodologically strong. This tier also includes evaluations that include only one design component, either process or outcome, but that component is particularly strong.



• Tier 3 Evaluations - These include a less rigorous outcome component paired with a less rigorous process component.



• Tier 4 Evaluations - These include only a less rigorous process or an outcome component.