12. Evaluation Use

The California Department of Social Services created the California Evidence-Based Clearinghouse for Child Welfare, which allows child welfare providers and professionals to “identify, select, and implement evidence-based child welfare practices that will improve child safety, increase permanency, increase family and community stability, and promote child and family well-being.” This tool helps identify evidence-based practices and provides guidance and support for program implementation. The clearinghouse’s numerical rating scale categorizes programs into six tiers of evidence and uses a relevance scale as a complement to the scientific rating scale and to demonstrate applicability for client populations. In addition to the clearinghouse, various evaluations have impacted decision making in discrete programs and practices, including:

• $80 million in ARP SLFRF are being leveraged for legal services was partially based on positive impacts observed in a state-run randomized evaluation.
• The California Department of Transportation’s Road Charge Program has used extensive data collection and evaluation to inform key program process and design decisions for this innovative transportation program. This includes using an initial pilot study to identify additional studies, evaluations, community-based research, and equity work.
• The California Workforce Development Board (CWDB) has engaged in several recent evaluations on California job centers, training partnerships, regional plan implementation, reentry services, and serving other highly barriered populations. These findings are informing grantmaking and program design decisions, including emphasizing industry engagement in programs and promoting additional investment in metrics and evaluation.

The Colorado Governor’s Office of State Planning and Budgeting (OSPB) and the Colorado state legislature have aligned evidence definitions for a five-step evidence continuum that informs budget and spending decisions. OSPB requires agencies to identify the level of evidence and the results of any evaluations of programs as a part of the budget proposal process. OSPB also requires agencies to report on cost-benefit analyses performed on agency programs as a part of the annual budget process.

Several projects funded through the American Rescue Plan Act (ARP) plan to include program evaluations. The Colorado Evaluation and Action Lab (Colorado Lab) partnered with the Governor’s Office of State Planning and Budgeting to determine appropriate evaluation strategies for ARP-funded projects with a number of state agencies, including the Departments of Higher Education, Health Care Policy and Financing, Human Services, and Labor and Employment.

A 2015 Connecticut law defines three tiers of evidence for programs operated by the Connecticut Departments of Correction, Children and Families, and Mental Health and Addiction Services, and the Court Support Services Division of the Judicial Branch: evidence-based, research-based, and promising. The law requires these agencies to categorize their programs by the evidence tiers in even-numbered fiscal years. Additionally, the law charges the Institute for Municipal and Regional Policy at Central Connecticut State University with submitting an annual report containing a cost-benefit analysis of the programs. In 2020, the report entitled Benefit-Cost Analyses of Evidence-Based Programs showed that 108 programs and services administered by the Judicial Branch’s Court Support Services Division and the Department of Correction were identified as evidence-based.

The Florida Department of Juvenile Justice requires all residential commitment prevention contractors to implement at least one evidence-based model from the agency’s Sourcebook of Delinquency Interventions. The sourcebook lists all juvenile justice programs according to three levels of evidence: evidence-based practices, promising practices, and practices with demonstrated effectiveness, as defined in the book. The Department also introduced a Standardized Program Evaluation Protocol, a monitoring tool to ensure providers implement programs with fidelity.

The North Carolina Office of State Budget and Management (OSBM) and agency partners conduct program inventories, research evidence reviews, and benefit-cost analyses through the Results First Initiative. In 2022, the state published project briefs with findings and next steps in the areas of juvenile justice and adult mental health. The state has also created evidence definitions to categorize programs based on their levels of evidence.    

The Office of Strategic Partnerships (OSP) launched the NC Project Portal, an innovative tool for developing and launching partnerships with government and for bolstering cross-sector collaboration. The portal includes information about opportunities for partnering with state agencies on their research needs and how to express interest in pursuing those opportunities. It includes 25+ projects that are open, in-progress, or completed as well as  project deliverables and information about the planned use of results where applicable.

In 2022, the Governor issued Recommended Budget Adjustments for FY 2022-2023. Several recommendations prioritized shifting dollars towards resources or human power proven to make governmental operations more efficient. The following recommendation was funded in the most recent budget (signed July 11): Establish four State Crime Laboratory positions (within the NC Department of Justice) – two DNA scientists and two drug chemists. Research conducted by the department demonstrated that, on average, each new scientist reduces turnaround time for cases by 5%; thus, four new scientists are expected to reduce turnaround times by 20%.  

The previous three budgets each appropriated $1 million in funding for the New Jersey Policy Lab, led by Rutgers University, that will contribute high-quality, nonpartisan research and analysis of policy-based solutions facing New Jersey. The initiative, granted by the New Jersey Office of the Secretary of Higher Education, will also build coalitions across governments and community organizations to support evidence-based policy initiatives, assist stakeholders in troubleshooting implementation issues, provide data modeling for policy recommendations, and evaluate the impacts of proposed statewide public policy measures.

New Mexico has published a series of inventory and cost-benefit reports in the areas of children’s behavioral health, adult behavioral health, early education, child welfare, criminal justice, healthcare, and education. In 2019, the Legislative Finance Committee’s evaluation unit published analyses on substance abuse treatment outcomes and Medicaid spending and managed care administration. In FY21, the evaluation unit partnered with Abdul Latif Jameel Poverty Action Lab North America (J-PAL NA) to evaluate the effectiveness of the state’s early college high schools. All reports from the evaluation unit can be found here.

Beginning in 2013, the New York State Division of Criminal Justice Services developed a cost-benefit analysis, based on an initial technical report, to outline the impact, costs, and benefits of specific criminal justice interventions. As a result of these efforts, New York has continued to operate Alternatives to Incarceration programs, a $17 million initiative in 2021-2022, to support evidence-based interventions.

In 2019, the New York Division of Criminal Justice Services partnered with the Center for Evidence-Based Crime Policy to host a 10-week course called “Evidence-Based Policing: Translating Research into Practice.” The course has continued and will be offered in spring 2022. New York was the first state to offer the course, which was completed by 26 law enforcement professionals, during which students developed and presented ideas for furthering the use of evidence-based policing strategies in their respective agencies.

Beginning in 2013, the New York State Division of Criminal Justice Services developed a cost-benefit analysis, based on an initial technical report, to outline the impact, costs, and benefits of specific criminal justice interventions. As a result of these efforts, New York has continued to operate Alternatives to Incarceration programs, a $17 million initiative in 2021-2022, to support evidence-based interventions.

In 2019, the New York Division of Criminal Justice Services partnered with the Center for Evidence-Based Crime Policy to host a 10-week course called “Evidence-Based Policing: Translating Research into Practice.” The course has continued and will be offered in spring 2022. New York was the first state to offer the course, which was completed by 26 law enforcement professionals, during which students developed and presented ideas for furthering the use of evidence-based policing strategies in their respective agencies.

The Ohio Department of Education’s evidence-based clearinghouse provides a common evidence framework by aligning evidence standards from various clearinghouses. The framework assists educators in identifying evidence-based interventions according to content focus area, subject, grade bands, urban/rural settings, student demographics, and the four levels of evidence in the federal Every Student Succeeds Act.

As part of the implementation of the 2003 Oregon law on evidence-based health, youth, and corrections programs, Oregon Health Authority (OHA) maintains an ongoing inventory of approved evidence-based practices in behavioral health, including culturally relevant services.

The Oregon Health Authority has established the Health Evidence Review Commission to continually review and update an ongoing inventory of evidence-based practices at work within OHA. The commission vets medical evidence to identify new and valuable practices and maintains a publicly available list of practices in a wide range of clinical and treatment areas. In 2020, Oregon Youth Authority provided an updated report identifying the cost-effectiveness of their programs and services, with the next report being released in late 2022.

In 2020, the Pennsylvania Workforce Development Board unanimously voted to adopt a four-tiered evidence framework for the state’s workforce system. The definitions for evidence of effectiveness include: strong evidence, moderate evidence, preliminary evidence, and pre-preliminary evidence.

In 2020, the Pennsylvania Department of Education (PDE) launched the PDE Evidence Resource Center to help Pennsylvania schools thoughtfully apply high-quality, relevant research to their local settings. By listing evidence-based interventions in the areas of instruction and curriculum, student support and professional development, and student wellness, this tool helps school districts identify and select strategies to address every aspect of school improvement. This tool is also being used for the PDE’s research agenda that allows the department to identify and measure its progress in meeting its research and learning priorities.

Tennessee’s Office of Evidence and Impact (OEI) defined five evidence steps. The Office also completed inventories of state- and federally-funded programs in the areas of corrections, mental health, and substance abuse services as a part of the state’s evidence-based budgeting initiative. In 2022, OEI published program inventories for higher education, children’s services, and human services.

Tennessee’s Department of Education, Department of Higher Education, and Department of Health have conducted program evaluations independently or in partnership with universities that have directly impacted policy, budget, and programmatic decisions. For example, state evaluations on voluntary pre-K, financial aid for nontraditional postsecondary students, and the smoking cessation program for pregnant women have informed timing of program expansions or affirmed the continuation of programs with high confidence in results.

Project KIDS is a special performance audit of the Office of the State Auditor that integrates financial, operational, and performance data to improve data-driven decision-making in Utah K-12 public education. Project KIDS follows resources down to the individual student level to understand how K-12 public education dollars are spent. It then analyzes course enrollment and demographic data to identify which students participate in particular classes and programs. Thus, individual resource profiles are created for all Utah students based on their unique course-taking patterns and demographics. Additionally, each student’s spending information is tied to their academic performance. District and charter school administrators are issued more detailed dashboards with privileged access to student-level data for their specific local education agency (LEA). By integrating student-level spending and performance, this methodology allows for an exploration of what resource allocations drive high student achievement. It also facilitates bottom-up, not top-down, student spending averages. Project KIDS recognizes that a top-down spending average fails to differentiate between various student needs and consequent spending. Thus, the individual student resource profiles are utilized to build up spending averages for each classroom, school, and local education agency. This framework enables stakeholders and the public to ask deeper questions about equitable and effective spending practices. They can use the school spending map and other analytic resources in weighing spending decisions.

Since 2012, at the direction of the legislature, the Washington State Institute for Public Policy (WSIPP) has produced a series of inventories of evidence-based, research-based, and promising programs. These inventories are developed with the aim of informing state agencies about effective and cost-beneficial options for service provision. In addition, a 2013 state law directed the Department of Corrections, in consultation with WSIPP, to: (1) compile an inventory of existing programs; (2) determine whether its programs were evidence-based; (3) assess the effectiveness of its programs, including conducting a cost-benefit analysis; and (4) phase out ineffective programs and implement evidence-based programs.

As a result of laws like these that require evidence reviews and/or direct outcome evaluations, WSIPP has published hundreds of cost-benefit analyses in a wide variety of issue areas over the past 10 years. The WSIPP cost-benefit framework allows the translation of program effects into dollars and cents and has been emulated by several state governments across the country.

The 2021 legislature directed WSIPP to publish benefit-cost analyses of wilderness therapy programs for youth mental health as well as the use of domestic steel in transportation and capital budget contracts. WSIPP also has ongoing projects examining the long-term costs and benefits of legalizing recreational cannabis, investigating the cost-effectiveness of integrating involuntary treatment for substance use disorders and mental health disorders, and exploring the costs and benefits of the state quality rating system for early childhood education providers.