Minnesota
Leading Example
10. Evaluation Policies
The state has evaluation policies and a learning agenda that outline the principles, people, and activities to use evidence to improve results.
10. Evaluation Policies
In 2018, Minnesota Management and Budget adopted an evaluation policy, which governs its use of evaluations and requires the public release of all completed evaluation reports regardless of findings. The policy outlines key principles for evaluation: rigor, relevance, independence, transparency, and ethics. Part of this requires pre-registration of evaluations and final results. Once complete, evaluations become a part of Minnesota’s Inventory, which shows the evidence base for 730 state interventions, and allows for sorting based on level of evidence, service population, settings, and outcomes.
11. Evaluation Resources
The state has dedicated resources for using evaluations to improve results.
11. Evaluation Resources
Minnesota Management and Budget’s FY23 budget includes $800,000 to conduct experimental and quasi-experimental impact evaluations for opiate epidemic response grant activities and general state-funded grant programs, along with inventorying the evidence-based practices supported by these grant programs. In addition, Minnesota Management and Budget received funding ($400,000) in FY23 to inventory evidence-based activities and conduct impact evaluations of county-based adult mental health programs. In total, more than 2% of MMB’s budget is spent on evaluation activities. In addition, Minnesota’s Statewide Health Improvement Partnership (SHIP), administered by the Department of Health, provides tens of millions of dollars in grants annually to communities for health care prevention efforts. This grant program supports grantees with technical assistance and requires grantees to report on evaluation and monitoring of outcomes.
12. Evaluation Use
The state has improved results by using evaluation, program inventories, and cost-benefit analysis.
12. Evaluation Use
As a result of a 2015 Minnesota law, Minnesota Management and Budget (MMB) has developed numerous inventories of evidence-based programs, including in the areas of juvenile justice, criminal justice, adult and children’s mental health, substance use disorder, higher education, child welfare, public health, and education. MMB also maintains the Minnesota Inventory, a searchable clearinghouse of more than 730 programs operating in the state. As part of the inventory, the state developed a guide for using evidence in policymaking and evidence definitions to categorize interventions as proven effective, promising, theory based, or no effect. These definitions have been adopted by other agencies and are frequently included in state statute.
Minnesota relied on an institutionalized culture of generating impact by identifying and prioritizing evidence in its use of State Fiscal Recovery Funds. The state relied on evidence-based practices listed in the Minnesota Inventory to help agencies and subrecipients identify what ARP-funded activities were evidence-based and transparently report them for all projects (even where evidence-based information did not need to be reported to the U.S. Department of Treasury). Through June 2022, an estimated $2.36 billion of Minnesota’s State Fiscal Recovery Funds support 67 unique evidence-based practices implemented by just over 1,000 subrecipients. Examples of evidence-based practices supported include hospital-based violence intervention programs, unemployment insurance, crisis lines, increasing healthy food access at food shelves, rapid re-housing, and social emotional learning curricula.
The state’s impact evaluation unit regularly conducts impact evaluations with the results used to inform policymaking. For instance, the results of two 2022 impact evaluations led to changes in required duties of the contractors and expanded grant funding (both programs—Project Echo and Buprenorphine Bootcamps sought to increase the access of buprenorphine for opioid use disorder). This law also requires outcome reporting and allows MMB to conduct impact evaluations of these settlement funds.
The Minnesota Board of Pharmacy and MMB partnered with The Abdul Latif Jameel Poverty Action Lab (J-PAL) and a team of academics to test new messages designed to reduce dangerous co-prescribing of opioids with gabapentin and benzodiazepines. The evaluation randomly assigned prescribers to a range of messages (as well as a no message group) and are testing which is most effective. MMB also partnered with J-PAL and academics on a project to identify trusted messages and messengers to encourage Black and Hispanic communities to increase their rates of COVID-19 testing and vaccine uptake. The work also used a randomized control trial to test the right level of incentives to encourage COVID-19 vaccine uptake.
In 2021, Minnesota enacted legislation that requires recipients of state-funded pre-K-12 education grants to conduct an evidence-based evaluation. The legislation states: “each grant recipient must compile a report that describes the data that was collected and evaluate the effectiveness of the strategies.”
Promising Examples
2. Performance Management
The state has achieved better outcomes through an enterprise performance management system that engages state leaders in using performance data to continuously improve results.
2. Performance Management
The Minnesota governor’s chief of staff convenes weekly goal review meetings for the administration’s strategic goals. During these meetings, the governor’s chief of staff discusses performance, progress, and next steps for hitting these goals. Further, Minnesota Management and Budget (MMB) publishes performance data for all state agencies in its biennial budget documents. In addition to being available for use by executive branch officials, these data are available to legislators and their staff as they develop budget and policy recommendations.
3. Culture
The state has a culture that supports the sustainable use of data and evidence to deliver results in a transparent, equitable, and ethical manner.
3. Culture
The Minnesota governor publicly communicates about Minnesota’s use of data and evidence to produce results. The governor, for example, issues press releases when the state publishes important new data and when Minnesota is recognized for data use by national organizations Results for America and The Pew Charitable Trusts. In such cases, the Governor highlights how the state uses data during interviews and takes time to publicly recognize researchers who provide data that is useful to Minnesota’s state and local policymakers.
Minnesota conducts an annual survey of legislative, county, and executive policymakers asking about the importance and usefulness of evidence in budget and policy decisions. In the 2021 survey, 85% of respondents said that having information about program effectiveness is very important for making decisions. Minnesota also conducts regular engagement surveys where employees are asked, among other questions, how their work contributes to their agency’s mission.
Minnesota maintains publicly available training videos for agency staff on how to develop evidence-based budget proposals. These materials are recirculated to key staff at least annually. Educational presentations on evidence-based practices and standards of evidence are regularly provided to state employees.
Minnesota offers regular trainings to state employees on results-based accountability (RBA) and process improvement and measurement. Minnesota Management and Budget also offers RBA consulting services to agencies.
The State of Minnesota has an achievement award policy that allows agencies to provide monetary awards to employees who demonstrate outstanding job performance for using data and evidence to improve the delivery of services or outcomes for people served. This policy requires that the process of rewarding individuals or groups be integrated with broader management process and performance management.
4. Innovation
The state uses innovation to achieve its priority goals and improve results for residents.
4. Innovation
In 2021, a cross-agency team with representation from the Minnesota Department of Human Services, Department of Employment and Economic Development, Department of Education, the Children’s Cabinet, and other relevant agencies used data on family access to licensed child care to guide funding and management decisions. This included giving preference to grant applicants in geographic areas with extreme childcare access issues. Additional analysis informed process improvements in the development of new childcare businesses, culminating in the development of a One Stop assistance network.
Researchers and practitioners from the State, Minnesota Emergency Medical Services Regulatory Board, and a nonprofit hospital combined and analyzed data from the Minnesota Department of Health and Minnesota State Ambulance Reporting System to help manage hospital capacity during COVID-19 surges. A 2022 paper authored by this partnership shows that these analyses were able to accurately predict hospital bed utilization for COVID-19 patients one to two weeks in advance.
Minnesota leveraged data to inform the implementation of its American Rescue Plan State and Local Recovery Funds to boost equitable access to services and resources, by
- Expanded Meal Access made funds available throughout the state using a methodology which ensured a fair and equitable distribution of funds to a variety of organizations geographically dispersed.
- The Academic and Mental Health Supports program funding was distributed to public school districts and charter schools based on average daily membership and concentration of historically underserved students, with an emphasis on students receiving special education services.
- Services funded by the Expanding Access to Tutoring program are targeted toward historically underserved communities, and services are most likely to be offered in underserved communities.
- Food Support Grant programming increased the number of eligible food shelf users by supporting delivery and transportations costs and investing in equipment needed to support food insecure residents more effectively
5. Data Leadership & Governance
The state has senior leadership and governance structures with the mandate to equitably use data to improve results.
5. Data Leadership & Governance
Minnesota has a Chief Data and Evaluation Officer with the authority, staff, and budget to collect, analyze, share, and use data to evaluate and improve the effectiveness of public programs and inform policy decisions affecting them. The Minnesota Chief Data and Evaluation Officer leads a team of analysts and research scientists who partner with state agencies to integrate useful information into strategic planning, budget formulation, and ongoing management.
6. Data Policies
The state has data policies that outline the principles, people, and activities that govern its data collection and use.
6. Data Policies
In the interest of transparency and improving outcomes for publicly funded programs, a Minnesota statute, the Minnesota Government Data Practices Act, establishes the state’s policy for sharing data with nonprofit organizations, academic institutions, and local government agencies. This publicly available policy indicates that all government data collected, created, received, maintained, or disseminated by a government entity shall be shared unless such sharing is otherwise precluded by federal law or a more specific state statute.
Minnesota’s Impact Evaluation Unit has established data sharing agreements with state agencies to provide secure access to large data sets covering topics including, among others, public health, behavioral health, healthcare claims, criminal justice, and public assistance.
To address concerns raised by Minnesota families through the course of applying for government services, four state agencies and the Attorney General’s Office developed a joint powers data sharing agreement. This agreement allows state agencies – operating more than 30 programs – to better serve children by allowing participating state agencies to share eligibility information of families to connect them with services for which they may be eligible and communicate with them on how to apply for and access those services seamlessly.
7. Data Infrastructure
The state has improved outcomes through technology infrastructure that allows it to efficiently collect, inventory, and share data.
7. Data Infrastructure
Minnesota has robust technology infrastructure with appropriate privacy controls for internal and external data sharing. Examples of public portals include the Minnesota Geospatial Commons, Transparency Minnesota, the Statewide Longitudinal Education Dataset, and the open data catalog.
The Evidence Base Demographics data tool, launched in late 2020, provides information on the demographic composition – race, ethnicity, socioeconomic status, or education, for example – of the people included in studies that are reported by national research clearinghouses, regardless of where they are located. It also allows users to dig into the Minnesota Inventory, providing access to data that can help illustrate who is reflected in the evidence base and whether specific groups and constituencies are represented.
Minnesota’s Statewide Longitudinal Education Data System and Early Childhood Education Data System match administrative education and employment data from five state agencies. Together, these two systems form a P-20 education data system, which captures, analyzes, and uses student data from preschool to high school, college, and the workforce. The agencies created a public tool to allow for analysis of aggregate-level educational data, including the ability to look at disparities in educational achievement, college graduation rates, employment, and other indicators.
8. Data Use
The state has improved outcomes by combining and analyzing cross-agency data to inform policy, budget, or management decisions.
8. Data Use
Researchers and practitioners from the State, Minnesota Emergency Medical Services Regulatory Board, and a nonprofit hospital combined and analyzed data from the Minnesota Department of Health and Minnesota State Ambulance Reporting System to help improve the State’s management of hospital capacity during COVID-19 surges. A 2022 paper authored by this partnership shows that these analyses were able to accurately predict hospital bed utilization for COVID-19 patients one to two weeks in advance.
In 2021, a cross-agency team with representation from the Department of Human Services, Department of Employment and Economic Development, Department of Education, the Children’s Cabinet, and other relevant agencies used data on family access to licensed childcare to guide funding and management decisions. This included giving preference to grant applicants in geographic areas with extreme childcare access issues. Additional analysis informed process improvements in the development of new childcare businesses, culminating in the development of a One Stop assistance network.
As part of its COVID-19 response, the State of Minnesota had a Community Resiliency and Recovery workgroup composed of external advisors – community leaders representing underrepresented groups including residents identifying as BIPOC, LGBTQIA+, and persons with a disability – at least monthly to discuss the progress of state’s COVID-19 response. A component of these meetings was a discussion of the public data on the Data by Race/Ethnicity section of the state’s COVID-19 dashboard. These meetings led to substantive changes to the dashboard, including adding data on congregate care settings such as nursing homes and people experiencing homelessness.
9. Evaluation Leadership & Governance
The state has a leadership and governance structure with the authority to use evaluations to improve results.
9. Evaluation Leadership & Governance
Minnesota Management and Budget has a Chief Data and Evaluation Officer with the authority, staff, and budget to conduct experimental and quasi-experimental impact evaluations, as well as advise the Governor and legislature on the evidence of effectiveness of publicly funded services. Minnesota also has an Impact Evaluation unit composed of research scientists, analysts, and administrators who are tasked with producing high quality evidence about the efficacy and impact of state-funded programs. This unit created a data committee composed of leadership and staff from the Department of Human Services to advise on current and future evaluations.
13. Results-Focused Budget Process
The state has improved outcomes by prioritizing data and evidence in its budget process through guidance to agencies and engagement with the legislature.
13. Results-Focused Budget Process
A 2017 Minnesota law requires state agencies to include performance data in their biennial budget documents. This performance data is frequently disaggregated to highlight disparities. Budget instructions for FY 2024-2025 direct agencies to include information about the evidence base for their budget proposals, a summary of evidence, citations, and amount to be spent on the activity. In addition, the state used evidence to inform funding decisions that resulted in $3.5 million in new or expanded evidence-based activities funded during the 2022 legislative session for a total of at least $150 million in new or expanded investments in evidence-based practices in FY 2020 through FY 2023.
15. Evidence-Based Grantmaking
The state has improved outcomes by allocating grant funding based on evidence of effectiveness.
15. Evidence-Based Grantmaking
In early 2022, Minnesota created a Grantmaking Impact Improvement Team composed of executive branch leaders with grantmaking and impact-oriented expertise. The Team also includes external advisors representing evidence-based policymaking and non-profit organizations. This group of experts established best practices intended to provide direction for how grant programs can maximize impact through evidence-based practices, community-based best practices, and accessible grantmaking processes. Additionally, they are charged with reviewing select grant programs for these standards and making associated recommendations.
A Minnesota law (Sec. 127) requires the Minnesota Management and Budget (MMB) to systematically review the Department of Human Services’ use of evidence in allocating hundreds of millions of dollars in grant funding each biennium. This evaluation found one-third of the state-funded grant programs that were reviewed support at least one evidence-based practice. Minnesota estimates that about $15 million (or 6%) of the biennial allocations to these programs are for activities that have been demonstrated to impact outcomes of interest.
In 2022, the Minnesota Department of Human Services used an evidence-based framework in its Housing with Supports for Adults with Serious Mental Illness program. Grantees were required to use evidence-based interventions to assist people in transitioning to and sustaining permanent supportive housing.
In 2021, the Minnesota Department of Education provided funding through a Full-Service Community Schools Grant program that exclusively funds the implementation and expansion of evidence-based full-service community schools. In 2020, the Department used evidence-based frameworks to guide funding decisions for the Minnesota Comprehensive Literacy State Development (CLSD) Grant program.
In 2021, the Minnesota Department of Health’s Strong Foundations: Evidence-Based Home Visiting Start-Ups program exclusively funded start-up and expansion of the following evidence-based home visiting models: Healthy Families America, Nurse Family Partnership, Family Spirit, and Home-Based Early Head Start.
In 2022, Serve Minnesota, a nonprofit organization that administers state AmeriCorps funding, utilized a tiered evidence framework from the AmeriCorps Evidence Exchange to determine fund allocations through the Minnesota AmeriCorps program. The Request for Proposals prioritized evidence-based interventions with ‘Moderate’ or ‘Strong’ evidence for consideration.
Two laws going into effect in 2022 make similar requirements in various programs. Starting July 1, 2022, state law required all state-funded grant programs overseen by the Minnesota Department of Education to compile grantee-level reports that indicate the evidence of effectiveness for the strategies used along with other performance data. Effective July 1, 2022, the Minnesota Legislature required Minnesota Management and Budget to inventory evidence-based activities and conduct impact evaluations of county-based adult mental health programs.
16. Repurpose for Results
The state has improved outcomes by moving funds away from programs that are ineffective and toward those that get results.
16. Repurpose for Results
A 2014 Minnesota law (subdivision 7) requires the Minnesota Department of Human Services to use the Self-Support Index to monitor each county’s performance in assisting clients to become self-sufficient. Counties that meet performance targets receive a 2.5% bonus payment from the state, and counties that perform below the expected target must submit a performance improvement plan. In counties where “no improvement is shown by the end of the multi-year plan, the county or tribe’s allocation must be decreased by 2.5%” (Section 256J.626(7)(a)(2)).
A 2016 Minnesota law (section 14, line 15.21) allows the state to use the savings achieved from reducing the sentences of minor drug offenders for evidence-based drug and mental health treatments for offenders still in prison or under supervised release. The evidence to support this law comes from the Department of Corrections’ research, which found that providing offenders drug treatment reduces recidivism rates.