Colorado

A 2013 Colorado law required all state agencies to submit annual performance reports to the Colorado state legislature. Reports must include: (1) performance measures for the major functions of the department; (2) performance goals for at least the subsequent three years; (3) a description of the strategies necessary to achieve those goals; and (4) a summary of the department’s most recent performance evaluation. The Governor’s Director of Operations and Cabinet Affairs oversees the development and execution of statewide performance goals. The Governor’s Office of Operations maintains the statewide performance management system required, by law, to be published on an annual basis. The office provides guidance on developing annual strategic goals and annual performance plans and identifies opportunities for process improvement (CRS 2-7-204). Consistent with this management system, executive branch agencies continue to identify annual Wildly Important Goals (WIGs) that align with their agency’s mission, the statewide strategic goals, working group goals, and the Reimagine State Government Initiative.

To support the Governor’s strategic priorities, the Governor’s Office of Operations maintains working groups (Economic Development, Environment and Renewables, Health, and Education and Workforce) with cabinet members and relevant leadership, including: deputy directors, division directors, and/or key program staff, as needed. Those working groups set annual performance goals aligned to support the Governor’s priorities and progress toward the goals displayed on the Governor’s Dashboard with data updated monthly. The Office of Operations, Office of State Planning and Budgeting, and Policy Office within the Governor’s Office regularly coordinate to provide feedback on performance goals, budget requests, and legislative items, respectively. Additionally, the Governor’s Office will be launching a website where the public can review the spending of ARP SLFRF, and the outputs and outcomes that the state is receiving as a result.

The State of Colorado is committed to continuously innovating to deliver results. The Governor’s Office of Operations is tasked with supporting three key areas: 1) performance management, 2) process improvement, and 3) professional development. The Office currently supports innovation by offering free training to all state employees in Lean, Agile, and Change Management, as well as a course in human-centered design.

Innovation is encouraged within state agencies as well. Examples of the innovations include:

• The Colorado Department of Transportation has an Office of Process Improvement that, among other activities, hosts a Lean Everyday Ideas program within the Department and tracks the impact of those projects.
• The Department of Public Health and Environment and the Department of Public Safety also have intranet sites for their employees on Lean and innovation projects with resources and active project tracking via an internal dashboard.
• In addition, several of our state agencies have hired innovation focused positions, including the Department of Higher Education and Department of Human Services.

To share best practices between state agencies, the IDEA Champions (p. 30) are a community of practice with at least one representative from most state agencies that meets quarterly to share innovative solutions. The group also develops resources for the Colorado Innovation Center website, which provides a database of process improvement resources, training videos, templates, and ongoing projects at state agencies.

The State of Colorado’s Office of Information Technology (OIT) includes a Chief Data Officer. The Chief Data Officer is responsible for developing and directing the state’s data strategy, data governance, and data management. The position establishes enterprise data policies, standards, and procedures to maximize the state’s investment in data management systems. The OIT’s statewide data management program leverages data and information as enterprise assets and establishes standards and processes to enable more agile solutions and government services. Additionally, OIT’s Government Data Advisory Board (GDAB), created by a 2009 law, advises the Chief Data Officer on activities and policies necessary to develop the interdepartmental data protocol, made up of members from state and local government. GDAB’s mission is to improve the efficiency and effectiveness of state government, citizen service delivery, and policymaking by providing guidance and recommendations on how the state should govern and manage data and data management systems.

In 2022, GDAB created a data governance subcommittee, which will define a statewide protocol for collecting, managing, storing, and linking personally identifying information (PII) and develop best practices for data management and data governance, including data lifecycle, reconciliation, and retention policies and procedures.

The Colorado Information Marketplace is the state’s open data repository with more than 2,000 datasets intended to increase government transparency. Additionally, the 2021 Colorado data law requires each agency to conduct an inventory of its own data assets, with assistance from the Office of Information Technology and the Government Data Advisory Board (GDAB). This inventory is still in development, though some agencies have already started on the effort, and it is being coordinated by the Chief Data Officer.

The 2021 State of Colorado Recovery Plan for the State and Local Fiscal Recovery Funds described investments to improve the Colorado Department of Healthcare Policy and Finance’s various data infrastructures, which include upgrades to IT systems and infrastructure, as well as the data and information these systems collect. Examples of these investments in data infrastructure include:

• Conducting a system mapping of program and IT systems to determine a mechanism to share data and information across offices in partnership with the Office of Aging and Adult Services within the Department of Human Services.
• Connecting case management agencies to the Colorado Regional Health Information Exchange Organization (CORHIO) to obtain hospital admission data in real-time.
• Expanding the infrastructure for a Social Health Information Exchange (SHIE) which provides case management agencies, regional healthcare coordinators, care coordinators, and health care providers with real-time connections to resources like food, energy assistance, wellness programs, and more, in partnership with the Office of eHealth Innovation.
• Investing in data repositories that enable more robust insights into gaps in care as well as the providers and services with positive outcomes, supports, and programs for individuals receiving home and community based services.

Beyond state open data, the Colorado Governor’s Office and the Colorado Evaluation and Action Lab co-designed the Linked Information Network of Colorado (LINC) to facilitate data sharing for research and analytics in 2019. The network is designed to share data across state agencies and provide de-identified data to perform robust, academically rigorous research to inform policy. LINC has a three-tier legal structure, which includes: (1) an enterprise memorandum of understanding (eMOU) signed by all data providers; (2) data-sharing agreements to secure, handle, and anonymize data for all LINC projects; and (3) data licenses with roles and responsibilities for users of LINC project data. Additionally, several state agencies – including the departments of higher education, labor and employment, and human services and entities (Colorado Workforce Development Council, and the Colorado Community College System) are working together to build a data trust to share data among state agencies for specific use cases, which will assist in sharing data in a well-governed and highly secure manner.

In 2019, the Colorado Governor’s Office and the Colorado Evaluation and Action Lab at the University of Denver co-designed the Linked Information Network of Colorado (LINC) to facilitate data sharing for research and analytics purposes as a way to improve state policies and programs. LINC, through the Colorado Evaluation and Action Lab, produces informative evidence about the impacts of state programs, and those reports are shared with agency leadership and staff during cabinet working group meetings. Using this data, the lab and state agencies have collaborated on projects to improve outcomes in areas such as education, renewable energy, reentry and racial disparities in criminal justice, prenatal health, and the early childhood education workforce.

• In 2021, LINC worked with the Office of Early Childhood, the Department of Higher Education, and the Department of Labor and Employment to provide better data on the early childhood workforce, including understanding professional pathways, turnover, and compensation disparities. This information is regularly used within the Office of Early Childhood and the Governor’s Early Childhood Cabinet Working Group to determine appropriate strategies to support and increase the number of early childhood professionals.
• LINC is also working with the Office of Early Childhood, the Office of Children, Youth, and Families, and the Department of Labor and Employment throughout 2022 to link data from the Colorado Community Response program with child welfare and employment data to determine predictors of child maltreatment and appropriate points of intervention.
• In addition, the Colorado Department of Higher Education was the first state agency in the nation to partner on a pilot project with the U.S. Census Bureau to match federal unemployment insurance data with postsecondary degree completion data.
• At the state and county level, the Colorado Department of Human Services’ C-Stat performance management system facilitates data sharing among its 64 counties by providing dashboards to track key metrics and Performance and Partnerships Exchanges to facilitate sharing of best practices. The Colorado Department of Public Safety worked with The Division of Criminal Justice to gather, analyze, and publish multi-agency data, prompting recommendations improving criminal justice outcomes and equity to be made by the Colorado Commission on Criminal and Juvenile Justice.

The Colorado Governor’s Office of State Planning and Budgeting (OSPB) and its director lead the use of data and evidence in the state’s budget process through the review of agency budget requests based upon their level of evidence and evaluation to inform priority policy and budget decisions. This office engages with the University of Denver’s Colorado Evaluation and Action Lab as well as the Colorado Evidence-Based Policy Collaborative composed of state agency representatives and community organizations. The office also administers a $500,000 annual fund for program evaluation and implementation grants, which provides competitive funding for agencies to undertake evaluation and implementation science projects. OSPB continuously engages with the legislative branch to determine how evidence can be better utilized in the budget process.

With the large influx of the America Rescue Plan Act (ARP), OSPB has applied the evidence continuum as a cornerstone of its decision making process to ensure the strongest outcomes for residents. The Governor’s Office is evaluating all ARP-funded programs against the evidence continuum and includes information on evidence-based practices in all federal reports.

The Colorado Governor’s Office of State Planning and Budgeting (OSPB) publishes the results of evaluations funded by the grant program on its website each year. OSPB also publishes annual guidance to state departments requiring the use of data and evidence in the state’s budget process. It also provides definitions for key research and evaluation terms and “encourages agencies to use the evidence continuum outside the budget process as a means to evaluate programming internally, and to guide discussions with OSPB analysts and internal and external stakeholders.” With the large influx of the America Rescue Plan Act (ARP), OSPB has applied the evidence continuum as a cornerstone of its decision making process to ensure the strongest outcomes for residents.

To align the guidance and definitions across the executive and legislative branches, the Colorado state legislature passed a law in 2021 that establishes a set of evidence-based definitions, aligned with the OSPB evidence continuum, to be used when analyzing a program or practice. The law further requires the joint budget committee staff to “independently analyze and describe the program or practice using the definitions and to include any evidence-based information as part of any recommendation it makes regarding a budget request or budget amendment request.”

The Colorado Department of Higher Education (CDHE) developed a research agenda to inform education and training policies and programs. Data sharing is required by Colorado law among the Department of Education, Department of Labor, and Department of Economic Development, which collectively generate research covering the education-to-workforce continuum.

The Colorado Department of Public Safety’s Juvenile Diversion Program invests 3% ($120,000) of program funds to conduct evaluations. Crime Victim Services’ competitive funding opportunities require all applicants to evaluate the effectiveness of their project. It also requires those who currently receive funding to answer questions regarding their current effectiveness at meeting their funded projects’ goals and objectives, as well as their adherence to their budget. Additionally, Colorado’s Office of State Planning and Budgeting also administers a $500,000 annual fund for program evaluation and implementation grants, which provides competitive funding to support program implementation or evaluation of outcomes.

The Colorado Governor’s Office of State Planning and Budgeting (OSPB) and the Colorado state legislature have aligned evidence definitions for a five-step evidence continuum that informs budget and spending decisions. OSPB requires agencies to identify the level of evidence and the results of any evaluations of programs as a part of the budget proposal process. OSPB also requires agencies to report on cost-benefit analyses performed on agency programs as a part of the annual budget process.

Several projects funded through the American Rescue Plan Act (ARP) plan to include program evaluations. The Colorado Evaluation and Action Lab (Colorado Lab) partnered with the Governor’s Office of State Planning and Budgeting to determine appropriate evaluation strategies for ARP-funded projects with a number of state agencies, including the Departments of Higher Education, Health Care Policy and Financing, Human Services, and Labor and Employment.

The Colorado Governor’s Office of State Planning and Budgeting (OSPB) continues to make evidence an integral part of the budget process. Since 2016, OSPB has required executive branch departments to include evidence information in budget requests that are considered for inclusion in the Governor’s annual budget proposal. OSPB has adopted the use of a five-step “evidence continuum” that represents the states of building and assessing evidence on government programs.

With the large influx of the America Rescue Plan Act (ARP), OSPB has applied the evidence continuum as a cornerstone of its decision making process to ensure the strongest outcomes for residents. By applying this framework, Colorado is investing in programs and projects that promote the use of evidence-based and informed practices and programs like wraparound services for 500 Medicaid members, drawing from learnings from a Denver housing social impact bond project and a pilot Community Aging in Place – Advancing Better Living for Elders (CAPABLE) program, an evidence-based fall prevention program that will serve 400 aging residents; and funding the University of Colorado Health Sciences Center and for training and education for health-care, behavioral health-care, and public health-care professionals, to further promote the use of evidence-based models of care for treatment of pain and substance use disorders.

All OSPB budget analysts receive training on evidence-based policy and work closely with their respective executive branch agencies to systematically incorporate evidence information into their budget requests. This evidence information is an integral part of the briefings and decision-making process in which the Governor weighs the hundreds of budget requests that he will ultimately prioritize and include in the Governor’s annual budget proposal submitted to the legislature.

In the 2022 legislative session, OSPB and the Colorado Legislature’s Joint Budget Committee began implementing HB21-284, Evidence-based Evaluations for Budget, which codified requirements to use a consistent evidence framework to assist the legislature’s budget decisions. The legislation aligns with the evidence continuum and also appropriates funding that bolsters the efforts of Joint Budget Committee staff to incorporate evidence into the budget process. The Governor’s budget includes assessments of the evidence for each proposal in accordance with HB21-284, and Joint Budget Committee staff review the Governor’s Office characterization of the evidence (e.g., reviewing and critiquing studies cited in the proposal) and also may perform their own independent assessment of the evidence. Staff from OSPB and the Joint Budget Committee continue to collaborate on the implementation of HB21-284 and refine processes to improve the information provided to decision makers.

Additionally, the 2013 Colorado State Measurement for Accountable, Responsive, and Transparent Government (SMART) Act required all Colorado state agencies to submit annual performance reports to the state legislature as part of the state’s budget process. These reports include: (1) performance measures for the major functions of the department; (2) performance goals for at least the following three years; (3) a description of the strategies necessary to achieve those goals; and (4) a summary of the department’s most recent performance evaluation.

In 2015, the Colorado General Assembly passed HB 15-1317, which authorized the Governor’s Office of State Planning and Budgeting (OSPB) to enter into Pay for Success (PFS) agreements. Colorado continues its Pay for Success programs to improve outcomes for youth at risk of out-of-home placements. Three programs were implemented with payments dependent on successful achievement of a set of preselected outcomes. Per the annual report: “As of May 2021, both the Fostering Opportunities and Multi-Systematic Therapy (MST) projects have hit their desired implementation benchmarks. In March 2021, the independent evaluator certified that the MST project had once again exceeded its desired implementation metrics, resulting in an additional success payment to investors of $305,190, the maximum success payment possible under the contract. In April 2021, the evaluator certified that the Fostering Opportunity project had reached a success level three (out of four levels), yielding a payment  to investors of $427,105. The Denver Runaway Project was terminated in September 2020 under the contract guidelines due to a low take-up of services.”  

In addition, Colorado’s Department of Corrections offender re-entry grant program, also referred to as the Work and Gain Education and Employment Skills (WAGEES) program, provides funds to community partners that provide re-entry services, through the intermediary The Latino Coalition. The Latino Coalition provides technical assistance and capacity building for community partners through twice-monthly site visits, quarterly meetings, regular group-based training, individualized coaching, and frequent communication with community partners. Performance measures are reviewed monthly with community partners, and when performance is low, strategies are identified to help improve outcomes.

In 2021, the Colorado Department of Education used an evidence-based framework to allocate funds to the Colorado Comprehensive Literacy State Development Grant (CLSD) program. Grantees were required to utilize evidence-based interventions that help children develop reading, writing, speaking, and listening skills, as defined by the Colorado Reading to Ensure Academic Development (READ) Act. The State’s grant program placed an emphasis on funding directly supporting historically underserved students, including children in poverty, English learners, and children with disabilities.

In 2018, the Colorado Department of Education grant program for school improvement, Empowering Action for School Improvement, required schools to use evidence-based practices as defined by the federal Every Student Succeeds Act (ESSA). The application gives points for the use of evidence-based strategies, as well as points based on past performance for applicants who are seeking to expand existing initiatives. Further, the Colorado Reading to Ensure Academic Development (READ) Act, amended in 2019, emphasizes the use of evidence-based instructional practices to support early literacy. It required use of program funds for external program evaluation; created a science-based early literacy block grant program of $2.5 million annually; and required districts receiving funds in the school year 2021-2022 to have K-3 teachers complete evidence-based training in teaching reading.

In July of 2021, the Colorado Workforce Development Council launched the Upskilling, Reskilling, and Next-Skilling Workers program to support evidence-based projects that improve workforce development and employment capacity. The application awards points for seven criteria, one of which includes the level of evidence of the proposed approach as defined by the Colorado Evidence Continuum.