California

In July 2022, the California Governor and the California Environmental Protection Agency released statewide strategic goals on climate and a climate dashboard to measure progress against goals. This complements the recently updated comprehensive, statewide Climate Adaptation Strategy that includes six key priorities supported with goals and actions, each of which has at least one metric to assess progress. Public engagement was a key part of that process and the state will release annual implementation reports that will promote transparency by tracking progress for each action in the Strategy.

California state agencies and departments have their own, independent performance management systems and continuous improvement cycles. Department-level performance management systems include: the Department of Social Services Cal-OAR CalWORKs monitoring system and the Department of Education’s School Dashboard, which documents state and local education performance across a variety of measures. For the Department of Health Care Services, California Board of State and Community Corrections and the Department of Developmental Services program monitoring dashboards and reports serve as a key continuous improvement and transparency tools; many departments implement associated performance and quality incentives. The Department of Aging’s Data Dashboard for Aging, the COVID-19 recovery SMARTER plan, and the statewide Climate Goals and Adaptation Strategy track progress towards key indicators and better targeted efforts, including specific outcome-oriented goals.

The newly merged California Office of Data and Innovation (ODI) within the Government Operations Agency (GovOps) has the explicit goal to foster a data and analytics–informed culture. This includes the development of a statewide training program to develop and deliver modern curriculum in research, user design and data practices. These training and data literacy initiatives will build on currently and previously offered trainings, including the statewide leadership development programs which focuses on results-driven leadership, and a workshop on designing rigorous evaluations GovOps offered in partnership with The Abdul Latif Jameel Poverty Action Lab (J-PAL). ODI is also developing resources to guide the ethical use of data, including adapting the Ethics and Algorithms toolkit for use at the state, and the development of an ethical toolkit for other parts of the data lifecycle.

The California FY21-22 Budget establishes and supports the state’s first-ever Chief Equity Officer within the Government Operations Agency (GovOps), which will build upon and implement the work of the California Leads Taskforce. The Chief Equity Officer will develop a uniform framework for creating equitable policies, practices, and metrics for hiring and procurement. To further this work, the GovOps agency has also been tasked with creating the first-ever statewide strategic five-year plan to transform government operations.

The California Governor spoke about the importance of data systems at the inaugural board meeting for the Cradle-to-Career Data system, and continues to speak enthusiastically about the development of the system and power of data to build more equitable futures in California.

The California Strategic Growth Council recently launched a statewide Racial Equity Resource Hub, which centralizes resources from across the state to help advance racial equity, including detailed analysis of disaggregated data and data tools. The hub and affiliated efforts are building statewide capacity, conversation, and culture around achieving equity, including through the use of data.  

To encourage a more data-centric culture, the California Health and Human Services Agency’s Center for Data Insights and Innovation (CDII) hosted an annual Data Expo, which included 950 participants from across the state. The focus of the virtual 2022 Expo was on how data and insights can lead to more equitable policies and healthier outcomes for all Californians, to advance data literacy for staff, and support a robust data community.

The California Office of Data and Innovation (ODI) includes CalInnovate, which uses best practices in human-centered design and technology to improve services for everyone. The Data and Innovation Fund resources ODI to partner with state agencies to identify, understand and solve problems using research, evidence, data and technology.

In the California Department of Technology, the Technology Modernization Fund is innovating by funding faster and more rapid ways to modernize systems in IT that will yield quick and meaningful results for the people of California.

The state has a range of other innovative strategies to improve outcomes:

• $80 million in ARP SLFRF are being leveraged for legal services was partially based on positive impacts observed in a state-run randomized evaluation.
• The California Education Learning Lab supports innovative teaching techniques, with a special emphasis on STEM and closing equity gaps.
• The California Initiative to Advance Precision Medicine supports collaborative health research that leverages big data, imaging and machine learning, genome sequencing, and other tools to develop targeted and innovative strategies to address issues like Adverse Childhood Experiences and health disparities.
• The California COVID-19 Assessment Tool is an innovative assessment tool incorporating various statistical projections and scenarios to find potential COVID-19 hotspots, predict which hospitals might reach capacity, and proactively allocate resources to such hotspots. Further, California released the source data sets on California’s Open Data Portal, allowing the public to examine the data underlying the assessment tool.
• The California Health and Human Services Agency established the Center for Data Insights and Innovation in 2021, with the authority to help support and shape the Agency’s data strategy and improve the lives of all Californians by turning data into insights, knowledge, and action. CDII supports programs that are person-centered, equity-focused, and data-driven. To accomplish this, CDII integrates knowledge and skills from several disciplines including strategic communication, data analysis, and human-centered design to identify actionable insights and build new, safer approaches to sharing data.

In 2022, California established the Office of Data and Innovation and bolstered the statewide data team (CalData), building on the previous appointment of the Chief Data Officer and work of the Office of Digital Innovation. The CDO and CalData are responsible for creating data strategies that improve programs statewide and lead to better outcomes, with a particular focus on connecting traditionally siloed data. The Chief Data Officer created CalData: California’s Data Strategy 2020 and supports the California Open Data Policy, and the California Open Data Portal. The CDO’s office has three lines of work with corresponding leadership positions in: data analytics and evaluation, data services and policy and programs. The team has authority over data governance and management statewide and as a key part of their work engages with state departments in communities of practice.

In addition, there are several Chief Data Officers and data leaders throughout the state at the agency or department-level including the:

• Director of the Center for Data Insights and Innovation and Chief Data Officer at the California Health & Human Services Agency;
• Chief for Data Analytics and Strategy at the Department of Developmental Services;
• Geospatial Data Officer at the CalTrans, the Department of Transportation.

The California Cradle-to-Career System has a comprehensive and inclusive governance system. The system hired its first Director, who reports to the 21-person governing board made up of representatives from data providers, policymakers, and the community. Supporting the decision-making governing board are two advisory boards on data and tools and community engagement who ensure that data is actionable, high-quality, equity-oriented, and with strong feedback loops to the community. The system also has community conversations, open meetings, and other ways to get involved, maximizing community input. This structure is intended to make the system the most inclusive and actionable data system in the country.

California’s statewide open data policy (2019) encourages departments to share data in standard and accessible formats through the California Open Data Portal. In 2022, California published a new, user-friendly, simplified Open Data Handbook, to help improve consistency in data publishing, and encourage collaboration, transparency, innovation, and effectiveness.

In Spring 2021, all entities under the authority of the Governor of California signed a single Interagency Data Exchange Agreement (IDEA), an umbrella memorandum of understanding (MOU) that is intended to facilitate data exchange between state entities in California. It is both a process tool and a set of resources designed to help state entities exchange data more efficiently, while ensuring that the exchange is legal and secure. Under IDEA, signatory entities draft a Business Use Case Proposal (BUCP) that details the specifics of their data exchange. IDEA is supported by detailed handbooks, a community of practice, and a resource library.

In 2022, California launched (AB 133) the statewide Health and Human Services Data Exchange Framework, including guiding principles, policies and procedures, and a single data sharing agreement. The Stakeholder Advisory Group collaborates closely with CHHS and brings expertise and perspectives of a number of organizations and state agencies on the development of the framework. This statewide data-sharing agreement will accelerate and expand the exchange of health information among health care entities, government agencies, and social service programs. The ultimate goal is that all residents should receive healthcare and social services with the assurance that their service providers can access the information they need to provide safe, effective, whole person care—while keeping their data private and secure.  

In summer 2021, California passed legislation to create the Cradle-to-Career System, which will be a suite of user-friendly resources focused on early learning through K-12 and higher education, as well as on the financial aid and social services that help students reach their goals. It will include:

• Planning and application tools for students, families, and educators to streamline the college and financial aid processes and monitor student progress.
• A linked research data set that will be accessible via dashboards, query tools, and an analytical data set for researchers, policymakers, educators, and community members to inform research, advocacy, and policy analysis.
• Community engagement and training to raise awareness of the data system and ensure it can be used by students, families, educators, researchers, and policymakers.

In spring 2022, data sharing agreements with 15 education, social services, and workforce entities were signed, and the state started procurements that will ensure the system is efficient, secure and leverages modern tools.  

The Cross-Systems Analytics and Assessment for Learning and Skills Attainment (CAAL-Skills) is an interagency initiative that integrates data from workforce system partners in employment training and development, apprenticeship standards, social services, education, and corrections to enable the analysis and evaluation of programs. It is actively undergoing an academic evaluation and technical improvements to allow it to interface with other systems.

California Nature is a public Geographic Information System that supports efforts to conserve 30% of the state’s lands and waters by 2030 for biodiversity protection (30×30) and to use Natural and Working Lands (NWL) to mitigate climate change and increase resilience to its effects. CA Nature is built on authoritative data about the state’s biodiversity, land management, land cover, and present and projected climate.      

Additionally, the state launched the first of its kind integrated data system for people experiencing homelessness. Through this dashboard, California policymakers and the public can understand and create novel insights of people experiencing homelessness to end homelessness.

The California Governor’s Council Postsecondary Education’s Recovery for Equity team examined cross-agency disaggregated data to develop strategies, policies, and practices to help the state’s post-secondary institutions emerge from the pandemic stronger and better poised to serve all students, including informing the development of a K16 grant program which encourages grantees to commitment to implement at least four of the identified recommendations emerging from the council’s data analysis and report of findings.

The Strategic Growth Council provides technical assistance programs and guidance to help communities, particularly disadvantaged, low-capacity ones, to apply for and execute large state grants, including greenhouse gas measurements and projections, and assessing the potential impact of the projects. This assistance allows more communities to formulate robust plans and partnerships with the state to achieve      climate-goals. They have also evaluated their TA programs several times and track demographics of technical assistance providers to help diversify the pool.  

The Governor’s Office of Planning and Research, Strategic Growth Council and partners provide a number of data tools on community equity, planning, regional conservation and climate resiliency, including a one-of-a-kind statewide climate assessment with local decision-support data tools. These resources are then used by localities and non-profits in planning processes and for state-supported grants.

Various California state agencies have leaders and governance structures that support evaluation, including: Deputy Secretary of Evaluation, appointed in 2022, at the Labor and Workforce Development Agency; Director of Research, Policy and Legislation at the California Workforce Development Board, who leads workforce evaluation activities undertaken by the State plan and policy team; the Evaluation and Surveillance Section at the California Tobacco Control Program that oversees various tobacco-related health evaluations; Research Office and Chief of Research at the Board of State and Community Corrections which support the use of data, evaluation and evidence; and the Chief of Research at the California Air Resources Board (CARB), whose division conducts in-house research and funds external studies that inform health-based air quality standards, reducing air pollution exposures, and protecting California from impacts of climate change. This is accomplished through a diverse portfolio of projects that evaluate CARB’s air quality and climate programs and regulatory efforts. CARB also hosts a Research Screening Committee who reviews and provides recommendations on research proposals and final reports.

The California Office of Data and Innovation oversees statewide evaluation work and has begun the process of developing shared approaches. The K16 Collaborative grant program is an example of a program with a learning agenda and transparent, cooperative, and iterative process built with grantees that might serve as a model for state policies. The learning agenda includes relevant priority research questions, aligned with strategic goals, and associated evaluation activities that would contribute to evidence-building around those specific goals. Further, California state agencies and programs have individual learning or research agendas to accelerate the use of evidence:

• Department of Education articulates their research priorities;
• California Workforce Development Board (CWDB)  state plan includes priority research questions, evaluation activities, major findings, and how these findings tie to major questions, policy strategy and future evaluation plans;  
• California Tobacco Control Program published a report, based on previous evaluation plans to highlight the findings and their implications of past evaluations, how the results relate original questions and plan and lessons learned.  
• California Air Resources Board produces three-year strategic research plans which include syntheses of past research, priority research questions, equity considerations, and ongoing and future research and evaluation in health and the environment, including the use of behavioral economics. This plan is developed with an open public process with community engagement and informs development and implementation of policies and regulations.  

The California Workforce Development Board (CWDB) has funded third-party evaluation projects in amounts ranging from $350,000 to $750,000, typically representing at least two percent of program funds. CWDB’s policy to allocate resources for evaluation studies, conducted either by the CWDB Research and Evaluation Team or by third-party evaluators.

In 2021-2022, the California Tobacco Control Program in the Department of Public Health spent seven and a half percent on evaluation and surveillance efforts. They also require their local projects to direct a minimum of 10% of their local program funds toward evaluations and to designate at least one qualified evaluator as the lead, which they can find in the Local Evaluator Program Directory. They also fund the UC Tobacco Control Evaluation Center to provide technical assistance on evaluation efforts and develop resources, guidance, and training.

The California Department of Social Services created the California Evidence-Based Clearinghouse for Child Welfare, which allows child welfare providers and professionals to “identify, select, and implement evidence-based child welfare practices that will improve child safety, increase permanency, increase family and community stability, and promote child and family well-being.” This tool helps identify evidence-based practices and provides guidance and support for program implementation. The clearinghouse’s numerical rating scale categorizes programs into six tiers of evidence and uses a relevance scale as a complement to the scientific rating scale and to demonstrate applicability for client populations. In addition to the clearinghouse, various evaluations have impacted decision making in discrete programs and practices, including:

• $80 million in ARP SLFRF are being leveraged for legal services was partially based on positive impacts observed in a state-run randomized evaluation.
• The California Department of Transportation’s Road Charge Program has used extensive data collection and evaluation to inform key program process and design decisions for this innovative transportation program. This includes using an initial pilot study to identify additional studies, evaluations, community-based research, and equity work.
• The California Workforce Development Board (CWDB) has engaged in several recent evaluations on California job centers, training partnerships, regional plan implementation, reentry services, and serving other highly barriered populations. These findings are informing grantmaking and program design decisions, including emphasizing industry engagement in programs and promoting additional investment in metrics and evaluation.

In both 2021 and 2022, the governmental organization California Volunteers used a tiered evidence framework to allocate funds through its AmeriCorps California program. The 2021 Request for Proposals specified that within Program Design (worth 50% of total points), Evidence Tier was worth 12% and Evidence Quality was worth 8%. In 2022, Evidence Tier was given 6% and Evidence Quality 4%. Both RFPs required evidence by asserting that “applicants assessed in the Pre-Preliminary evidence tier who do not provide adequate responses to the Evaluation Quality standards will not meet the threshold requirements for this Funding Opportunity and will not be considered for competitive funding.” The 2021 RFP defined evidence through suggesting both the AmeriCorps Evidence Exchange and the Corporation for National and Community Service (CNCS), and the 2022 grant referenced the AmeriCorps Evidence Exchange. Applicants were encouraged to use interventions assessed as having ‘Moderate’ or ‘Strong’ tiered evidence, and to utilize the Mandatory Supplemental Guidance offered that further defined evidence.

In 2021, the California Department of Education used a tiered evidence-based framework from The California Evidence-Based Clearinghouse for Child Welfare (CEBC) to define evidence and allocate funding for the California Comprehensive Literacy State Development (CLSD) Grant program. Evidence-based strategies were required components for funding consideration. The Request for Proposals also allocated 8 points (out of 84) to grantees who described evidence-based practices and the evidence used to support the selected strategies. Within the RFP, grantees were encouraged to use evidence-based practices to direct attention towards disadvantaged children, including children living in poverty, English learners, and children with disabilities.

In 2021, the State Bar of California devoted $58.5 million to support its Homelessness Prevention Competitive Grant Program focused on providing legal services to prevent eviction, an evidence-based practice. The program awards up to 20 bonus points for applicants who focus on homelessness and housing insecurity within rural and underserved populations.