RESPONSES TO THE AUDIT:
The California Governor's Office of Emergency Services (CalOES)
August 4, 2023Grant Parks
California State Auditor
621 Capitol Mall, Suite 1200
Sacramento, CA 95814
Dear Mr. Parks,
The California Governor’s Office of Emergency Services (Cal OES) received the California State Auditor’s (CSA) High Risk Assessment 2023-601 redacted Draft Report on July 31, 2023, via encrypted email. Cal OES appreciates the opportunity to review and comment on the 2023 High Risk Assessment. Cal OES’s comments are solely on the section titled, “[Redacted] Aging Water Infrastructure Threaten California’s Water Supply and Public Safety.”
The CSA’s Report states: “Since our last assessment in 2021, [Redacted] However, Emergency Services’ approval of emergency action plans lags behind. Emergency Services has only approved emergency action plans–which outline action to be taken during an emergency to minimize or eliminate the potential for loss of life and property damage-for 419 of the nearly 900 dams required to submit such plans, or about 48 percent.”
①Cal OES has consistently met the deadlines in statute for reviewing submitted emergency action plans (EAPs) and will continue to work closely with dam owners on their EAPs so they are not only complete, but also effective for dam owners and affected public safety agencies downstream. In addition, Cal OES has created tools to assist dam owners in completing their EAPs including a template, review tool, and example of an acceptable plan.
The CSA’s Report states: “Although this number represents progress–an increase from the 107 approved plans in 2021–it will take several years at the current rate of approval for the State to have clear emergency plans in place for all dams that require them. Further, there are 121 dams without approved emergency plans [Redacted] having Extremely High downstream hazard ratings, indicating a risk of considerable loss of human life.”
Cal OES approved 312 EAPs for the two-year period of 2021 to 2023. That’s an increase of 292% from the 107 EAPs approved since CSA’s last assessment in 2021. Furthermore, of the mentioned 121 Extreme High Hazard (EH) EAPs, 72 are either currently under review or have been reviewed at least once, returned, and Cal OES is awaiting resubmission. Additionally, another 41 EAPs belong to dam owners that have more than one EAP to submit. Experience dictates that once these owners have completed a satisfactory EAP, their subsequent submittals will be better quality and in relatively rapid succession.
Existing law does not establish a deadline for the dam owner to resubmit an EAP returned for correction. Cal OES’s historical practice has been to reach out to the dam owners after six months of no contact. Cal OES will reduce the time of follow-up to one month, and each month thereafter until the corrected EAP is resubmitted.
Lastly, the CSA’s Report states “…the large number of emergency action plans yet to be approved by Emergency Services shows that sufficient corrective action has not yet occurred.”
①Cal OES has taken significant corrective actions and continues to work on EAP approvals. Cal OES anticipates the tools we’ve developed, current approval processes, and administrative changes in the timing and frequency of outreach will reduce the number of outstanding EAPs.
Cal OES appreciates the assistance and guidance offered during CSA’s assessment. If you have additional questions or concerns, please contact Ralph Zavala, Cal OES Internal Audits Office Chief, at (916) 845-8437.
Sincerely,
Original PDF copy signed
NANCY WARD
Director
c: Ralph Zavala, Chief, Internal Audits Office
CALIFORNIA STATE AUDITOR’S COMMENT ON THE RESPONSE FROM THE CALIFORNIA GOVERNOR’S OFFICE OF EMERGENCY SERVICES
To provide clarity and perspective, we are commenting on the response to our assessment from CalOES. The number below corresponds with the number we have placed in the margin of the response.① Although we indicate that CalOES has made some progress in approving emergency plans on page 21, 121 dams with extremely high downstream hazard ratings—indicating a risk of considerable loss of human life—still lack approved plans.
The California State Transportation Agency
Date: August 4, 2023To: Grant Parks
California State Auditor
From: Carlos Quant
Deputy Secretary, Budget and Administration
California State Transportation Agency
Subject: California State Transportation Agency Response to 2023-601—State High-Risk Assessment
To Whom it May Concern:
CalSTA concurs with the Auditor’s findings. This decision is a testament to the substantial progress Caltrans, the California Transportation Commission and our partners have made as we work together to improve our state’s critical transportation infrastructure. This progress has been especially noteworthy since the passage of Senate Bill (SB) 1, the Road Repair and Accountability Act of 2017 – landmark legislation that ushered in a new era of infrastructure investment to rebuild California. Our elected officials and the people of California entrusted us with their hard-earned tax dollars to upgrade the state’s aging infrastructure, and we have delivered and will continue to make good on that trust. Coupled with Governor Newsom’s infrastructure streamlining package and a $15 billion investment in clean transportation infrastructure, along with recent increased federal infrastructure funding, our state is in an incredible and unique position to keep making progress and accelerate our transition to a cleaner, safer, more equitable and more connected transportation system that benefits all Californians.
Please contact Carlos Quant with any questions at Carlos.Quant@calsta.ca.gov.
Thank you,
Carlos Quant
Deputy Secretary, Budget and Administration
The California Department of Public Health
August 4, 2023Grant Parks
California State Auditor
621 Capitol Mall, Suite 1200
Sacramento, CA 95814
Dear Mr. Parks:
The California Department of Public Health (Public Health) thanks the California State Auditor for the opportunity to comment on its draft report of the updated assessment of high-risk issues faced by the State and select State agencies.
Public Health appreciates that CSA acknowledges the signficant progress we have made to eliminate the concerns that initially identified us as a high-risk department. Removal from this list represents great efforts made by numerous Public Health programs and employees to adopt previous audit recommendations and implement corrective action to control risk to the public.
We take seriously our charge to advance and protect the health and well-being of California’s diverse peoples and communities and have worked to quickly address recommendations that could pose a risk to residents’ overall health or safety.
Public Health understands we still have a limited number of outstanding recommendations and commit to continuing to fully implement these. We will report our progress to the State Auditor at the designated time intervals.
Thank you for the opportunity to respond to the assessment. If you have any questions, please contact Rob Hughes, Deputy Director, Office of Compliance, at (916) 306-2277.
Sincerely,
Tomás J. Aragón, M.D., Dr.P.H.
Director and State Public Health Officer
The California Department of Technology
August 3, 2023Grant Parks (via GovOps Agency Secretary Amy Tong)
California State Auditor
621 Capitol Mall, Suite 1200
Sacramento, CA 95814
SUBJECT: 2023-601 – STATE HIGH-RISK UPDATE – INFORMATION SECURITY AND STATE IT PROJECTS
Dear Mr. Parks:
The State’s Information Security Remains a High-Risk Issue:
The California Department of Technology (CDT) acknowledges that statewide information security faces significant risks, given the increasingly complex and sophisticated cyber threat landscape. This year has witnessed a remarkable increase in the frequency and sophistication of cyber-attacks globally – greater than those that occurred in the last five years combined. In response to these heightened threats, we coordinated with State and Federal partners to significantly enhance State cybersecurity resiliency measures. As a result, annually the enhanced measures have successfully deterred more than 300 confirmed attacks and mitigation of vulnerabilities. The CDT oversight program is fully modernized to focus on supporting reporting Agencies with operational resiliency to respond to modern cyber-attacks.
①The California State Auditor’s (CSA) high-risk audit focused on a subset of the CDT efforts and failed to acknowledge how CDT baselines risk and actions performed to enhance statewide maturity and resiliency. The CSA referenced 52 reviews underway. These reviews are the CDT Information Security Program Audits (ISPA) that audit the highest-risk departments against policy compliance measures. In addition, we provide tailored help to entities and remediate gaps for the remaining lower-risk departments.
The CDT employs a thorough approach to assess and strengthen the security of all State entities. This comprehensive approach involves the Information Security Program Audits (ISPA), Independent Security Assessments (ISA), attack surface analysis, and the National Cybersecurity Review (NCSR). In addition to these compliance baseline measures; the CDT has evolved its oversight program to support departments addressing any identified issues. This program provides remediation assistance through consultative advisory services to help departments manage internal risks. The CDT has expanded its services 24 by including seven continuous threat detection and response services for the internal networks of the least resilient departments identified through this oversight approach. Through this program, the CDT can ensure a higher level of statewide protection.
②The CSA references 52 of 107 entities reviewed in a four-year period. These audits are solely focused on high-risk entities as determined by CDT. The remaining entities which are in the low-risk pool are scored using ISAs, the external attack surface baselines, NCSR, and a comprehensive review of their risk remediation plans. Both entity types receive remediation services through programs within our advisory, with tailored hands-on guidance, continuous monitoring of their internal networks, and other operational services provided within our California Cybersecurity Integration Center (Cal-CSIC).
The objective is to provide a clear outline of the oversight program designed to measure risk within the CDT framework. The program specifically covers eight state agencies under the Executive Branch, and several un-affiliated and independent state agencies, totaling 139 unique entities:
- The program conducted policy audits for 61 unique departments.
- More than 85 audits were conducted, considering re-audits and departments that received progress check-ins.
- A total of 250 technical vulnerability assessments were performed, with 121 unique departments receiving technical vulnerability assessments. A subset of these departments receives an audit due to their high-risk nature.
- The program generated 115 unique NCSR scores.
- Additionally, 139 baseline attack surface scores were calculated.
As CDT baselines entities through oversight measures by aligning operational efforts to build in resiliency to help them with protective measures.
- Over 30 resource-constrained departments are receiving continuous internal and wide area network (WAN) monitoring utilized by all departments.
- Implemented a Vulnerability Disclosure Process and remediated over 50 external risks.
- The advisory services team conducted over 240 tailored workshops providing internal and hands-on assistance to remediate risk for departments.
- Delivered 18 policies and standards and 70 example templates for departmental internal use.
- Established the Technology Stabilization Fund to help departments with stabilizing critical services.
- Expanded staffing within advisory services by seven to help with independent organizations such as the referenced California Public Employees’ Retirement System (CalPERS), California State Teachers’ Retirement System (CalSTRS), and the State Controller’s Office (SCO).
- Established incident response teams with CDT and Cal-CSIC to minimize the impact of cyber incidents and provide rapid recovery for affected departments such as the referenced Department of Finance (DOF) incident.
The frequency and sophistication will continue to increase, and new technology will continue to expand the threat landscape. As the threat landscape continues to evolve, CDT will adapt its oversight program to encompass standards, advisory, and operational measures. We appreciate input from CSA on any aspect of our measures. The CDT takes a holistic, risk-based approach to oversight and remediation, as focusing on policy reviews and audits alone is not sufficient.
CDT Has Not Made Sufficient Progress in its Oversight of State Information Technology Projects:
③ ④CDT’s Project Approval Lifecycle (PAL) process is a robust and rigorous planning process designed to mitigate the risk of failed IT projects and protect the State’s technology investment. CDT asserts that the PAL process and subsequent project oversight functions have significantly improved project outcomes. CDT provided data to the CSA in response to the CSA’s 2022-114 audit indicating the current project outcomes are at or positively above industry benchmarks for IT projects as follows: The Department of Technology conducted an analysis in 2018 in response to a Legislative Analyst Office (LOA) request.
We compared the State of California project outcomes of 178 I projects to the Standish annual CHAOS report of 2014 as follows:
| 2007 - 2017 | California State | Industry Average |
|---|---|---|
| Successful Projects | 28% | 16.2% |
| Challenged Projects | 63% | 52.7% |
| Failed Projects | 9% | 31.1% |
⑤CDT has since updated our analysis: Comparison to the 2020 Standish Group Report shows the following:
| FY 2018 – FY 2022 | California State | Industry Average |
|---|---|---|
| Successful Projects | 67% | 31% |
| Challenged Projects | 33% | 50% |
| Failed Projects | 0% | 19% |
CDT attributes this improving trend to a higher quality planning, improvements to the oversight process since 2017, and early escalation of issues to state entity leadership when required.
⑥The PAL planning process is scalable based on the IT project's business and/or technical complexity. This means that the more complex a project is, the more planning is needed. The CSA did not include information provided by CDT about all factors that contribute to the time duration of the PAL process, some of which are outside the CDT control, including the annual budget process, changes in state entity business priorities, funding availability, skills and experience of the department staff, and the quality of the state entities planning documents. The CDT has experienced PAL durations as short as 36 days.
In March 2023, the CDT made improvements in the PAL process in response to a yearlong improvement effort and evaluation via interviews of various stakeholders, including three Agencies, 12 Departments, the Legislative Analyst Office, the Department of Finance, and external experts from the Project Management Institute and Sacramento State University. The objectives achieved were reducing burdensome pain points, processes that adapt to various project management methodologies, streamlined processing, challenge-based procurements, and ADA compliance.
⑦As part of CSA’s audit, the team reviewed CDT's oversight of four IT projects and found that although three were identified that required immediate corrective action, CDT had not used its authority to ensure the problems were resolved.
- CWS-CARES
- Caltrans TAMS
- DMV DxP
- FI$Cal
The auditor noted that CDT issued no corrective action plan letter related to these four projects. However, it does not mean that department corrective action plans or corrective actions were not initiated because of CDT recommendations or escalations.
The Auditor chose four projects from a set of 240 projects and analyzed the performance data provided to the auditor. However, the auditor did not provide information about the effectiveness of project oversight for the remaining 236 projects of the State project portfolio. The details on the effectiveness of project oversight were given only for the four selected projects.
Three of the four projects examined by the auditor [CWS-CARES, DMV DxP, and Caltrans TAMS] have not been completed and are still in progress. The ability to determine whether the original project requirements, as defined by the scope of work, were delivered on time is premature. The fourth project, FI$Cal, is complete and provides large-scale benefits to the state as planned. The conclusions drawn from the remaining three projects should not be used to infer the effectiveness of the planning and oversight work over the entire State project portfolio. In the case of CWS-CARES, currently beginning implementation, significant CDT intervention took place, resulting in major changes in the development planning. Regarding the DMV DxP project, phase-1 implementation was successful. Phase-2 risks were identified, and CDT escalated these to executive management resulting in additional planning activities to address the risks identified in the oversight reports and escalation. For Caltrans TAMS, CDT provided observations, recommendations, and guidance, resulting in the termination of a vendor contract and a currently underway re-procurement.
We remain committed to the ongoing evaluation of the effectiveness of the planning and oversight activities as we continue to explore methods to effectively correlate project planning and oversight efforts to successful project outcomes.
Thank you for the opportunity to respond to your draft High-Risk Review report regarding CDT. Please contact Kirk Marston at 916-208-6896, if you have questions.
Sincerely,
Liana Bailey-Crimmins
State Chief Information Officer and Director
California Department of Technology
cc: Amy Tong, Secretary, Government Operations Agency
Jared Johnson, Chief Deputy Director, California Department of Technology
- Next page: RESPONSES TO THE AUDIT: (Pg 2 of 2)
- Previous page: REMOVED HIGH-RISK AGENCIES AND ISSUES:
