DHS OIG, OIG-17-60, CBP Continues to Improve its Ethics and Integrity Training, but Further Improvements are Needed (2017)

CBP Continues to Improve its Ethics and Integrity Training, but Further Improvements are Needed May 31, 2017 OIG-17-60 DHS OIG Highlights CBP Continues to Improve Its Ethics and Integrity Training, but Further Improvements are Needed May 31, 2017 Why We Did This Audit In House Report 114–215 (July 21, 2015), which accompanied the Consolidated Appropriations Act, 2016, Public Law No. 114–113, the U.S. House of Representatives, Committee on Appropriations, directed the Office of Inspector General to assess and report on CBP’s ethics and integrity training for its agents and officers [H.R. Rep. No.114– 215, at 22 (2015)]. We conducted this audit to determine whether CBP made training improvements; tracked training completion and effectiveness; and communicated in a comprehensive manner its integrity strategy. What We Recommend We made two recommendations for CBP to improve its oversight and management of its ethics and integrity training.

For Further Information: Contact our Office of Public Affairs at (202) 254-4100, or email us at [email protected] What We Found U.S. Customs and Border Protection (CBP) has made improvements to, and continues to develop, its ethics and integrity training for officers and agents. It tracks training completion, and has begun to measure and assess training effectiveness. However, the Performance and Learning Management System used to track training completion needs improvement. Also, locally developed training content on ethics and integrity varies by location and operating environment, and CBP does not maintain a repository or any formal process for the field to share locally developed information.

As a result, CBP misses valuable opportunities to deliver consistent high-quality ethics and integrity training courses across multiple operating environments and components. Finally, CBP has not effectively communicated or followed up with the field on its overall Integrity and Personal Accountability Strategy. One purpose of the strategy is to ensure that ethics and integrity training is provided for all CBP employees. More broadly, the strategy aims to promote a culture of integrity and accountability by increasing awareness through messaging, training, and enhanced communication.

If employees have not received or do not understand the importance of the integrity strategy, CBP cannot succeed in achieving this important initiative. CBP Response CBP concurred with the two recommendations and has initiated actions that should improve its ethics and integrity training. We consider recommendation 1 resolved and open and recommendation 2 resolved and closed. www.oig.dhs.gov OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Securlty Washington, DC 20528 / www.oig.dhs.gov May 31, 2017 MEMORANDUM FOR: Kevin K. McAleenan Acting Commissioner U.S. Customs and Border Protection Christopher Hall Assistant Commissioner Office of Training and Development U.S. Customs and Border Protection John Roth ~~~~ Inspector General CBP Continues to Improve its Ethics and Integrity Training, but Furlher Improvements are Needed FROM: SUBJECT: Attached for your action is our final report, CBP Continues to Improve its Ethics and Integrity Training, but Furlher Improvements are Needed. We incorporated the formal comments provided by your office.

The report contains two recommendations aimed at improving CBP's ability to more effectively oversee and manage its ethics and integrity training program and foster a workforce culture of integrity. Your office concurred with the recommendations. Based on information provided in the response to the draft report, we consider recommendation 1 resolved and open and recommendation 2 is resolved and closed. Once your office has fully implemented the recommendation, please submit a formal closeout letter to us within 30 days so that we may close the recommendation.

The memorandum should be accompanied by evidence of completion of agreed-upon corrective actions. Please send your response or closure request to [email protected]. Consistent with our responsibility under the Inspector General Act, we will provide copies of our report to congressional committees with oversight and appropriation responsibility over the Department of Homeland Security. We will post the report on our website for public dissemination.

Please call me with any questions, or your staff may contact Maureen Duddy, Deputy Assistant Inspector General for Audits, at (202) 254-4100. Attachment OFFICE OF INSPECTOR GENERAL Department of Homeland Security Table of Contents Background .................................................................................................... 1 Results of Audit ............................................................................................. 3 CBP Could Share Local Best Practices More Effectively and Emphasize Its Integrity Strategy ...........................................................

4 The Performance and Learning Management System for Tracking Training Completion Needs Improvement ............................................... 6 CBP Has Begun to Measure and Assess Training Effectiveness and Apply Evaluation Data .......................................................................... 7 Conclusion...................................................................................................... 9 Recommendations.........................................................................................

10 Appendixes Appendix A: Objective, Scope, and Methodology ................................. 12 Appendix B: CBP Comments to the Draft Report .................................. 16 Appendix C: Ethics and Integrity Training Improvements Since FY 2012 ................................................................. 19 Appendix D: The Kirkpatrick Model Levels of Measurement, Benefits, and Limitations ..............................................................

21 Appendix E: Office of Audits Major Contributors to This Report .......... 23 Appendix F: Report Distribution .......................................................... 24 Abbreviations AMO ATC CBP CFR DUI GAO HQ OCC OFO OIG Air and Marine Operations Advanced Training Center U.S. Customs and Border Protection Code of Federal Regulations Driving Under the Influence Government Accountability Office Headquarters Office of Chief Counsel Office of Field Operations Office of Inspector General www.oig.dhs.gov OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security OPR OTD PALMS P.L. SIP SLCP SOP TRAEN USBP Office of Professional Responsibility (formerly Internal Affairs) Office of Training and Development Performance and Learning Management System Public Law Self-Inspection Program Second Level Command Preparation Standard Operating Procedure Training, Records, and Enrollment Network U.S. Border Patrol www.oig.dhs.gov OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security Background U.S. Customs and Border Protection (CBP) is one of the Department of Homeland Security’s largest and, according to CBP, one of the most complex components, with a priority mission of keeping terrorists and their weapons out of the United States. To add to its complexity, CBP is also responsible for securing the border — approximately 7,000 miles of land borders and 95,000 miles of coastline — and facilitating lawful international trade and travel.

The agency takes a comprehensive approach to border management and control, combining customs, immigration, border security, and agricultural protection into one coordinated and supportive activity. CBP recognizes that its employees operate in an environment in which even a single corrupt act may pose a threat to national security; thus, a culture of integrity and a strong, aggressive program to address corruption and misconduct are vital to the agency and to the Nation. In 2011, the Homeland Security Studies and Analysis Institute completed a review of CBP’s workforce integrity. The Institute found that CBP did not have comprehensive guidance for its integrity programs and initiatives.

One year later, the Government Accountability Office (GAO) GAO Report-13-59, Additional Actions Needed to Strengthen CBP Efforts to Mitigate Risk of Employee Corruption and Misconduct, similarly concluded that CBP did not have an integrity strategy as required in the Secure Borders, Safe Travel, Legal Trade: U.S. Custom’s and Border Protection Fiscal Year 2009–2014 Strategic Plan. In light of these reports, CBP began drafting an Integrity and Personal Accountability Strategy (integrity strategy), which was formally announced in September 2014. That same month the DHS Secretary delegated criminal misconduct investigative authority to CBP as part of what the former Commissioner called “a larger effort to hold the workforce accountable for maintaining a high standard of integrity and aligning CBP with law enforcement best practices throughout the country.” CBP officials report that the agency’s integrity strategy establishes a unified and multi-layered approach organized around four related mission areas: prevention, detection, investigation, and response to corruption and misconduct. The strategy also addresses cross-cutting issues of integration and awareness.

In a cover memo, CBP’s Commissioner defines personal integrity as “a commitment that each of us makes to doing the right things, the right way, at the right time, including the reporting of inappropriate, corrupt, or criminal actions when observed.” The integrity mission is to “strengthen CBP’s culture of unwavering integrity and professionalism by increasing awareness and implementing comprehensive, integrated programs and capabilities designed to www.oig.dhs.gov 1 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security prevent, detect, investigate, and respond to instances of corruption and misconduct across all levels in the agency.” The implementation process requires measuring, analyzing, and ensuring CBP’s collective actions reinforce integrity as a core value. Within CBP, the Office of Training and Development (OTD) is responsible for centralized leadership and direction of all training programs, as well as the continuous improvement and expansion of CBP learning capabilities. This includes training at the: x Advanced Training Center — a CBP training center that houses the Instructional Design Center, Distance Learning Center, and The Law Enforcement Officer/Agent Safety and Compliance Directorate; formerly, the Use of Force Center of Excellence; x Pre-Academy — training at duty stations prior to the academy; x Basic Academy — the Boot Camp for officers and agents; and x Post-Academy — on-the-job training. CBP officials said that field training is managed by individual field locations and aligns with each unique operating environment.

The Office of Chief Counsel (OCC) provides mandatory ethics training to select employees subject to financial disclosure rules, and all CBP employees are required to take an online Integrity Awareness course annually. Instead of routinely offering standalone courses in ethics and integrity, CBP typically includes this type of training in more comprehensive courses. As an illustration of ongoing congressional interest in this area, House Report 114–215 explaining the fiscal year appropriations for the Department of Homeland Security, the U.S. House of Representatives, Committee on Appropriations, directed the Office of Inspector General to “assess and report on CBP’s ethics and integrity training for its agents and officers, as well as CBP mechanisms for operational oversight related to ethics and integrity” [H.R. Rep. No. 114-215, at 22 (2015)].

Because of the broad and undefined nature of this mandate, we corresponded with congressional staff who acknowledged the steps CBP had taken to address concerns regarding allegations of misconduct associated with CBP’s rapid hiring of border patrol agents. Those steps included improving training and assuming criminal investigative authority from U.S. Immigration and Customs Enforcement for allegations of CBP misconduct. Nevertheless, concerns remained, and OIG was tasked with determining how training has improved, and how CBP is tracking whether training is effective, as well as providing rigorous oversight with regard to allegations of personnel conduct. As a result, we evaluated ongoing OIG projects and found another www.oig.dhs.gov 2 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security audit team was looking at whether conduct and discipline is equitable and consistent department-wide.

Although the DHS Conduct and Discipline audit will not directly answer whether CBP is providing rigorous oversight with regard to allegations of personnel misconduct, it will include a high-level overview of CBP’s process for reporting and addressing misconduct, as well as some limited testing of CBP’s disciplinary action cases. We then revised our objective to focus on whether CBP made training improvements; tracked and measured training completion and effectiveness; and communicated in a comprehensive manner its integrity strategy. Results of Audit CBP has made improvements to, and continues to develop, its ethics and integrity training for officers and agents. The agency tracks training completion using various methods and evaluation tools to measure and assess training effectiveness.

As a result, CBP has created, expanded, and redesigned training courses to promote ethics and integrity at the Advanced Training Center (ATC), the academies, and in the field. However, CBP has not effectively communicated or conducted follow-up with the field on its overall integrity strategy. As a result, CBP cannot fully accomplish the goals of this important strategy. CBP mandates annual integrity training for all employees.

CBP is developing ethics training for all employees, which is expected to become operational on Performance and Learning Management System (PALMS) during calendar year 2017. It develops and delivers instruction on these topics in multiple academy and ATC modules. Locally developed training on ethics and integrity varies by location and operating environment, and CBP does not maintain a repository or any formal process for the field to share locally developed information. Consequently, CBP misses valuable opportunities to deliver consistent, high- quality ethics and integrity training courses across multiple operating environments and components.

CBP tracks and assesses training completion using PALMS for all agency-wide mandatory training. In addition, academies use ACADIS — an electronic personnel and training system — to store and track academy training-related data. Nonetheless, there are concerns over PALMS reporting functionality and accuracy. Having reliable data is essential for CBP to meet fully its reporting requirements and oversee its training program.

CBP uses various evaluation tools to measure and assess training effectiveness. For example, OTD began formally using the Kirkpatrick Model in late 2015 to evaluate its advanced training and academy courses, while www.oig.dhs.gov 3 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security continuing to rely on End-of-Course Evaluation forms to obtain training participant feedback in the field. According to the DHS Learning Evaluation Guide, dated April 2016, although it is not the only model to evaluate training, the Kirkpatrick Model is well suited for DHS needs. Evaluating training effectiveness is an ongoing process.

Finally, CBP has not effectively communicated or followed up with the field on its overall integrity strategy. One purpose of the strategy is to ensure that ethics and integrity training is provided for all CBP employees. More broadly, the strategy aims to promote a culture of integrity and accountability by increasing awareness through messaging, training, and effective communication. If employees have not received or do not understand the importance of the integrity strategy, CBP cannot succeed in achieving this important initiative.

CBP Could Share Local Best Practices More Effectively and Emphasize Its Integrity Strategy CBP has made improvements in the development and delivery of ethics and integrity training. The agency has introduced new courses and programs for its employees, its academies, and its field locations. However, it could share local best practices more effectively. CBP developed an integrity strategy to promote a culture of integrity and accountability by increasing awareness through messaging, training, and effective communication.

Yet, CBP has not effectively communicated or followed up with the field on its overall integrity strategy. Training Improvements and Best Practices CBP policy requires the component to maintain a workforce that demonstrates high standards of ethical and professional conduct to ensure the proper performance of government service. One purpose of CBP’s integrity strategy is to ensure that ethics and integrity training is provided for all CBP employees. CBP has taken notable steps to improve the quality and availability of ethics and integrity training.

Instruction on this topic is included in multiple courses offered to employees, supervisors, and managers; in-person at the ATC; and through various training methods at CBP’s academies and field locations (see appendix C). Another improvement is OCC’s planned fiscal year 2017 implementation of its annual and recurring ethics training for all of its employees, which is offered on-line through PALMS. OTD works with the Office of Field Operation (OFO), U.S. Border Patrol (USBP), and Air and Marine Operations (AMO) academies to develop the pre-, basic, www.oig.dhs.gov 4 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security and post-academy training, including ethics and integrity elements. According to CBP officials, the training program at the academies has been improved significantly through scenario-based training.

We also found and observed many examples of high-quality training developed in the field. This training was not shared throughout CBP or with other field locations because CBP does not have a repository or any formal process to facilitate that result. The OFO field offices, USBP sectors, and AMO units (collectively, “the field”) have autonomy to develop training unique to their locations, missions, and operating conditions. According to field officials, they coordinate with their local OCC representatives to develop and deliver ethics training sessions.

Specifically, when we visited the USBP at Laredo Sector, personnel reported working with OCC to add an ethics training module in 2016 to its week-long annual training. OCC officials said that the ethics class, depending on the number of agents’ questions leads to valuable integrity-related discussions. Aspects of the annual training include mandatory training and others, such as fraud document detection; an Intel session; relevant policies; first aid/CPR; financial literacy; and weapons and agent specific re-certification. During the Tucson field visit, we found best practices at OFO, USBP, and AMO.

For example, the USBP at Tucson Sector developed the following programs and initiatives to address its unique operating environment and ethics and integrity issues: x A Tactical Awareness Training Program now part of basic academy training was developed after two agents were assaulted and did not have the necessary skills to defend themselves. This program conveys agents’ rights and responsibilities pertaining to use of deadly force and threat assessments. It also covers mindset, agent preparedness, handling situations appropriately, security, and situational awareness. x The Driving-Under-the-Influence (DUI) initiative was developed in Tucson Sector after Agent Michael V. Gallagher was killed by a drunk driver. The associated mandatory DUI training touches all agents and focuses on making sure employees understand the impact of their decisions, on and off duty.

This mandatory DUI training is conducted nationwide. x The Report Writing and Courtroom Testimony course was created by OCC at Tucson Sector to improve agent abilities in both writing and testifying. This training helps agents convey evidence-based and corroborated facts, and www.oig.dhs.gov 5 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security prepares them for cross examination and credible testimony. This Report Writing and Courtroom Testimony training is offered nationwide. Although there are several options for CBP employees to obtain ethics and integrity training, the agency does not have a repository or any formal process for the field to share locally developed training for use agency-wide.

As a result, CBP misses valuable opportunities to deliver consistent component-wide ethics and integrity training. A CBP training official said sharing locally developed training through a course catalogue for all components would be beneficial. CBP’s Integrity Strategy Throughout our site visits, senior leaders and training officers at Laredo’s USBP, AMO, and OFO field locations stated that they were unaware of the integrity strategy. Additionally, officials at the USBP, AMO, and OFO academies also had not heard of this document.

CBP’s integrity strategy outlines the responsibility of every person in CBP to internalize and promote a culture of integrity. It calls for an implementation process to measure and analyze the collective reinforcement of integrity as a core value and the need for awareness through integrity messaging, training, and effective communication. According to Office of Policy and Planning officials, senior leaders in the field should have communicated the integrity strategy’s goals and objectives to their respective field locations. Without headquarters (HQ) officials effectively communicating the strategy and having meaningful follow-up with senior field leaders, CBP cannot ensure successful implementation of the integrity strategy.

The Performance and Learning Management System for Tracking Training Completion Needs Improvement CBP uses multiple processes and systems to track ethics and integrity training. Directors and program managers at HQ and field offices self-report the completion of all mandatory training through CBP’s Self-Inspection Program (SIP). CBP’s Management Inspection Division is responsible for validating annual SIP reports. Additionally, CBP uses PALMS to track completion of its online CBP-wide annual ethics and integrity awareness training.1 CBP also uses training attendance sign-in sheets, certificates of completion, manual files, 1 Prior to June 2015, CBP used Training Records and Enrollment Network System (TRAEN) to track training completion. www.oig.dhs.gov 6 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security spreadsheets, and post-academy files to track its locally developed field training.

Academies also use ACADIS — an electronic personnel and training system — to store and track academy training-related data. According to DHS, CBP HQ, and field officials, in 2015 and 2016 PALMS was neither accurate nor was the reporting module fully operational. These deficiencies led CBP’s SIP division to waive training reporting for fiscal year 2015. We observed CBP officials at the OTD in May 2016 unsuccessfully attempt to generate CBP reports of completed and overdue training, noting PALMS error messages and inaccurate totals.

During field visits in July and August of that same year at Laredo and Tucson, we observed, and training officials said, they could not run accurate reports of completed and overdue training. Five months later in October 2016, the PALMS vendor developed a temporary solution to produce accurate CBP HQ level reports. However, field office training personnel remain unable to generate accurate training reports through PALMS. CBP HQ officials said the vendor will implement a solution to this problem sometime in 2017.

Without consistent and accurate PALMS reporting, CBP: x duplicates efforts to run reports and ensure training completion; x cannot determine who received, completed, and needs to attend training — meaning staff may not receive critical training to complete their job functions and mission requirements; and x cannot satisfy mandated reporting requirements and plan for future training needs. Having reliable data is essential for CBP to properly manage its training program. A DHS OIG review of the implementation and functionality of PALMS is ongoing in a separate audit, with an anticipated final report in fiscal year 2017. As a result, we do not make a recommendation to address this issue in this report.

CBP Has Begun to Measure and Assess Training Effectiveness and Apply Evaluation Data Although CBP does not measure the effectiveness of every individual training course, it does have an overarching methodology to evaluate its training program. For example, OTD and academies use the Kirkpatrick Model to measure the overall effectiveness of their own training courses; and the field uses OTD’s End-of-Course Evaluation form. The latter operates in the same way as Kirkpatrick level 1 to obtain training participant feedback. This distinction is due to OTD’s decision to formally adopt the Kirkpatrick Model in www.oig.dhs.gov 7 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security late 2015 as its tool for evaluating OTD advanced courses.

Based on evaluation feedback from Kirkpatrick and OTD’s End-of-Course Evaluation forms, CBP has created, expanded, and redesigned training courses to promote ethics and integrity at OTD, the academies, and in the field. Federal regulations require agencies to evaluate their training programs at least annually to determine how well they contribute to mission accomplishment and meet organizational performance goals.2 DHS defines training evaluation as a process used to measure the value and effectiveness of training. The Kirkpatrick Model for evaluating training effectiveness is an industry standard. The April 2016 DHS Learning Evaluation Guide makes note that, although the Kirkpatrick Model is not the only model to evaluate training, it is well suited for DHS needs.

The Kirkpatrick Model provides a framework for organizations to measure how well training provides employees with the necessary skills, knowledge, and abilities to successfully accomplish their jobs and support the organization’s mission and goals. This framework contains four different levels of evaluation used to measure the effectiveness of training. Table 1 highlights the four evaluation levels outlined in the DHS Learning Evaluation Guide. See appendix D for a detailed explanation of the evaluation levels, as well as the model’s benefits and limitations.

Table 1: Kirkpatrick’s Four Levels of Evaluation Measurement Results Level 1- Reaction To what degree trainees react favorably to the learning event Level 2- Learning Level 3- Behavior Level 4- Results To what degree trainees acquire intended knowledge/skills from the learning event To what degree trainees apply what they learned during training back on the job To what degree targeted outcomes occur, as a result of the training and reinforcement Negative reactions can indicate potential shortcomings with a training course Student performs required task while in the training environment Transfer of learning from training to work environment, identifying barriers Return on investment, residing at executive or training director level for analysis Feedback Methods Surveys, evaluations, and questionnaires Knowledge test and skill demonstration Surveys, observations, and interviews Questionnaires, evaluations, and surveys Source: OIG analysis of the DHS Learning Evaluation Guide, dated April 2016 2 Title 5 Code of Federal Regulations (CFR) § 410.202 Planning and Evaluating Training www.oig.dhs.gov 8 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security Since OTD issued its Standard Operating Procedure (SOP) in November 2015, officials said they tested the Kirkpatrick Model levels 1 through 3 to determine the most appropriate use of the model for each OTD-developed course. OTD provided OIG with an evaluation plan for the Second Level Command Preparation (SLCP) course that detailed the entire training evaluation. The SLCP plan specifies the type of information that needs to be collected; the process for collecting the data; roles and responsibilities of CBP officials; and how to use the data collected to evaluate training courses and make improvements. The OTD official said each OTD-delivered course has a unique evaluation plan in draft or final format for implementation in fiscal year 2017.

The USBP, OFO, and AMO academies use the Kirkpatrick Model to measure training program effectiveness and currently use the ACADIS system to manage the results. Level 3 feedback is reviewed to improve curriculum using comprehensive reports including data provided by participants and their supervisors or training officers. Participant feedback is collected in the End-of- Course Evaluation form for field developed and locally delivered training to measure training effectiveness and assist in future course development. Finally, at the Laredo and Tucson field offices, USBP officials said they meet regularly and review training feedback to enhance existing and future instruction, like the Leadership and Supervisors training in Tucson and the 1- hour ethics training in Laredo.

As a result of CBP’s evaluation tools and data collection, OTD has expanded its course development and improved the manner of delivery for its 2016 Revised Senior Leadership and other training. All academies have enhanced ethics- related training modules and integrated more scenario-based training for officers and agents. These trainings as listed in appendix C include Basic Training Program Redesign Academy; Post-Academy Training Redesign; Enhanced Scenario Based Training; and OCC-Nationality and Immigration Law and Applied Authorities. As CBP officials continue to assess evaluation results and the best use of the Kirkpatrick Model, they will be able to make more informed decisions on training improvements.

Conclusion CBP has made improvements to, and continues to develop, its ethics and integrity training for officers and agents. It tracks training completion using various methods and evaluation tools to measure and assess training effectiveness. This has led to the creation, expansion, and redesign of courses to promote ethics and integrity at the Advanced Training Center, the academies, and in the field. www.oig.dhs.gov 9 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security However, further improvements are needed. CBP does not maintain a repository or any formal process for the field to share high-quality, locally developed training.

Nor has the agency effectively communicated to, or followed up with, the field regarding implementation of its overall integrity strategy. Correcting these deficiencies will enhance CBP’s ability to more effectively oversee and manage its ethics and integrity training program and foster a workforce culture of integrity. Recommendations Recommendation 1: We recommend that the Assistant Commissioner for Training create a repository for the field offices to maintain and share unique field-developed training. Recommendation 2: We recommend that the Commissioner highlight the importance of CBP’s integrity strategy throughout the agency and follow up with field staff to ensure it is effectively communicated.

Management Comments and OIG Analysis Below is a summary of CBP’s comments to the draft report and the OIG analysis. Verbatim comments are in appendix B. Recommendation 1 CBP Comments: Concur. CBP's Office of Training and Development (OTD) Distance Learning Center (DLC) and the National Training Plan (NTP) will host a meeting with the CBP operational components of U.S. Border Patrol, Office of Field Operations, and Air and Marine Operations to coordinate and develop a repository to share locally developed training and best practices. Estimated Completion Date (ECD): September 30, 2017 OIG Analysis: CBP’s planned action is responsive to the intent of the recommendation and recommendation #1 is considered resolved and open.

OIG will close the recommendation when CBP provides evidence of fully implementing the creation of a repository to share unique field-developed training. Recommendation 2 CBP Comments: Concur. On December 22, 2016, CBP’s Commissioner issued a message to all CBP employees redistributing CBP's Integrity and Personal Accountability Strategy, and reemphasized the importance of the strategy, www.oig.dhs.gov 10 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security which outlines the responsibility of every employee — including CBP's leadership — to foster a culture of integrity. CBP is confident that it has met the intent of the recommendation and respectfully requests closure.

OIG Analysis: CBP’s action is responsive to the intent of the recommendation and recommendation #2 is resolved and closed. www.oig.dhs.gov 11 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security Appendix A Objective, Scope, and Methodology The Department of Homeland Security Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107−296) by amendment to the Inspector General Act of 1978. House Report 114-215, which accompanied Consolidated Appropriations Act, 2016, Public Law (P.L.) No. 114–113, directed OIG to “assess and report on CBP’s ethics and integrity training for its agents and officers, as well as CBP mechanisms for operational oversight related to ethics and integrity” [H. R. Rep. No. 114-215, at 22 (2015)]. Because of the broad and undefined nature of this mandate, we corresponded with congressional staff who acknowledged the steps CBP had taken to address concerns regarding allegations of misconduct associated with CBP’s rapid hiring of border patrol agents.

Those improvements included improving training and assuming criminal investigative authority from U.S. Immigration and Customs Enforcement for allegations of CBP misconduct. However, concerns remained, and OIG was tasked with determining how training has improved and how CBP is tracking whether training is effective, as well as providing rigorous oversight with regard to allegations of personnel conduct. As a result, we evaluated ongoing OIG projects and found another audit team was looking at whether conduct and discipline is equitable and consistent department-wide. Although the DHS Conduct and Discipline audit will not directly answer whether CBP is providing rigorous oversight with regard to allegations of personnel misconduct, it will include a high-level overview of CBP’s process for reporting and addressing misconduct, as well as some limited testing of CBP’s disciplinary action cases.

We then revised our objective to focus on whether CBP made training improvements, communicated in a comprehensive manner its integrity strategy, and tracked and measured training completion and effectiveness. To accomplish our objective we reviewed CBP’s oversight of ethics and integrity training from fiscal years 2012 through 2016. We researched and reviewed Federal, DHS, Office of Personnel Management (OPM), and CBP training requirements, policies, and procedures pertaining to ethics and integrity training. We researched industry standards pertaining to training evaluation methods.

We also reviewed prior DHS OIG, General Accountability Office (GAO), Homeland Security Studies and Analysis Institute, Police Executive Research Forum, and the Integrity Advisory Panel reports for previously identified findings related to our audit. We determined whether HQ www.oig.dhs.gov 12 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security communicated the 2014 CBP Integrity and Personal Accountability Strategy to the field and its impact on ethics and integrity related training. We assessed CBP’s control structure, policies, procedures, and practices applicable to its ethics and integrity training program. Our limited assessment would not necessarily disclose all material weaknesses in this control structure.

However, our assessment disclosed weaknesses in tracking of training, and communication of its integrity strategy and field-developed training. These weaknesses are discussed in the body of this report. We conducted interviews with officials from CBP offices and other offices to determine roles and responsibilities in the development, implementation, and oversight of the integrity strategy and CBP’s ethics and integrity training program: CBP Advanced Training Center in Harper’s Ferry, West Virginia x OTD Instructional Design Center x OTD The Law Enforcement Officer/Agent Safety and Compliance Directorate, formerly the Use of Force Center For Excellence x OTD Distance Learning Center CBP Headquarters, Washington, DC x CBP Air and Marines Operations (AMO) Training and Safety Standards Group x CBP Office of Border Patrol HQ Mission Readiness Operations Division (MROD) Training & Traumatic Incident Management Division x CBP Office of Chief Counsel (OCC) x CBP Office of Field Operations Analytical Management Systems Control Office (AMSCO) x CBP Office of Field Operations (OFO) HQ Training Branch x CBP Office of Field Operations Integrity and Professionalism Standards Division (ISPD) x CBP Office of Human Resources Management (HRM) x CBP Office of Information Technology (OIT) x CBP Office of Policy and Planning (OPP) x CBP Office of Professional Responsibility (OPR – formerly Internal Affairs) x CBP Office of Public Affairs (OPA) x CBP Office of Training and Development (OTD) x CBP OTD Budget Office x CBP Self-Inspection Program (SIP) Office x DHS Office Chief Human Capital Officer (OCHO) PALMS Administrator x DHS-OIG Investigations x DHS-OIG Workforce and Training Development Office x GAO www.oig.dhs.gov 13 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security To understand CBP’s field training operations and related ethics and integrity training, we conducted site visits and interviewed officials at Laredo, Texas (AMO, USBP, OFO, OPR, and OCC); Tucson, Arizona (AMO, USBP, OFO, and OCC); USBP basic academy in Artesia, New Mexico; and OFO and AMO basic academies in Brunswick, Georgia. We reviewed and analyzed HQ directives to field locations and academies, locally developed policies, and training SOP manuals.

We examined course syllabi and curricula; in-service and other training agendas; musters; and webcasts. Auditors attended academy trainings; and observed several scenario-based training with ethics and integrity content. The audit team used a data collection instrument as part of our review process to substantiate and record field observations and testing performed. We evaluated the integrity of records management by reviewing electronic and manual files used for tracking training completion, such as attendance sign-in sheets; certificates of completion; PALMS and Training, Records, and Enrollment Network (TRAEN) system-generated reports; spreadsheets, and post-academy training files.

We also received demonstrations of primary systems of record including PALMS; TRAEN; and ACADIS. We performed limited data reliability testing of PALMS data because of known data integrity issues and concerns over the system’s reporting functionality. We observed CBP officials at the OTD in May 2016 unsuccessfully attempt to generate reports of completed and overdue training, noting PALMS error messages and inaccurate totals. In October of that same year, the PALMS vendor developed a temporary solution for CBP’s Advanced Training Center Distance Learning Center, enabling them to run accurate reports for training completion.

In October we did a limited validation of the system’s reporting functionality. From site visits we judgmentally selected 42 training completion certificates for the annual CBP Integrity Awareness training, see table 2 for details. The exception was the Brunswick, GA AMO Academy, at which there were only two agents assigned to that location. USBP Table 2 – Sample Selection of Training Completion Certificates for Integrity Awareness Training Location Laredo, TX 5 Tucson, AZ 5 Academies 5 Total Source: OIG analysis of CBP data Total 15 15 12 42 AMO 5 5 2 OFO 5 5 5 www.oig.dhs.gov 14 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security We compared our certificates to OTD’s October PALMS-generated report to determine whether the system accurately reflected training completion for our sample.

We did not identify any significant discrepancies and believe, for the purpose of answering the congressional request, the PALMS data was sufficiently reliable to meet our audit objectives. For assessing CBP’s evaluation of training effectiveness, we reviewed DHS, OTD, OPM, and industry best practices. We evaluated End-of-Course Evaluations; Kirkpatrick Model Levels 1–3 surveys; after action reports; evaluation plans; and other documentary evidence showing that CBP measured the effectiveness of ethics and integrity training. We conducted this performance audit between April and October 2016 pursuant to the Inspector General Act of 1978, as amended, and according to the generally accepted government auditing standards.

Those standards require that we plan and perform the audit to obtain sufficient and appropriate evidence to provide a reasonable assurance for our findings and conclusions based upon our audit objectives. We believe that the audit evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. www.oig.dhs.gov 15 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security Appendix B CBP’s Comments to the Draft Report www.oig.dhs.gov 16 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security www.oig.dhs.gov 17 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security www.oig.dhs.gov 18 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security Appendix C Ethics and Integrity Training Improvements since FY 2012 Since 2012, CBP has implemented several new training courses and programs that include ethics and integrity elements. The following tables list notable improvements. Customs and Border Protection-Wide Training/Program Name Annual Integrity Awareness Training New Employee Orientation Integrity Committees Integrity Advisory Panel Below 100 Training Ethics, Integrity, and Professionalism Musters Michael V. Gallagher (Anti-DUI) Instructional Systems Specialists (OTD) Revised Senior Leadership Training (OTD) Integrity and Professionalism Campaign Supervisory Technical Training Course (USB) Supervisory Professionalism Engagement Dialogue Trainings (OFO) Integrity Officer Program (OFO) Source: OIG analysis of CBP data Implementation Timeframe FYs 2012 2012 2014 2015 2016 2013 to 2016 2015 2012 2016 2014 2012 2015 2014 www.oig.dhs.gov 19 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security Academy Training/Program Name Basic Training Program Redesign (USBP) Post-Academy Training Redesign (USBP) Enhanced Scenario Based Trainings (USBP and OFO) Office of Chief Counsel-Nationality and Immigration Law and Applied Authorities (116 hours) Redesigned Anti-Corruption and Integrity Reinforcement (AMO and OFO) Law Enforcement Professionalism (OFO) Source: OIG analysis of CBP data Field Training/Program Name One-Hour Ethics Training (Laredo) PRIDE Initiative (Laredo) Ask the Director (Tucson) Flying Armed (Tucson) Leadership for Supervisors (Tucson) Leadership for Non-Supervisors (Tucson) Redesigned Report Writing and Courtroom Testimony (Tucson) Source: OIG analysis of CBP data Implementation Timeframe FYs 2016 2016 2016 2016 2013 2016 Implementation Timeframe FYs 2016 2015 2016 2014 2016 2012 2013 www.oig.dhs.gov 20 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security Appendix D The Kirkpatrick Model Levels of Measurement, Benefits, and Limitations OTD’s Standard Operating Procedure (SOP), dated November 2015, details the four levels of measurement of the Kirkpatrick Model.

Level 1 feedback is intended to capture the trainee’s view of the training and student confidence and attitude about skills obtained, instructor performance, and training environment. The data is intended for the course managers and instructor personnel and should be collected for all students at the end of short courses and throughout longer trainings. CBP Form 800 is the standard level 1 form used to collect trainee reaction data. Level 2 evaluations are arguably the most important level of evaluation for OTD because it ensures students have either demonstrated or failed to demonstrate critical work tasks.

Trends from this data can be used to validate the instructional strategy and reliability of the training delivery. Level 3 feedback evaluates the transfer and application of skills learned in the training to on-the-job performance. This is achieved by determining at what level graduates are performing the critical tasks from the course in the workplace. The graduate’s supervisor or someone with knowledge of the individual’s performance completes the surveys, interviews, or field observations to make this determination.

Level 4 determines training’s impact and benefit to the organization and will only be conducted on a case-by-case basis developed by the Director of the Instructional Design Center (IDC). According to “A Critical Analysis of Evaluation Practice: The Kirkpatrick Model and the Principle of Beneficence,” in the Journal of Evaluation and Program Planning 27, 2004, there are benefits and limitations of the model. Benefits x The Kirkpatrick Model addresses the need of training professionals to understand training evaluation in a systematic way. x The four-level model provides a straightforward system or language for talking about training outcomes and the kinds of information that can be provided to assess the extent to which training programs have achieved certain objectives. x The four-level model provides a method of determining how training affects the organization’s ability to accomplish its missions. www.oig.dhs.gov 21 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security x The Kirkpatrick Model simplifies the complex process of training evaluation. Limitations x The four-level model does not take into consideration individual or contextual influences that affect how employees learn and organizations design and delivery training. x The learning culture of an organization, work unit goals, climate for learning transfer, and adequacy of employee work tools and equipment all affect how successful employees will be in the learning process; the Kirkpatrick model relies on an assumption that without learning, employees will not change behaviors, but research has failed to confirm this link. x Additionally, the Kirkpatrick Model assumes that the importance of information gathered increases moving from level 1 to level 4.

However, evaluators may have certain preconceived notions that affect their ability to conduct accurate and reliable evaluations. For example, if employees, trainers, or evaluators are potentially faced with negative actions, based on the evaluation data, an evaluator may skew the data to produce favorable results. www.oig.dhs.gov 22 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security Appendix E Office of Audits Major Contributors to This Report Brooke Bebow, Director Cecilia Carroll, Audit Manager Tessa Clement, Program Analyst Eddie Jones, Auditor Falon Newman-Duckworth, Auditor Michael Staver, Program Analyst Kevin Dolloson, Communications Analyst Ellen Gallagher, Communications Analyst Corneliu Buzesan, Independent Referencer www.oig.dhs.gov 23 OIG-17-60 OFFICE OF INSPECTOR GENERAL Department of Homeland Security Appendix F Report Distribution Department of Homeland Security Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director, GAO/OIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Commissioner CBP Assistant Commissioner for Training CBP Audit Liaison Office Office of Management and Budget Chief, Homeland Security Branch DHS OIG Budget Examiner Congress Congressional Oversight and Appropriations Committees www.oig.dhs.gov 24 OIG-17-60 ADDITIONAL INFORMATION AND COPIES To view this and any of our other reports, please visit our website at: www.oig.dhs.gov. For further information or questions, please contact Office of Inspector General Public Affairs at: [email protected]. Follow us on Twitter at: @dhsoig.

OIG HOTLINE

To report fraud, waste, or abuse, visit our website at www.oig.dhs.gov and click on the red "Hotline" tab. If you cannot access our website, call our hotline at (800) 323-8603, fax our hotline at (202) 254-4297, or write to us at: Department of Homeland Security Office of Inspector General, Mail Stop 0305 Attention: Hotline 245 Murray Drive, SW Washington, DC 20528-0305