DHS OIG, OIG-15-91, CBP is on Track to Meet ACE Milestones, but It Needs to Enhance Internal Controls (2015)

CBP is on Track to Meet ACE Milestones, but It Needs to Enhance Internal Controls May 11, 2015 OIG-15-91 DHS OIG HIGHLIGHTS CBP is on Track to Meet ACE Milestones, but It Needs to Enhance Internal Controls May 11, 2015 Why We Did This The Automated Commercial Environment (ACE) is the commercial trade system designed to automate border processing to enhance border security and foster our Nation’s economic security. ACE is part of a multi-year U.S. Customs and Border Protection (CBP) modernization effort that is being deployed in phases and must be completed by December 2016. We conducted the audit to determine whether CBP is on track to meet its milestones for the implementation of ACE. What We Recommend We made one recommendation to continuously assess, evaluate, and update internal controls, to include a risk assessment that identifies potential data reliability gaps; and develop and implement specific, measurable, achievable, relevant, and time-sensitive performance measures.

This recommendation, when implemented, should improve the efficiency and effectiveness of the program. For Further Information: Contact our Office of Public Affairs at (202) 254-4100, or email us at [email protected] What We Found CBP spent approximately $3.2 billion on the development of the ACE program from fiscal year 2001 through July 2013, when it was restructured to a rapid deployment strategy called Agile. Currently, in large part due to the implementation of Agile, CBP is on track to meet its milestones for the deployment of ACE. However, CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program.

Specifically, CBP has not conducted risk assessments to identify potential gaps in data reliability, and has not fully developed and implemented performance measures for the program. Similar internal control issues were identified in a 2014 KPMG financial statement audit. If these weaknesses continue and internal controls do not keep pace with Agile and the rapid implementations of the ACE program, deployment schedules could be adversely impacted, resulting in missed future deadlines and compromised effectiveness of ACE. CBP Response In its response to our draft report, CBP reported that it appreciated the Office of Inspector General’s (OIG) recognition that the ACE program is on track to meet its implementation milestones and OIG’s positive conclusion that the Agile development methodology improves the effectiveness and efficiency of the ACE development process.

However, CBP did not concur with our report recommendation. www.oig.dhs.gov OIG-15-91 . vttllf, r ~ ~~ `yf~i' T OFFICE OF INSPECTOR GENERAL Department of Homeland Security Washington, DC 20528 / www.oig.dhs.gov MEMORANDUM FOR: Brenda Smith MAY 11 2015 FROM: SUBJECT: Assistant Commissioner Office of International Trade U.S. Customs and Border Protection Mark Bell ~ ~ ~~~-'~ Assistant Inspector General for Audits CBP is on Track to Meet ACE Milestones, but It Needs to Enhance Internal Controls For your action is our final report, CBP is on Track to Meet ACE Milestones, but It Needs to Enhance Internal Controls. We incorporated the formal comments provided by your office. The report contains one recommendation aimed at improving the Automated Commercial Environment program. Your office did not concur with the recommendation.

Based on information provided in your response to the draft report, we consider the recommendation open and unresolved. As prescribed by the Department of Homeland Security Directive 077-01, Follow-Up and Resolutions for the Office of Inspector General Report Recommendations, within 90 days of the date of this memorandum, please provide our office with a written response that includes your (1) agreement or disagreement, (2) corrective action plan, and (3) target completion date for the recommendation. Also, please include responsible parties and any other supporting documentation necessary to inform us about the current status of the recommendation. Until your response is received and evaluated, the recommendation will be considered open and unresolved.

Consistent with our responsibility under the Inspector General Act, we will provide copies of our report to congressional committees with oversight and appropriation responsibility over the Department of Homeland Security. We will post the report on our website for public dissemination. Please call me with any questions, or your staff may contact Paul Wood, Acting Deputy Assistant Inspector General for Audits, at (202) 254-4100. You can also send your response to OIGAuditsFollowup(c~,oi~ dhs•gov.

Attachment OFFICE OF INSPECTOR GENERAL Department of Homeland Security Background In support of the 1993 Customs Modernization Act, U.S. Customs and Border Protection (CBP) has been modernizing the business processes essential to securing U.S. borders, speeding the flow of legitimate shipments, and targeting illicit goods that require scrutiny. The International Trade Data System (ITDS) is an electronic information exchange database through which businesses transmit data required by participating agencies for the importation or exportation of cargo. Executive Order 13659 mandates CBP to provide participating agencies the capabilities, agreements, and other requirements necessary to use the ITDS and supporting systems, such as the Automated Commercial Environment (ACE), by December 31, 2016. ACE is a key technology driver of these initiatives and will be the primary means of receiving users’ standard data and other relevant documentation required for the release of imported cargo and clearance of cargo for export.

Once fully implemented, ACE will become the central trade data collection system for all Federal agencies, and the single point of access for this data, which includes data collection, processing, dissemination, and storage. The development of the ACE program began in 2001. In 2006, Program Assessment and Design Review (PADR) procedures were developed in order to improve oversight of the ACE program. These procedures required CBP to work with the Department of Homeland Security’s Chief Information Officer (CIO) to certify each release was ready to proceed beyond critical design review and production readiness review.

In 2009, CBP presented a release to the PADR team for evaluation and approval. This review was expanded to encompass aspects of the overall ACE program. The PADR team identified deficiencies such as significant delays, cost overruns, and unclear system requirements. According to CBP, the primary contract did not contain clear deliverables and the method of development did not allow for unknown variables common to software development.

The PADR team recommended that ACE not continue beyond production readiness review. The Department of Homeland Security’s (DHS) CIO concurred with the recommendation that CBP halt future system development to re-evaluate the project, which placed the program into breach status. In September 2012, the Acquisition Review Board conducted a program review of ACE and determined CBP was allowed to continue operational capabilities and approved a transition plan in December 2012. CBP started a pilot program in 2013 which restructured its process to Agile, a rapid deployment strategy that had a shorter delivery cycle and more oversight and accountability.

In June 2013, the program was removed from breach status and ACE www.oig.dhs.gov 2 OIG-15-91 OFFICE OF INSPECTOR GENERAL Department of Homeland Security development restarted using Agile. All costs prior to the completion of the pilot program were considered sunk1 costs. CBP spent approximately $3.2 billion on the development of the ACE program from fiscal year 2001 through July 2013, when it was restructured. CBP’s fiscal year (FY) 2014 appropriation for the ACE program was approximately $141 million.

Results of Audit CBP is on track to meet its milestones for the implementation of the ACE program. Currently, CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time, mainly due to the recently implemented rapid deployment strategy—Agile. However, CBP has not ensured the internal control environment has kept pace with the rapid deployment of the ACE program. Specifically, CBP has not conducted risk assessments to identify potential gaps in data reliability, and has not fully developed and implemented performance measures for the program.

In 2014, KPMG conducted a financial statement audit and identified similar internal control issues. If these weaknesses continue and internal controls do not keep pace with the rapid implementations that Agile delivers, development and deployment schedules could be adversely impacted, resulting in missed future deadlines and compromised effectiveness of ACE. CBP’s ACE Program on Track CBP is on track to meet its milestones for the implementation of the ACE program and it should meet the December 31, 2016, Presidential mandate. In 2013, CBP restructured its development process for the ACE program and changed to Agile.

This rapid deployment strategy supports the practice of shorter software delivery, more oversight and accountability, and allows more flexibility to accommodate changing requirements and shifting priorities. Specifically, the strategy calls for delivery of software in small, short increments and emphasizes collaborative teams. CBP uses a 13-week cycle, or increment, that is broken down into 1 week of planning and six, 2-week-long development periods called “sprints.” CBP determines what software must be developed within an increment and each of the 12 teams develops a portion of the software during the sprints, which 1 CBP defines sunk costs as all funds obligated prior to July 2013, which includes $46.4 million in Agile development costs. This separates the cost of the restructured ACE development under Agile from the previous development that was placed in breach status.

When the future costs (through FY 2026) and sunk costs are combined, the estimated life cycle cost is approximately $4.23 billion. www.oig.dhs.gov 3 OIG-15-91 OFFICE OF INSPECTOR GENERAL Department of Homeland Security combine into a larger body of work at the end of each increment. This process provided CBP with flexibility and oversight of development. For example, CBP required development teams to update management daily and demonstrate work completed biweekly. The Agile strategy improves the effectiveness and efficiency of the development process and alleviates cost and schedule variances.

According to CBP’s Increment and Deployment Schedule, there are seven deployments (A through G) of ACE capabilities. Deployments A, B, C, and D were implemented on schedule; with Deployment D being implemented in January 2015. CBP’s goal is to have all seven deployments completed by July 2016, which is 5 months before the mandated deadline. CBP has completed four of its seven scheduled deployments and remains on track to complete ACE on time (see appendix B for an ACE timeline).

Risk Assessment Needed CBP has not conducted a risk assessment to determine its vulnerabilities and identify what controls are needed to address them. The internal control environment provides the structure to help an entity achieve its objectives. According to the Committee of Sponsoring Organizations’ Internal Control Integrated Framework, internal controls keep an organization on course toward meeting goals and achieving its mission. In addition, GAO’s Internal Control Management and Evaluation Tool states Federal managers need to continually assess and evaluate their internal control structure to assure that it is: 1. well designed and operated; 2. appropriately updated to meet changing conditions; and 3. providing reasonable assurance that the objectives of the agency are being achieved.

As a component of the internal control environment, a risk assessment provides the basis for developing appropriate control activities. Even though CBP did complete a Risk Management Plan, it did not provide proof it was used to formally assess risk. In addition, the plan’s focus identified risks in the development process, and may not identify other risks to the system. During the Department of Homeland Security FY 2014 Integrated Audit, KPMG tested key information technology controls for ACE.

KPMG issued a Notice of Findings and Recommendations stating CBP had not adequately maintained separation of duties or other controls within the ACE production and development environments. Additionally, during our audit, we found that CBP granted software developers access to the live database in order to update the system. This may create a risk that developers have access to ACE without www.oig.dhs.gov 4 OIG-15-91 OFFICE OF INSPECTOR GENERAL Department of Homeland Security proper internal controls. Without a risk assessment, CBP may be unaware of potential risks or other data reliability concerns in ACE.

Performance Measure Updates Needed ACE performance measures under the rapid deployment strategy have not been fully developed and implemented. CBP created measures for the original development strategy; however, it has not updated them to reflect the desired outcomes of the rapid deployment strategy. For example, CBP’s FY 2012 and 2014 Performance Measure Scorecards each contained 10 common measures with an annual target for CBP. Of the 10 targets, 8 remained unchanged from the FY 2012 scorecard compared to the FY 2014 scorecard.

CBP began the rapid deployment strategy in 2013; therefore, the performance measures were created for the original deployment strategy. Specifically, on the 2014 scorecard, a CBP official explained that one measure calculated how much faster CBP processed truck cargo against the time it took in 2006. This calculation seems outdated because it compared the 2014 scorecard to 2006 rather than measuring it against 2013 and as such, may not be a relevant measure of the ACE program. CBP’s performance measures do not explain how the information presented is showing progress toward its goal under the rapid deployment strategy.

The measures are not well defined or explained, do not have completion criteria, timeframes, or the level of specificity needed to show how they prove an accomplishment. Performance measures are an important tool to ensure a project is meeting its stated objectives, and should be specific, measurable, achievable, relevant, and time-sensitive (SMART). However, CBP has recently drafted new performance measures, which were not ready for review. Recommendations Recommendation: We recommend that the Assistant Commissioner, Office of International Trade, U.S. Customs and Border Protection, continuously assess, evaluate, and update internal controls during each 13- week development increment.

Specifically: a. Conduct a risk assessment to identify potential data reliability gaps; and b. Develop and implement specific, measurable, achievable, relevant, and time-sensitive (SMART) performance measures. www.oig.dhs.gov 5 OIG-15-91 OFFICE OF INSPECTOR GENERAL Department of Homeland Security Management Comments and OIG Analysis In its response to our draft report, CBP did not concur with our report recommendation. A summary of CBP’s response and our analysis follows. We have included a copy of the management comments in their entirety in appendix A. CBP also provided technical comments to our draft report. Management Comments: CBP did not concur with our recommendation and stated that it does not agree that the ACE program has not conducted risk assessments to identify potential gaps in data reliability or fully developed and implemented performance measures for the program.

According to CBP, the ACE program is consistently performing risk assessments to identify potential data reliability gaps and has fully developed and implemented SMART performance measures as appropriate for assessing deployed features. In addition, risk assessments are being completed before each deployment as part of the CBP’s Production Readiness Review (PRR) process, and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process. Finally, according to CBP, program officials have provided OIG with many documents to demonstrate that well- developed SMART performance measures are being used, along with the implementation of other internal controls. OIG Analysis: In its response to our draft report, CBP stated, “Risk assessments are being completed before each deployment as part of the CBP’s Production Readiness Review (PRR) process, and the ACE program is required to successfully meet every constraint identified by the risk assessments in order to be approved for deployment during the PRR process.” However, as stated, this review is conducted prior to deployment.

CBP further stated in its technical comments that, “After deployment we complete risk assessments of data reliability through validations and edits.” OIG contends that doing “validation and edits” is merely testing the data and not the same as completing a risk assessment which provides the basis for developing appropriate risk responses. Per the Government Accountability Office’s (GAO) Standards for Internal Control in the Federal Government, management should identify, analyze, and respond to risks related to achieving the defined objective. This would include assessing inherent and residual risk, internal and external factors, potential of fraud, magnitude of impact, likelihood of occurrence, and nature of risk, among other factors. www.oig.dhs.gov 6 OIG-15-91 OFFICE OF INSPECTOR GENERAL Department of Homeland Security In fact, according to CBP’s own technical comments, “The ACE Program did not identify data reliability as a risk and [as] such, data reliability is not part of the ACE Program risk report.” As detailed in the Background section of this report, “ACE will become the central trade data collection system for all Federal agencies, and the single point of access for this data, which includes data collection, processing, dissemination, and storage.” As the ACE program will be used to collect duties, taxes, and fees, and to make other key decisions, the OIG contends that data reliability is the cornerstone of the ACE program. Furthermore, given the multitude of data reliability issues within DHS found during prior OIG and GAO audits, we maintain that CBP should conduct a comprehensive risk assessment to identify potential data reliability gaps.

In addition, the OIG still recommends that CBP develop and implement SMART performance measures. In its response to our draft report, CBP stated, “CBP program Officials have provided OIG many documents to demonstrate that well-developed SMART performance measures are being utilized….” From a review of CBP’s provided documents, we determined that those documents are merely data collection spreadsheets and not performance measures. The measures are not specific or time sensitive. In other words, it is unclear what the goal is and when it is to be achieved.

In developing an effective risk assessment, GAO states that management should, “Define objectives in specific terms so they are understood at all levels of the entity. This involves clearly defining what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement.” CBP’s Performance Measurement Indictor spreadsheet does not have SMART performance measures. Therefore, the OIG considers this recommendation unresolved and open. We will resolve the recommendation once CBP agrees to conduct a data reliability risk assessment and develops improved SMART performance measures.

We will close the recommendation when we receive appropriate completion documentation. Objective, Scope, and Methodology The DHS Office of Inspector General was established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General Act of 1978. This is one of a series of audit, inspection, and special reports prepared as part of our oversight responsibilities to promote economy, efficiency, and effectiveness within the Department. We conducted an audit of the ACE program to determine whether CBP was on track to meet its milestones for the program implementation.

To achieve our audit objective, we identified and reviewed applicable Federal laws, regulations, www.oig.dhs.gov 7 OIG-15-91 OFFICE OF INSPECTOR GENERAL Department of Homeland Security and CBP policies and procedures regarding the ACE program. The audit covered the development of the ACE program from January 2013 through September 2014. We interviewed CBP personnel responsible for the development, management, and administration of ACE, including key stakeholders from the ACE Business Office, Office of Information & Technology, Office of International Trade, Office of Technology Innovation & Acquisition, Office of Field Operations, Outcomes and Analysis Branch, and Trade Management and Information Division. We also interviewed subject matter experts and contractors responsible for ACE development.

Additionally, we discussed ACE activities by the end users with officials at two ports and with two partnering government agencies, the Environmental Protection Agency, and Food Safety and Inspection Service. We performed limited survey work to ensure the accuracy and completeness of information used in this audit. Additionally, we determined whether CBP had established internal controls for ACE that provided reasonable assurance ACE will achieve its objectives. We conducted this performance audit between June and October 2014, pursuant to the Inspector General Act of 1978, as amended, and according to generally accepted government auditing standards.

Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based upon our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions. Office of Audits major contributors to this report are: Brooke Bebow, Director; Patrick Tobo, Audit Manager; Priscilla Cast, Co-Auditor-in-Charge; Anthony Colache, Co-Auditor-in-Charge; Tia Jackson, Program Analyst; Frank Lucas, Auditor; Sandra Ward-Greer, Auditor; Kevin Dolloson, Communications Analyst; and Marissa Weinshel, Independent Referencer. www.oig.dhs.gov 8 OIG-15-91 OFFICE OF INSPECTOR GENERAL Department of Homeland Security Appendix A Management Comments to the Draft Report www.oig.dhs.gov 9 OIG-15-91 OFFICE OF INSPECTOR GENERAL Department of Homeland Security www.oig.dhs.gov 10 OIG-15-91 6 1 0 2 r e b m e c e D 6 1 0 2 y r a u n a J d e t a d n a M E C A r o f e n i l d a e d n o i t e l p m o c F t n e m y o l p e D e b o t d e t c e p x e d e t e l p m o c 5 1 0 2 y r a u n a J D t n e m y o l p e D d e t c e p x e d e t e l p m o c e b o t 3 1 0 2 e n u J s n i a t b o P B C r o f l a v o r p p a e l i g A 1 1 0 2 r e b o t c O 0 1 0 2 r e b o t c O n o i t i s i u q c A S H D d r a o B w e i v e R o t n o i s i c e d d e e c o r p s e t a i t i n i P B C E C A f o n o i s i v e r i g n n n a l p n o i t a t n e m u c o d 1 0 0 2 l i r p A E C A l a n i g i r O t n e m p o l e v e D s n i g e b L A R E N E G R O T C E P S N I F O E C I F F O y t i r u c e S d n a l e m o H f o t n e m t r a p e D e n i l e m i T m a r g o r P E C A B x i d n e p p A 6 1 0 2 y l u J 5 1 0 2 y l u J 4 1 0 2 y a M 3 1 0 2 y r a u n a J 0 1 0 2 r e b m e c e D G t n e m y o l p e D e b o t d e t c e p x e d e t e l p m o c E t n e m y o l p e D e b o t d e t c e p x e d e t e l p m o c s t n e m y o l p e D d a h C d n a . B A , y b d e s a e l e r n e e b e t a d s i h t s t r a t s P B C r o f t o l i p e l i g A E C A s e v o r p p a P B C n o i t i s i u q c a d e s i v e r y g e t a r t s .

I G O S H D y b 9 0 0 2 e n u J m a r g o r P t n e m s s e s s A w e i v e R n g i s e D m a e T s d n e m m o c e r t o n E C A e u n i t n o c d e t a e r C : e c r u o S 1 1 1 9 - 5 1 - G O I v o g . s h d . g i o . w w w OFFICE OF INSPECTOR GENERAL Department of Homeland Security Appendix C Report Distribution Department of Homeland Security Secretary Deputy Secretary Chief of Staff General Counsel Executive Secretary Director, GAO/OIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs CBP Audit Liaison Office of Management and Budget Chief, Homeland Security Branch DHS OIG Budget Examiner Congress Congressional Oversight and Appropriations Committees www.oig.dhs.gov OIG-15-91 12 ADDITIONAL INFORMATION AND COPIES To view this and any of our other reports, please visit our website at: www.oig.dhs.gov. For further information or questions, please contact Office of Inspector General Public Affairs at: [email protected]. Follow us on Twitter at: @dhsoig.

OIG HOTLINE

To report fraud, waste, or abuse, visit our website at www.oig.dhs.gov and click on the red "Hotline" tab. If you cannot access our website, call our hotline at (800) 323-8603, fax our hotline at (202) 254-4297, or write to us at: Department of Homeland Security Office of Inspector General, Mail Stop 0305 Attention: Hotline 245 Murray Drive, SW Washington, DC 20528-0305