DHS OIG, OIG-11-42, Planning and Funding Issues Hindered CBP's Implementation of the System Availability Project (2011)

Department of Homeland Security Office of Inspector General Planning and Funding Issues Hindered CBP's Implementation of the System Availability Project (Redacted) OIG-11-42 February 2011 Office ofInspector General U.S. Department of Homeland Security Washington, DC 20528 Homeland Security FEB 04 ZOl1 Preface The Department of Romeland Security (DRS) Office ofInspector General (OIG) was established by the Homeland Security Act of2002 (Public Law 107-296) by amendment to the Inspector General Act of1978. This is one of a series of audit, inspection, and special reports prepared as part of our oversight responsibilities to promote economy, efficiency, and effectiveness within the department. This report addresses the strengths and weaknesses of the U.S. Customs and Border Protection's management of the System Availability project. It is based on interviews with employees and officials of relevant agencies and institutions, direct observations, and a review of applicable documents.

The recommendations herein have been developed to the best knowledge available to our office, and have been discussed in draft with those responsible for implementation. We trust this report will result in more effective, efficient, and economical operations. We express our appreciation to all of those who contributed to the preparation of this report. ~8 Assistant Inspector General Office ofInformation Technology Audits Table of Contents/Abbreviations Executive Summary .............................................................................................................1 Background ..........................................................................................................................2 Results of Audit ...................................................................................................................4 Actions Taken to Implement the System Availability Project.............................................4 System Availability Project Was Not Properly Planned and Implemented.........................5 Not All At-Risk Sites Were Included in the Project ......................................................5 The System Availability Project Did Not Have Adequate Funding ..............................7 Key Planning Documents Were Not Prepared...............................................................7 Recommendation .................................................................................................................9 Management Comments and OIG Analysis ........................................................................9 Appendixes Appendix A: Purpose, Scope, and Methodology.......................................................10 Appendix B: Management Comments to the Draft Report .......................................11 Appendix C: Major Contributors to This Report ......................................................12 Appendix D: Report Distribution ..............................................................................13 Abbreviations ARB AP CBA CBP CIO DHS FY IT IRB LAN LAX LCC NAD OAM OBP OFO OIG OIT Acquisition Review Board Acquisition Plan Cost-Benefit Analysis Customs and Border Protection Chief Information Officer Department of Homeland Security Fiscal Year Information Technology Investment Review Board local area network Los Angeles International Airport Life Cycle Cost Need Analysis Document Office of Air and Marine Office of Border Patrol Office of Field Operations Office of Inspector General Office of Information and Technology Office of Management and Budget Project Management Guidebook System Life Cycle Wide Area Network OMB PMG SLC WAN Figures Figure 1: Figure 2: Figure 3: Figure 4: System Availability Sites Completed ..........................................................5 CBP Sites Status ..........................................................................................6 Project Funding Cost in Millions.................................................................7 CBP Documentation Requirements .............................................................9 I OIG Department of Homeland Security Office of Inspector General Executive Summary On August 11, 2007, a network outage occurred at Los Angeles International Airport that interrupted passenger processing by Customs and Border Protection employees for 10 hours. The outage was caused by the failure of a network card on one of the workstations.

We reviewed the circumstances surrounding this outage, and in May 2008 reported that there is a high risk of similar outages at other Customs and Border Protection sites. Our objective was to determine whether Customs and Border Protection has effectively designed and implemented a plan to reduce the risk of network outages at other field sites. Customs and Border Protection has taken steps to improve network capabilities and reduce network downtime at some field sites. Specifically, it initiated the System Availability project by awarding a task order for information technology services.

In addition, it worked with business sponsors to develop operational requirements to ensure that systems capabilities are maintained. It also deployed survey teams to conduct site surveys at each field site. However, Customs and Border Protection did not properly plan and implement the System Availability project. It did not ensure that adequate funding was available, include all at-risk sites, or develop planning documents needed to justify project requirements and cost.

Customs and Border Protection ran out of funding and ended the project in February 2010. As a result, hundreds of field sites did not receive the needed upgrades and remain vulnerable to network outages. We are recommending that the Customs and Border Protection Chief Information Officer reassess the original objectives of the System Availability project and develop a new program according to Department of Homeland Security and Customs and Border Protection policies and procedures to upgrade the network at all remaining at-risk sites. Planning and Funding Issues Hindered the Implementation of CBP’s System Availability Project Background In August 2007, Customs and Border Protection’s (CBP) network at Los Angeles International Airport (LAX) experienced an outage for more than 10 hours that stranded nearly 17,000 passengers.

We conducted an audit of the LAX outage, and in May 2008 reported that there was a high risk that a similar outage could occur at other CBP sites.1 We recommended that CBP’s Chief Information Officer (CIO) determine whether the corrective actions taken at LAX should be implemented at other sites. In response, CBP began the System Availability project to improve CBP network capabilities and to provide enhanced levels of network availability, performance, and reliability at the sites. The System Availability project objectives are to deploy new infrastructure equipment to the local area networks (LAN) at CBP field sites, which includes The project’s goals are to improve network capabilities and reduce network downtime. For sites receiving infrastructure upgrades, CBP for the LAN and wide area network (WAN) in order to ensure that no exist.

1 OIG-08-58, Lessons Learned from the August 11, 2007, Network Outage at Los Angeles International Airport, May 2008. Planning and Funding Issues Hindered the Implementation of CBP’s System Availability Project CBP planned to implement the System Availability project at field sites managed by its three business sponsors—Office of Field Operation (OFO), Office of Border Patrol (OBP), and Office of Air and Marine (OAM)—that rely on the network infrastructure for their critical missions. These business sponsors have a total of 646 sites, divided as follows: • OFO has 333 official ports of entry in the United States, including 15 pre-clearance offices and 21 field offices. • OBP has 234 facilities nationwide and is the primary federal law enforcement organization responsible for preventing the entry of terrorists and their weapons between official CBP ports of entry. OBP is also responsible for preventing the illicit trafficking of people and contraband between the official ports of entry. • OAM operates out of 79 locations, which include three domain awareness centers, three training centers, and a maintenance facility.

OAM is the world’s largest aviation and maritime law enforcement organization. Its mission is to protect the American people and the Nation’s critical infrastructure through the coordinated use of integrated air and marine forces to detect, interdict, and prevent acts of terrorism and the unlawful movement of people, illegal drugs, and other contraband toward or across U.S. borders. CBP identified the System Availability project as a high priority to quickly begin remediating the conditions at LAX and other priority sites to prevent similar network outages. To improve system availability, CBP issued a sole-source delivery order to an existing CBP modernization contract.

The original modernization program includes tasks to reengineer CBP’s operational processes and develop new technology infrastructure, computer systems, and software applications to support these processes. CBP also uses this existing contract to develop the high-tech trade system, Automated Commercial Environment. Planning and Funding Issues Hindered the Implementation of CBP’s System Availability Project Results of Audit Actions Taken to Implement the System Availability Project CBP initiated the System Availability project to improve network capabilities and eliminate system outages at field sites. Over a 3-year period, it took the following actions to implement the project: • • • • Awarded a sole-source task order for information technology (IT) services to perform project planning, analysis, site surveys, network design, and project implementation and support.

Worked with business sponsors to develop operational requirements to ensure that system capabilities are maintained. Deployed teams to conduct site surveys and develop site-specific requirements to improve system availability. These surveys identified site-specific design and equipment requirements. Developed network designs for each site that provide a detailed, component-level description of the system availability solution to be deployed. • Accomplished and completed infrastructure upgrades for 67 sites in various phases of the System Availability project.

Despite its actions, CBP did not complete implementation of the System Availability project at all at-risk field sites. The business sponsors provided a list of 207 priority sites to be included in the System Availability project. However, the Project Management Office was able to implement all network upgrades, In May 2009, CBP reassessed the System Availability project’s scope and determined that only the (LAN upgrade) was required and that were optional. However, after reducing the the project’s scope, CBP was able to install LAN upgrades at only 67 sites, leaving 140 priority sites vulnerable to potential disruptions.

The System Availability project ended in February 2010, when all funding was depleted. Figure 1 displays the number of sites completed by priority and Figure 1. System Availability Sites Completed Planning and Funding Issues Hindered the Implementation of CBP’s System Availability Project Page 4 43 43 6 6 24 0 0 0 0 0 0 0 Total 67 43 6 6 Total sites where all phases of project were implemented Site surveys and network designs were performed at some of the sites that did not receive a network upgrade. For example, the and Marine site received only a site survey as part of the System Availability project. , Air CBP sites that did not receive the needed upgrades remain vulnerable to service disruptions such as those at LAX.

In addition, the sites may not be able to support critical new applications being deployed across the enterprise. Further, capabilities of existing applications and being transmitted in a timely manner. may degrade mission from System Availability Project Was Not Properly Planned and Implemented CBP did not properly plan the System Availability project, nor did it implement the project fully to achieve its stated objectives. Specifically, CBP did not include all the at-risk sites in the scope of the project, obtain adequate funding for full project implementation, or develop planning documents needed to identify requirements and justify funding. As a result, hundreds of sites did not receive the upgrade and remain vulnerable.

Not All At-Risk Sites Were Included in the Project The System Availability project did not include all sites at risk of disruption and failure. Specifically, 579 sites that rely on the network infrastructure, with numerous not receive the needed upgrades and remain vulnerable to disruptions. did Planning and Funding Issues Hindered the Implementation of CBP’s System Availability Project The network assessment study of the CBP LAN revealed many at the field sites. CBP business sponsors have a total of 646 sites that rely on the network infrastructure for their critical missions. The System Availability project included only 207 sites (32% of the 646 sites).

These 207 sites, known as “priority sites,” comprise The priority site list featured sites that have high volumes of traffic in passengers and goods entering and leaving the United States. For example, Miami International Airport’s daily passenger processing statistics are estimated at 25,000 people for the summer months, and the San Ysidro port of entry estimates 50,000 vehicles and 54,000 pedestrians crossing from Mexico to the United States daily. Of the 207 priority sites, CBP implemented the required upgrades at 67, leaving 140 sites vulnerable. Further, CBP excluded 439 sites from the project as non-priority locations, bringing the number of sites that did not receive upgrades to 579 (see figure 2).

As a result, the excluded sites remain vulnerable to IT interruptions and malfunctions. Figure 2. CBP Site Status CBP Sites Priority sites that received upgrades: 67 Priority sites that did not receive upgrades: 140 Non-priority sites that have not received upgrades: 439 Planning and Funding Issues Hindered the Implementation of CBP’s System Availability Project The System Availability Project Did Not Have Adequate Funding CBP did not acquire adequate funding to complete the System Availability project. The project received $59 million, which was enough to upgrade 67 sites.

The funding shortfall occurred in part because CBP did not follow normal IT budget investment processes for the project. Specifically, CBP did not prepare and submit Office of Management and Budget (OMB) Exhibits 53 and 300 for this project. CBP officials stated that these budget documents were not submitted because this project did not meet applicable dollar thresholds. However, per DHS guidance, the System Availability project is a Level 3 project with a threshold requirement that falls between $50 million and $300 million in life cycle costs (LCCs).

As a result, CBP was required to submit an OMB Exhibit 300 to DHS and obtain approval from the DHS Enterprise Architecture Board. CBP initiated the System Availability project in May 2007, with $5 million in funding from CBP salaries and expenses. Following the LAX outage at the end of fiscal year (FY) 2007, the System Availability project received available year-end funds. FYs 2008 and 2009 year-end funds were also made available.

At the end of the project in February 2010, approximately $59 million had been obligated, thus meeting the requirement to prepare the OMB Exhibit 300. When funding ran out in February 2010, CBP terminated the project, leaving 140 priority sites and 439 nonpriority sites without upgrades. Figure 3 identifies the project funding costs by fiscal year. Figure 3.

Project Funding Costs in Millions Fiscal Year Ended Total Funds Obligated 2007 2008 2009 Total $32 $26 $1 $59 Key Planning Documents Were Not Prepared CBP did not prepare several key planning documents for the System Availability project. Specifically, many of the solution engineering and planning documents required by CBP and DHS Planning and Funding Issues Hindered the Implementation of CBP’s System Availability Project Page 7 policy were not created. Key planning documents were not completed because the situation was urgent and CBP needed to act quickly to implement upgrades at the field sites. For example, CBP did not prepare a Project Charter that formally authorizes the existence of a project and authorizes the project manager to apply the organization’s resources to the project activities.

Per CBP’s System Life Cycle Handbook (SLC), Appendix A.3, and Office of Information Technology (OIT) Project Management Guidebook (PMG), Version 2.0, Section 1.2, a project charter is required for all projects. Without a Project Charter, there is no evidence of communication between all affected groups to facilitate coordination of project activities. Additionally, CBP did not prepare a Need Analysis Document (NAD), which provides the basic information required for the CBP Acquisition Review Board (ARB) to decide on proposed IT projects while they are still in the conceptual stage. A NAD is a brief summary of a project that includes the description, required mission and capabilities, justification, and budget LCC estimate.

Per the OIT PMG, Section 1.3, every new CBP project or enhancement to an existing project, regardless of cost or size, must develop a NAD. CBP also did not prepare a business case to organize the information necessary to justify the project and to make a funding decision in a consistent and structured format, for ARB and Investment Review Board (IRB) reviews. Nor did CBP prepare a cost-benefit analysis (CBA), which includes all resources required to develop and operate the project over its life cycle. As shown in figure 4, per CBP’s PMG, Section 1.4.2, the business case and CBA are required for projects with LCCs greater than $20 million.

Planning and Funding Issues Hindered the Implementation of CBP’s System Availability Project Figure 4. CBP Documentation Requirements Project Acquisition/LCC NAD Required? Business Case and CBA Required? Less than $500,000 acquisition or $2 million LCC $500,000 to $5 million acquisition or $2 million to $20 million LCC Greater than $5 million acquisition or $20 million LCC YES YES YES NO NO YES CBP, IRB CBP Approval Authority Required OIT, ARB CBP, IRB Also, an Acquisition Plan (AP) was not prepared for the System Availability project.

An AP forms the basis for the statement of work. Per CBP’s SLC, an AP is required for each contracted service that exceeds the $100,000 threshold. Without these required documents, CBP was not able to develop a good estimate of the project’s overall cost. As a result, the project was not completely funded and ultimately was terminated before all priority sites could be upgraded.

Recommendation We recommend that the CBP Chief Information Officer, in coordination with the Office of Border Patrol, Air & Marine, and Field Operations, reassess the original objectives of the System Availability project and develop a new program in accordance with DHS and CBP policies and procedures to upgrade the network at all remaining at-risk sites. Management Comments and OIG Analysis The Assistant Commissioner concurs with our recommendation. We consider the recommendation resolved. CBP is looking forward to working collaboratively with the OIG.

Please see Appendix B for DHS’ future corrected actions plan Planning and Funding Issues Hindered the Implementation of CBP’s System Availability Project Appendix A Purpose, Scope, and Methodology The objective of our review was to determine whether CBP has effectively designed and implemented a plan to reduce the risk of network outages. Specifically, we determined whether (1) CBP has designed a corrective action plan to reduce the risk of network outages at border stations and ports of entry; and (2) CBP has implemented effectively a corrective action plan to reduce the risk of outages at border stations and ports of entry. We interviewed program executives, program mangers, contracting officers, field deployment and field support technicians, and IT specialists to understand how the Program Office developed and implemented the System Availability project. We reviewed key program documents such as the program management plan, implementation plans, and cost estimates.

We also reviewed master spend plans, invoices, and procurement requisitions to determine whether and how cost goals and milestones were met. We reviewed contract delivery orders, contract modifications, and attachments to gain an understanding of contract terms and contractor responsibilities. Our fieldwork was conducted at CBP sites at headquarters offices and procurement offices in Washington, DC; We conducted this performance audit between March 2010 and August 2010 pursuant to the Inspector General Act of 1978, as amended, and according to generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives.

We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Planning and Funding Issues Hindered the Implementation of CBP’s System Availability Project Appendix B Management Comments to the Draft Report IlOO "''''''l....., ."""" w,' """"'-."a, OC " , , . \1 f \101"A~ n: 1M FOR R1nl.~R1l1.. SKII'~~R J.~SJ'Lt' J UR <';U<~RAl DEPART).{E'IT OF IJO\fEl A'ID WfTR1TY r'RO\lo SI I!)FCT RO'I"""" 10 L'>< OtT"" Or' Imp""", U<o",ol', l}j.n R""", Cr.1JtkO. "PI"",ini 'TId FoOCin;; 1,","' ll;",I""d cnr', hrpl<",.,""";,m ,,[,h, ~l'''''"' ",,'W'I;,y r,,,j<-.."''' 1!nJLI.: ,'0" fo< r<oyj.jiTI~ u, Wilh, <cv" ofyoUJ d"tt '''l'O" <Ilt;,I," 'Pi,...,;n;; ,n;) FI.<ldiIlii l>ll.':> Hin<!,,,,1 U,~, C"""'''', "00 l3<,,<k' rn",,'",,', 'nr',) h,~~o"''''C''L"" ,,[,he ~}''''''" A,,";I."'hL) r",j«L.' ..-.I the ,,,,,,,><iw>;t)· EO co,""""" ,'" ,t>< ;"''''', ;0 Itti, "PO" Tho "luI ",,"<i''"- ".' ~".-nn"","';'",1i",,,,,,1 ", nlP.

A ,,,m,,,,,')' "f rnp ,,','"' "ro """",t;,·, pI"n t" WJ",,,, the ,,,,,,,",,,,,...Lti,<, ;, f"",,,l<J I>ol"wo K"OIIHu••d.n.. '1: 'I', rt<OlIlllload Ib'11~' t'~I' ('bl.t' Inl.",..JioR <JIll". " .."" ,b••n~;n.' obj«I;.·...1'b. ~)'to"' ,h.H.hil;ty r",jm ,n,t d.,..h'r • n,... pI";""' ;n ."",tL<n« ,,;,h OilS ..d eRP ""I",;,~ ,"d p.-.,",du",. to "1';"'" Ib, " ..",.........11 "",.in",; .1·";,. ,;1",. ('liP II",,,,'"'" OW m"",,, "'," 1o, r."""'''''.... ,;on- ('Rr "ill ,.,,'1,~"""" ,",,",~i., k' J<1,m",., LI" "'''r< """ """'''''' ""l";<=><f>L, n,.",,,,,, ", ..ld"", 01(; ,rind;,~~ elll", l:lJ1lOlCd <onlelotioo ot d'" Injh.1 ""Tc~" i' JUJ>< )0. ,01 1 Wid, "'f!-'<J to II" """;L"iL)' ",'", "'0'1"1"'" CAP ""-, ".""L;I;"I ml'"",.,-";,,,, ""Nn I'" ",;1 ",,,1.1'" """I 'y ..k,,,,,';,,II"""·' wh'" =. to "",,3< h"", ",OCr '0 ,I>.- e~I' ['1X'" ",~ui,i~ [o'lr."'d ,ul--t., ''''<!II oo><d ,C: • J<s.,allA'" 0""1 '" on,,·;.1 l.'" 0,,1, 'y>1""" '" t" 'oo;,Kl...... ",1" ",,,,I" he ",I<cLC"<l ~l' ","""",;,w 0;",1,,,= "f ,he info"",,"O".

TJ-.<rd,<,<. em' oj .... ;<I<'IL.h """'L;';L> ""mmml,. rl,,,,,~ """d", cnp', "me<= poi,. to ;"""'''i 'h< row fC]>Ol1 '''''''r.c<. If ;0" I",,. '"'~ ~""li".""'o...,!;n!' ,h" ""l"""". mcm\>..--r 01 l"'" I Mnk ;''''' for you, p1,,,,,,, ""n,.,,, tIl< 0' In"" ~],. ,\,I,k,. Il"",,, cnp ,>,..liL I.;';~"'." (20,) 144_~,\3? ,,,:f'''''It'''' , Planning and Funding Issues Hindered the Implementation of CBP’s System Availability Project Appendix C Major Contributors to This Report Sharon Huiswoud, IT Audit Director Sharell Matthews, IT Audit Manager Beverly Dale, Team Leader Domingo Alvarez, Senior IT Auditor Frederick Shappee, Program Analyst Pamela Chambliss- Williams Referencer Planning and Funding Issues Hindered the Implementation of CBP’s System Availability Project Appendix D Report Distribution Department of Homeland Security Secretary Deputy Secretary Chief of Staff Deputy Chief of Staff General Counsel Executive Secretariat Director, GAO/OIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs Respective Under Secretary DHS Component Liaison Office of Management and Budget Chief, Homeland Security Branch DHS OIG Budget Examiner Congress Congressional Oversight and Appropriations Committees, as appropriate Planning and Funding Issues Hindered the Implementation of CBP’s System Availability Project ADDITIONAL INFORMATION AND COPIES To obtain additional copies of this report, please call the Office of Inspector General (OIG) at (202) 254-4100, fax your request to (202) 254-4305, or visit the OIG web site at www.dhs.gov/oig.

OIG HOTLINE

To report alleged fraud, waste, abuse or mismanagement, or any other kind of criminal or noncriminal misconduct relative to department programs or operations: • Call our Hotline at 1-800-323-8603; • Fax the complaint directly to us at (202) 254-4292; • Email us at [email protected]; or • Write to us at: DHS Office of Inspector General/MAIL STOP 2600, Attention: Office of Investigations - Hotline, 245 Murray Drive, SW, Building 410, Washington, DC 20528. The OIG seeks to protect the identity of each writer and caller.