Photo: SSgt. Alexander Martinez. Illustration: Pete Linforth
The Air Force has decided to start over on a software upgrade for its air and space operations centers (AOCs), the heart of USAF’s combat connectivity and information fusion. After years of blowing past deadlines and going hundreds of millions of dollars over budget, the Air Force in July fired the contractor developing AOC Weapon System 10.2 and decided to do the work largely in-house.
Although military software development is always fraught with uncertainty, service officials hope this new approach will allow the program to proceed at a fraction of the expected cost of relying on a contractor—assuming Congress allows funds to be reprogrammed to do so.
Despite the setbacks, the AOC software upgrade is critical and must go forward, Lt. Gen. Arnold W. Bunch Jr., USAF’s top uniformed acquisition official, told Air Force Magazine. The reincarnated project is named, appropriately enough, AOC Pathfinder. The upgrade, previously led by Northrop Grumman, is necessary to better sort, collate, and organize the vast streams of information that flow into an AOC.
The modernization had previously been budgeted for $3.5 billion over its lifetime. How USAF plans to achieve the same ends is still to be determined, as is what exactly the service will need after it spends its first $36 million—if and when it gets it. But according to Bunch, it has to happen one way or another, labeling the effort critical.
The new acquisition strategy aims to apply industry’s best practices, an Air Force spokesperson explained. The idea is to obtain “more rapid and transformative AOC capabilities, to create near-immediate and repeated value to our warfighters,” she said.
Of the $36 million USAF has asked to spend on the revamped project, some $22 million would be spent on development contracts, $10 million would buy commercial off-the-shelf products and services, and $4 million would pay for other government support.
AOC Pathfinder will take the novel approach of relying on airmen to figure out the way ahead. Assuming it gets the green light, USAF will partner with Defense Digital Service—the agency called in to review the predecessor Northrop Grumman project when it was foundering—and the Defense Innovation Unit-Experimental (DIUx), which bills itself a “fast-moving government entity” able to quickly solve defense problems.
USAF expects about 100 people to work on Pathfinder, including government, military, civilian, and support contractors. It will be managed out of Hanscom AFB, Mass.
The goal is to achieve “a partial replacement AOC software capability” just a year from the moment AOC Pathfinder gets underway. The “path” part is still being worked out.
BEHIND SCHEDULE
The original project was intended as an upgrade to AOC Weapon System 10.1, which receives, hosts, and parses incoming data used in “fusion warfare,” as it has done for two decades. Tagged AOC-WS 10.2, the upgrade’s focus was on cybersecurity and open architecture, two elements that weren’t a priority when 10.1 came online in the 1990s. While chiefly a software improvement, the program also aimed to deliver some hardware in the form of computers, servers, and related equipment.
Back then, “we weren’t as worried about cyber,” Bunch told Air Force Magazine. “Cyber vulnerabilities and cybersecurity weren’t on any of our screens.”
Adding cybersecurity open architecture was no mean undertaking.
First, Lockheed Martin, having worked on the project from 2006 to 2011 in what Bunch’s office called “a small initial investment to begin concept development activities,” opted not to bid when the Air Force put the AOC upgrade out for contract.
The project wasn’t officially a program of record until Frank Kendall III, then undersecretary of defense for acquisition, technology, and logistics, decided it was technologically feasible in 2013, a determination defined as Milestone B. That’s when Northrop Grumman stepped in and was awarded the development contract.
Northrop Grumman’s price for the development phase of the project was $374 million in 2013, but had since risen to $745 million, according to Bunch’s office. Over its lifetime, including procurement and support, the program’s total cost was recently estimated at over $3.5 billion.
The AOC modernization was also nearly three years behind schedule and not expected to go live until December 2019. Northrop Grumman declined to comment on the program or its termination.
The doubled development cost, continued difficulty meeting milestones, and the schedule delay exceeding 12 months brought the program to a halt this May—as these criteria officially triggered the need for a critical change—months before it was officially terminated. USAF couldn’t keep the project going without an additional $66.3 million, which Congress declined to provide on top of the program’s already appropriated—and spent—$28.9 million this year.
USAF issued a stop-work order April 19.
While the scope of AOC Pathfinder hasn’t been set, the goal is the same: Modernize the AOC.
The AOC 10.2 program rested on two pillars: Cybersecurity must be built into the current AOC network, and the components of the network need to be upgradeable. By way of example: a new mobile app, which can be installed on older smartphones. Such an app can be just a few days old but work on a much older phone because the phone was created with an “open architecture” that accepts new programs.
This capability is crucial because some of the apps that the AOC uses now and will likely create in the future aren’t wholly owned or controlled by USAF, Bunch said. Some of them are third party.
RE-BASELINED
“If I move a captain who works in air operations in CENTCOM [US Central Command] to the Pacific area, I don’t have to retrain [him or her],” Bunch said, adding he wants AOC veterans “to have standard things they’re looking at.” Avoiding “retrains,” as Bunch put it, is another one of the goals of the new network, aside from the cybersecurity and programming openness.
As the program overran its schedule several times, Bunch was forced to submit a Critical Change Report to Congress. (This was the program’s second critical change, the first one being the 2013 change bringing Northrop on board.)
According to DOD guidelines, such a report outlines—among other things—whether the program is “essential to national security,” if any alternatives could provide “equal or greater capability at less cost,” whether new cost estimates are reasonable, and whether the management of the program is “adequate.” Though this is similar to a so-called Nunn-McCurdy breach, somewhat different rules are at work.
“We had a bunch of people go in and look,” Bunch said, describing the group as in-house experts and members of the Office of the Undersecretary of Defense for AT&L’s Defense Digital Service, which is now a key partner on AOC Pathfinder.
DDS is a small team that “talks to engineers, talks to decision-makers” for about a week and comes back with recommendations based on best commercial practices and other technical wisdom.
A variety of other DOD experts also delivered feedback on the program in the form of, among other things, unreleasable “deficiency reports.”
After getting feedback from these experts, Bunch’s office changed the methodology of the project, he said. The program was re-baselined, including a change to how Northrop Grumman developed the software and how it conducted interim testing on the software instead of waiting on milestones.
“You have to rapidly test that code,” former acting Air Force Secretary Lisa S. Disbrow told Air Force Magazine about the new methodology, describing the team’s findings. “If you don’t—if you take an incremental approach, which we tend to do in the acquisition system—you allow the code to fail while you’re building other code.” Under traditional processes, USAF wouldn’t necessarily know if the initial code was failing because it wouldn’t test it until after the Air Force had “a number of different systems together,” she said.
Setting a new baseline—a fresh scope of work to be done, the schedule, the cost—required more money than was appropriated, so Bunch had to ask Congress to reprogram Fiscal 2017 dollars to keep the project above water. That didn’t happen, and the Air Force is keeping everyone abreast of the program’s development and needs because “with a critical change, you have to be very transparent,” Bunch said.
While AOC 10.2 is in limbo, 10.1 must keep functioning, and that means maintenance. Just seven days after USAF ordered Northrop Grumman to stop working on AOC 10.2, it awarded Raytheon a $375 million contract to sustain 10.1.
While Northrop Grumman’s contract was about replacing “legacy infrastructure with a new, modern architecture,” sustainment wasn’t part of the deal.
“The contract just awarded to Raytheon is to modernize and sustain the AOC weapon system,” the Air Force explained. “Raytheon’s immediate effort will focus on sustaining 10.1 as the currently fielded system and will transition to modernizing and sustaining 10.2 if/when it is fielded.”
Raytheon’s Todd C. Probert, vice president for mission support and modernization, intelligence, information, and services, said in an interview that “depending on the need,” readiness or relevance will take the lead as the focus of sustainment.
OLD BUT RESILIENT
How does that affect Raytheon’s work on a system that needs to be completely recast? There is no classic set of requirements, Probert said.
Whatever the requirements are, Probert said Raytheon will be hiring nearly 200 engineers and related personnel for the project, which he called “a really exciting and really important weapons system for the government.”
Bunch’s office said the awarding of the sustainment contract was “unrelated and happens to be coincidental to the AOC 10.2 stop-work order.” Jacobs Technology of Texas was in charge of sustainment previous to Raytheon.
In addition to a shortage of coders to deal with software-driven projects, DOD struggles with management. There isn’t enough expertise in place to manage state-of-the-art coders, Disbrow said. The Air Force is “looking at what’s the right mix” of contract, civilian, and DOD entities, she said.
“I really think it’s more that cyber and IT are areas that have rapidly progressed outside the department and we don’t have the workforce in place,” said Disbrow.
The very antiquated nature of some of USAF’s systems could ultimately be their saving grace, Disbrow noted.
“Just being as old as they are—not in the network—builds in some resilience,” s?he said. “We’ve got shortfalls out there we have to address. And I think we are—it’s a matter of how long [before] you can get it all done.”
The lessons derived from the AOC modernization debacle can be applied to other modernization programs as well.
“Almost all the weapons systems we’re modernizing are software-centric,” Disbrow told Air Force Magazine in June, adding that the service is trying to “change the way we develop and the way we acquire these systems.”
Military software development is never easy, and every entity agrees on one thing: Upgrading the AOC is critical. Whatever gets done has got to be done quickly.