The Air Force’s Kessel Run software factory, which brought speed and rapid change through modern agile development practices and “DevSecOps” to USAF programs, is changing the way it does business.
Gone will be the small teams of Active-Duty Airmen and Air Force civilian coders working side-by-side with contract engineers paid by the hour. Instead, government engineers will manage software development work by vendors—more like conventional development.
Col. Richard Lopez, the materiel leader of Kessel Run, told Air & Space Forces Magazine that Kessel Run will now be “government-led, vendor-managed.”
Kessel Run has iconic status. Founded by warriors and technologists frustrated with the glacial pace of military software acquisition, it was intended to bring the speed and iterative approach of Silicon Valley startups to Air Force software development. Lauded as a rare success story, this detachment of the Air Force Life Cycle Management Command at Hanscom Air Force Base, Mass., became a blueprint for other rapid development “software factories” throughout the Air Force and Space Force, many of which used similar naming conventions drawing from Star Wars and the attitude that comes with it—the plucky band of rebels.
The name comes from a brief quote in the original Star Wars movie in which Harrison Ford, as Han Solo, brags that his Millenium Falcon spacecraft is “the ship that made the Kessel Run in less than twelve parsecs;” Kessel Run is a smuggler’s corridor, another reference to working around the rules.
In the military software world, Kessel Run was all aobut a single digital age challenge: How to adapt a procurement system crafted for hardware things made of steel and concrete to buy something living and continuously changing, like software.
Lopez said his changes will preserve the “real secret sauce” of Kessel Run, which was about introducing modern software development practices like DevSecOps and agile, to the U.S. military. DevSecOps puts development, security, and operational deployment into a single team, rather than closed cells; agile is a software development philosophy that breaks coding objectives down into manageable chunks, and then rolls out interative changes in sprints, which can be as short as hours but typically one-to-four weeks in length.
“It’s still DevSecOps. It’s still agile,” Lopez said. “So the hand-in-hand work with the vendors isn’t changing. It’s just the individuals that are coding are now going to be the vendors, rather than mixed government-vendor teams.”
Lopez won a two-year extension on his appointment as materiel leader, through 2026, in order to be sure he can see the transformation through to a successful conclusion.
But critics say Lopez’s plan is really an effort to pull an organization that grew out of a failure of the Air Force’s traditional software acquisition process, back to a more conventional government model.
“It’s back to the future,” said one senior software engineer, a service veteran who has worked at Kessel Run since 2019. What Lopez is describing, the engineer said, “is the way that software development was done forever within the DOD, which generally has gotten us monstrosities and software that really isn’t fit for purpose.”
The engineer spoke without attribution because of their concern about potential repercussions for publicly criticizing leadership. At least one other former Kessel Run employee Air & Space Forces Magazine contacted shared those concerns.
Needed Changes?
Lopez disputes the idea that the changes he is introducing represent a return to so-called “traditional” software acquisition. “It’s not throwing a requirement over the fence and then waiting years and having something come back that wasn’t what you expected,” he said. “This is every single day, working hand-in-hand, integrated in many respects, with the vendor.”
As part of the changes, Lopez added, Kessel Run will move to a single vendor model, where each product portfolio is run by a single vendor paid to deliver a capability or an application, rather than operating in multi-vendor hybrid teams paid by the hour.
Lopez said industry wants the shift to single-vendor teams and that doing so will improve accountability and performance. “There’s a greater sense of, I don’t want to say ownership, because we own the code. It’s ours. But now [the vendors] deliver an end-to-end solution, which is something that industry has been asking us to do, and I think it’s a great opportunity now to work better with industry,” he said.
Bryon Kroger, one of the founders of the software factory back in 2017 and now CEO of Rise8, a military software developer, said some kind of change at Kessel Run was needed. Kroger left Kessel Run in 2019, and said that by 2022, when Lopez was appointed to lead Kessel Run, the software factory was “failing” and “not doing its mission.” said Kroger, who left Kessel Run in 2019.
He attributed that failure in part to decisions he and others made in the early days of the organization.
“There were good reasons why we did everything we did, but also not all of it was right,” Kroger said. “In fact, some of the things that people think made Kessel Run unique and special, if I could go back and do them over again, I’d do them differently.”
But it wasn’t just leadership decisions, he said. “It’s the Air Force that failed Kessel Run and the movement towards better software development.”
Kessel Run Culture
From its beginnings, there were no uniforms at Kessel Run, said Ken Kato, one of the founding hires at Kessel Run who went on to create and lead the Navy’s software factory, Black Pearl.
The ban on uniforms was in part for security, Kato said, noting Kessel Run started in a shared, unsecured office space in downtown Boston. But it also helped to erase differences and break down hierarchies.
“It removes the gap between military personnel and civilians. It removes the gap of rank versus ideas,” he said.
Kato summed up that early Kessel Run culture as “strong opinions, loosely held.” That encouraged everyone, regardless of status, to advocate passionately for their opinion but also be ready to change their mind if new information emerged.
“Each team member, as persons, as developers, as individuals, can come up with ideas independently, and present them, the same way you would in any tech startup: Here’s an idea. What do you think?” he said, without having to worry about being seen as disrespectful to a senior officer.
But the culture also emphasized something lawmakers, Air Force leaders, and acquisition reformers all value, said Kato: speed.
“Typically it takes defense organizations as long as three years” to procure and deploy software, he said. Kessel Run was able to do it in three months.
Changing course
Kroger said Lopez is trying to do the right thing, but may have misdiagnosed the problem. “I think his view is, it’s failing because the permanent [staffers] and the culture are all wrong, and they just need to get back to doing what they’re supposed to do, which is award contracts, and let the contractors build the software. … The empire struck back.”
Part of the issue was that some senior officers objected to the Kessel Run culture, Kroger said. They saw “‘ideas over rank’ as a license for insubordination,” he said.
Pairing Air Force developers with more experienced contractors as mentors in mixed teams grew out of a vision to build an organic software development capability for the Air Force: “a squadron of coders,” Kroger said. But the concept quickly ran up against practical issues.
“How long does it take to create a 10-year programmer? Well, it takes 10 years,” Kroger said. “You can accelerate that timeline with the pairing and apprenticeship model we were using, but it’s still years.” Kessel Run was getting Air Force coders assigned on temporary duty, normally for just six months, said Kroger: “We were turning over 50 percent of our staff every six months. So obviously that has problems.”
Even when the Air Force assigned personnel on permanent duty for three-year postings, “they would go somewhere else and not do software ever again,” Kroger said. “There was no career path for these people.”
Turnover was also been a problem at the leadership level, he said, where the materiel commander and his or her top staff rotate in and out on two-year assignments. “What software company in the world today turns over their entire C suite every two years? Literally the entire C suite. And that’s what’s happening at Kessel Run.”
Over time, he said the lack of a career path for Air Force and government civilian personnel combined with budgetary pressures pushing down compensation for contractors, combined to reduce the effectiveness of the hybrid model. “People are right to criticize some aspects of it and change them,” he said.
Kroger agrees that a “vendor-executed” strategy like the one Lopez is proposing is the right way forward. “It takes time” to build an organic software development capability, he said. “We don’t have time.” For critical capabilities, the Air Force has to buy in highly skilled engineers.
But such a strategy requires government acquisition managers to be highly skilled in software development, so they can write contracts that take advantage of agile software development practices. Conventional government procurement starts with a list of requirements, but that can lead to software that is outdated as soon as or even before it’s compiled.
“You can’t put a contract in front of a contractor as a list of requirements and call it agile because they complete the requirements in sprints,” Kroger said.
The Secret Sauce
Retired Col. Brian Beachkofski, who led Kessel Run from 2020-2022, says the key to understanding the impact Kessel Run achieved comes down to two letters: PM.
In software development, PM means product manager, the person responsible for ensuring developers on the team deliver the capabilities that users need. In government acquisition parlance, PM means program manager, which is an oversight role. “They are just looking at product execution. Is spending this quarter in line with the plan? Are we making progress like we thought?” said Beachkofski, now chief technology officer of Rithmm, a sports data company.
Part of Kessel run’s secret sauce, he explained, was that they had government product managers “responsible for making sure that our user has the capabilities that they need from the software. So who they’re responsible to is actually the user, as opposed to it being more of a check on cost, schedule, performance, on contract execution,” as with a program manager.
The hybrid team model with government product managers, Beachkofski said, meant that government employees and contractors were sitting “on the same side of the table,” alleviating potential tension in the relationship.
“In a traditional acquisition office, you would have the contractor, who’s delivering on the requirements and going to hand over a software package at the end of the contract. By having a time and materials contract [where the vendors are paid by the hour] where the government and the contractors work together. There was more of, ‘We’re on the same side of the table to solve the user requirement,’” he said.
Beachkofski acknowledged that, by the time he left in 2022, Kessel Run was “drifting” away from its adherence to DevSecOps, mainly because the training budget was cut.
“I had to make changes I wished I didn’t have to because of the financial situation, but I’m proud of our ability to support the user and deliver when it counted most,” he said.
He pointed to the organization’s success in rapidly scaling Slapshot, a mission planning app, to ensure the safe evacuation of more than 123,000 people from Afghanistan, amid the chaos of the Taliban takeover.
Ultimately, Kroger said, part of the role of Kessel Run ought to be to create more “educated buyers” of software development services in the Air Force.
A six-month rotation on a hybrid team might not make an Airman a seasoned software engineer, but if they move into acquisition and become a program manager later, that exposure to coding should make them better contract writers, he said.
“I would still like to have some military folks come in and pair [with contract developer mentors] … with the goal of, once they’re done with their six months building an app, they move into the program office, and now they’re much more informed buyers,” he said. “They can provide a better counterbalance to the contractors in engineering, architecture and design, and I think the results are better contracts and better outcomes from those contracts.”
The anonymous Kessel Run engineer, however, says that’s easier said than done.
“There are very few people that even know what the [contract] language should look like,” the engineer said. “How do you translate what Kessel Run does in agile software development into a contract using standard government language? They haven’t even figured that out.”
