The drama of the HealthCare.gov Ad Hoc team is now a modern tech fable: a small cadre of young geeks from Silicon Valley and President Obama’s election campaign parachute into the federal bureaucracy to rescue the site and help exceed the goal of 8 million insured households nationwide. But even as they worked 80-hour weeks to salvage the botched creation of thousands of technocrats employed by 55 different contractors, another drama was occurring in stealth. Members of the Ad Hoc team were already looking ahead to the next version, recruiting a second wave of programmers drawn from startups as well as larger companies like Google.
That team, officially dubbed Marketplace 2.0, is creating core features of the next generation of HealthCare.gov that will debut when the next enrollment period begins November 15. Key upgrades include a simpler interface for a majority of users, a more robust plan comparison tool, and a new login system rebuilt from the ground up. The effort doesn’t deal with components like rating engines, tax credits and financial management; instead it focuses on making it easier for applicants to connect with insurance plans. Officials at Centers for Medicare and Medicaid Services outlined the key pieces for WIRED:
Application 2.0: This is the new front end for the application, including a component that determines an applicant’s eligibility for a plan. Though the underlying logic is the same as the recent one developed by the Ad Hoc team and used by more than 5.45 million people in the first go-round, the 2.0 team aims to create a simpler, more efficient user interface designed to boost the percentage of visits that lead to health plan subscriptions. Officials estimate that about 35 to 40 percent of applicants with more complicated household situations will still have to use the more painstaking process from the 2014 enrollment period. But these officials see the new system eventually handling a broader range of scenarios. Application 2.0 is also designed to work more effectively when applicants use mobile devices. More than a fourth of those who enrolled in the first sign-up period came to the site through mobile devices.
Plan Compare 2.0: Browsing and selecting plans was difficult on the original site. Way too often, it would crash, losing all applicant information and forcing the user to repeat the process. The worst of this was fixed by the surge team, but the new system will retain data even more reliably and have a new set of screens that guide applicants more clearly through the steps.
A tool for window shopping will enable those who aren’t sure they want to sign up to sample plans anonymously without having to go through the effort of putting in personal information. This feature will improve upon the heavily trafficked so-called premium estimation tool offered in the first signup.
Scalable Login System: This will be another built-from-scratch system handling identity management and account creation. It replaces the flaky login system that drove the first rescue team crazy and chased away countless potential applicants.
The effort to create a new version of the Federally Facilitated Marketplace started last November. US Chief Technical Officer Todd Park asked one of the Ad Hoc team members, former Googler Jini Kim, to see if some of her colleagues would be interested in joining a second unit. She began recruiting just before Thanksgiving, targeting friends from Google and others who worked at startups, particularly those associated with the Y Combinator tech incubator.
“I remember getting the phone call,” says Brandon Ballinger, who agreed to come onboard. “Jini asked, ‘Are you ready to skip Thanksgiving, skip Christmas and skip New Years?” Kim also called Joey Liaw, a former Gmail engineer who also worked for Sean Parker’s much-hyped video chat service called Airtime and for a YC startup. Later some of those people would call in their own friends and by December, there were eight new engineers crunching at CSM’s facilities outside of Baltimore. David Chang, a 2012 MIT graduate had just shut down his Y Combinator startup and accepted a job at Khan Academy. The other five included recent Stanford graduates, someone from Google Scholar, and engineers from startups in New York City and Silicon Valley.
What Kim pitched as a quick stint become something of a marathon, as work continued nonstop through Memorial Day and beyond.. (“I started with zero Hilton points and now I’m Hilton platinum,” says one California drop-in.) Kim served as de facto product manager and Liaw as the tech lead (Kim has recently eased up on her involvement.)
The small group of recruits became known as the Marketplace Light team, charged with providing a simplified process—referred to internally as EZ App—to sign up with an insurer. This streamlined method would be available to those whose family and work situations were uncomplicated. (A majority of applicants, perhaps 65 percent, meet those standards).
“They knew this platform needed some architectural changes and the idea was that we would work on the next generation,” Liaw says. “We know how to deploy consumer applications that scale. We use the Silicon Valley playbook utilizing open source and off-the-shelf –technologies. This is something that we’re good at—we’ve done it over and over.”
But there was some uncertainly on how much The Marketplace Light team would contribute. “Circumstances were pretty tough,” Kim says. “You’re in the polar vortex of Columbia, Maryland, you’re in a hotel, you’re working with legacy systems that don’t make sense. We could have done a lot more.”
The engineers at first hoped to do a significant overhaul of the front end in time for the big rush at the end of the open enrollment period in April. “The idea was essentially to lessen the load,” says Greg Gershman, a member of the original rescue team who plans to return to CMS. “If they had done that it would have definitely taken traffic off the main site. But they ran into troubles.” To the frustration of some team members, there was simply not enough time to work through the elaborate security authorizations that were required to implement any changes in a government website. “Security is one of the biggest obstacles we had to deal with,” says one engineer, who complains that regulations require useless protocols for relatively little protection. They saw it as the most nettling of many example of how government IT regulations, even well meaning ones, have morphed into a red-tape nightmare that slows progress without delivering benefits. ‘The focus is not so much on security but more like checking a lot of boxes.”
Even as the team found a way to work effectively with CMS lifers—both sides realized they could learn from each other—some engineers felt their bosses were being overly cautious about instituting dramatic changes during this first open enrollment because of the high political stakes. There was a generalized fear that an error in implementation could lead to a whipping by Republican representative Darrell Issa in a congressional hearing. “People are more scared of things here,” says Ben Komalo, a Marketplace Light engineer who recently returned to his job at Khan Academy . “The costs of failure are perceived as being much higher than where we’re from.”
Nonetheless, the Marketplace Light crew got some of their work into HealthCare.gov during the open enrollment period. This included improvements to the registration screens that applicants confronted when they visited the site. These screens were crucial—if they were confusing or presented frustrating obstacles, some applicants would simply exit. On the early iterations of the site, this happened a lot, in many cases because of poor design. “For example,” one engineer says, “you had to choose a user name that had a special character such as an underscore or a dollar sign, which normally is a requirement applied to passwords, not to user names. So a lot of people were confused by that.” Worse, if someone chose a user name that already existed, the applicant would not find out until the end of the third page—at which point the system would make the applicant start over.
The Marketplace Light team fixed this by using something apparently in short supply during the original design of HealthCare.gov: common sense. “We just did the most basic simple thing that anybody would do,” a team member says. “Make one page, and use the e-mail as the user name.”
When the new screens went live in February, the team resisted the temptation to push them out to all users, opting instead for a carefully calibrated rollout. They quietly exposed changes to one percent of users and compared the results by using what’s known as A/B testing, consistently comparing results of the test version to the existing version. Only after determining that the rates of error were low and the rates of conversion were high, did they expand the upgrade to more and more applicants, eventually offering the improvements to all users several weeks later.
Such simple revisions to the application process, including trimming the sign-in pages from five to two, led to a ten percent increase in successful completions overall. For those who accessed the site via mobile devices, the boost was thirty percent.
After the open enrollment period, the team got the go-ahead to proceed, and Marketplace Light was redubbed Marketplace 2.0. The CMS hasn’t decided whether those who signed up last year will have to re-enroll, but certainly there will be millions more trying to sign up for a new plan, perhaps their first. (Meanwhile, HealthCare.gov continues to take applications from new Medicaid recipients and people whose circumstances justify a change in health plans.)
The team plans to roll out changes throughout the summer and early fall, testing with A/B software and then quickly make fixes and improvements. Though this is standard practice in Silicon Valley, it’s fairly novel in government.
Perhaps the biggest step towards reliability involves the data center that will run these new parts of the system. The 2.0 team has convinced CMS to host parts of the new HeathCare.gov on the same flexible infrastructure used by smart tech companies, from tiny startups to Netflix: Amazon Web Services. This was not an easy sell due to government concerns about security. The usual process for government data operations makes it incredibly difficult to secure all the servers that may be needed at a given time. Government workers must first obtain authorization by calling the data center and making a formal request that specifies the number of servers needed and the programs that will run on them. This cumbersome procedure makes outages inevitable during peak periods. In contrast, as the Marketplace Light engineers knew well, with AWS the entire process is automated. When Amazon’s system detects a surge, it provisions enough computing power to handle the bigger load.
The team fought to overcome cultural resistance to using outside servers and then undertook the painstaking process of certifying AWS as secure enough to handle government healthcare information. “There were a lot of specific barriers,” Ballinger says. “Because this type of thing is so rare, actually nobody can enumerate the entire sequence of steps. At one point we built a chart of every step we needed to take in order to use AWS, and it had, like, 30 different nodes.”
The team prevailed, largely because by then the newcomers had the enthusiastic support of CMS lifers like Jon Booth, the director of the sub-agency’s Web and New Media group, who says that the go-ahead was made easier because some Health and Human Services customers had successfully completed security audits of AWS and had started to run programs on it. “It was a no-brainer for us,” Booth says.
Whether or not Marketplace 2.0 works as promised, the effort has already achieved a turnaround of sorts. In the past, such an effort would have avoided risky innovation, producing something with Rube Goldberg-esque code bases, a lockdown on data centers, and hundreds of veteran engineers schooled in the traditional ways of producing government IT. (Like… the original version of HealthCare.gov.) Not this time, or at least not as much. Marketplace 2.0 will use contemporary tools, a third-party automated data center, and a system of A/B testing and rapid iteration. And though the Marketplace 2.0 team will grow a bit in the coming months—with some original members of the rescue team returning as contractors—tech lead Liaw reports that the current headcount stands at ten.
Moreover, the rescue effort has provided a blueprint for reforming long-moribund government IT. The next step is getting more people like those on the Ad Hoc and Marketplace 2.0 teams into public service. Indeed, many of them have become evangelists for this cause. “This is my second job—I’m only 24,” says David Chang, who took over as de facto product manager after Jini Kim. “It’s phenomenal to have this impact on how government does IT, how millions of people get healthcare.”
While Affordable Care Act itself remains controversial, the government is finally using smarter techniques on the website that signs you up for it. That’s got to be healthier.
