I just read a CNN article that essentially says that launching the Obamacare Health Exchange was an impossible task (http://money.cnn.com/2013/10/03/technology/obamacare-glitch/index.html). (I also like the URL which maintains that the issues are still glitches – I think everyone is beginning to realize that they are way past the ‘glitch’ stage).

The CNN article quotes Neil Quinn, CTO for Prolexic:

The rule of thumb, according to cybersecurity firm Prolexic’s CTO Neil Quinn, is for a website-runner to prepare for two to five times the web traffic expected at peak levels.
“But obviously, they weren’t going to be given a huge amount of time or budget on this,” Quinn said. “In an ideal world, you get some more informed estimates of what you are going to see [in terms of traffic].”

WHAT????

Is he trying to tell us that they weren’t given an adequate amount of time and budget to implement the cornerstone of Obama’s 8 years as President??? You’ve got to be kidding me!!!! That’s just laughable. I wonder how many tech people CNN Money had to talk to to get that quote.

The article then goes on to state:

It’s also possible that the Affordable Care Act legislation doesn’t allow the government to do a limited test run or roll out the exchanges slowly, such as going state by state or starting with people whose last names begin with letters A through M.
Those concerns, along with privacy concerns and logistical issues, created what Prince called “essentially an impossible task.” Any option the government chose came with serious cons, he said.

Using a massive cloud service like Amazon’s would have handled the large traffic load — but it would also mean storing sensitive health care information in the cloud and on rented servers, which would have drawn criticism. Instead the government used its own system and servers that it could control, which adds security, but it didn’t have the funds to build robustly enough to handle week-one traffic.

While I agree that the legislation didn’t provide for going state by state or starting with people whose last names begin with letters A through M (which, based on reports would have only mildly helped since this would have only reduced traffic by ½), the legislation certainly didn’t prevent load testing which should have easily identified these issues.

I also agree that a cloud service like Amazon’s would have handled the traffic load -economically I might add. However, what the article failed to mention is that Amazon has a special ‘region’, GovCloud, specifically designed with securing sensitive data. From Amazon Web Services:

AWS GovCloud (US) is an isolated AWS Region designed to allow US government agencies and customers to move sensitive workloads into the cloud by addressing their specific regulatory and compliance requirements. The AWS GovCloud (US) framework adheres to U.S. International Traffic in Arms Regulations (ITAR) regulations as well as the Federal Risk and Authorization Management Program (FedRAMPSM) requirements. FedRAMP is a U.S. government-wide program that provides a standardized approach to security assessment, authorization, and continuous monitoring for cloud products and services. AWS GovCloud (US) has received an Agency Authorization to Operate (ATO) from the US Department of Health and Human Services (HHS) utilizing a FedRAMP accredited Third Party Assessment Organization (3PAO).

So again, while people may try to make light of the issues by calling them ‘glitches’ or an ‘impossible task’ the bottom line is that what is happening with the exchange is just an inexcusable, embarrassing, totally preventable fiasco that had incompetent leadership.

While I realize that the Obamacare Exchanges were large and complicated projects, they also had a large lead time and massive amounts of money thrown at them. There are certainly plenty of people who know how to and have implemented scalability solutions that would have prevented this total and complete embarrassment. In my mind, the government certainly didn’t get value for the money it spent!

Were the following questions not asked and answered?

1. When do we expect the highest usage?
2. What do we expect the peak usage to be?
3. Can we load test the application to ensure it can handle the expected volume?
4. Do we have the right people who have done this before?

For the first question, millions of dollars was spent in advertising to get people to visit the exchange. With all of that anticipation and build up, it is pretty evident that October 1st would be a key and critical day. If this was a private business and people had options, people who experienced the problems they had when trying to access the exchanges would just go elsewhere. Since people don’t have a choice, its good for the exchanges that they’ll be back. Anyway, this wasn’t a surprise that there was a large volume on October 1st, so that’s not an excuse.

I can’t answer the question regarding what the expected peak usage was. It seems like anyone talking about this avoids this question. If some spokesperson said that we received 4 times, even double, the traffic that was expected, that would be a better excuse than saying something like the exchange got more traffic than we ever got from our Medicare/Medicaid website. All I can say is that you have different sites and different requirements. I’ve run sites that comfortably handled all the needed traffic from a VPS to load balanced, dedicated hardware sites that I had to continually monitor based on promotions that marketing ran. Some of the comments that spokespeople are making are just laughable. What analysis and planning was done to ensure these issues didn’t happen?

It’s simple! If you have the money and you know you have a high-profile, high-risk application you load test it! I can’t say for certain whether this was done, but it sure looks like it wasn’t. Obviously the money wasn’t a problem, so the fact that this looks like it wasn’t done is totally inexcusable!

Do you have the right people, e.g. the expertise to do the job right the first time? Again, based on what people experienced with the health exchanges, it certainly doesn’t look like it. Lots of internet companies know how to scale and many know how to do it at a reasonable cost. This isn’t the year 2000 when scaling was difficult and expensive. There are multiple ways to handle scaling, depending on the application, unfortunately, this appears to have been given no thought. Obviously, the project didn’t have the right people.

To me, the bottom line is that, for the money, there was a serious lack of technical knowledge, leadership and understanding of when the application would be most used. Someone simply either didn’t do their job or didn’t know how to do their job. ‘Glitches’ doesn’t even begin to describe what is happening. I think more appropriate words are inexcusable, embarrassment, totally preventable and fiasco.