I feel for all the providers whose products are/were compromised.
I feel for all the users who may have been negatively impacted.
I feel for the OpenSSL Software Foundation (OSF) team members who have contributed to the project.
I feel for all those who will now walk the long road ahead of rebuilding trust, rebuilding systems, and quantifying the potential damage caused by the OpenSSL Heartbleed.
The heartbeat bug and the disclosure timeline will prove to be quite a disruptive event in the consumer technology and enterprise IT markets. More on that in a minute...
How did we escape this potential blood bath?We, like most of the sane world, take advantage of open source software for use in our internal systems as well as in our product offering, VNS3 the cloud network appliance. CohesiveFT extensively tests and vets all aspects of the VNS3 system before making a new version generally available. Additionally we, like most of the responsible ISVs/service providers, take advantage of the downstream Linux providers' practice of just including fixes for security vulnerabilities in certain security libraries like OpenSSL. The result of which is feature freeze on what we spent time and energy testing while still benefiting from the ongoing security patches coming out of the open source project.
I <3 Open Source and OpenSSL
Many are quick to blame open source or the guys (and possibly girls) behind the OpenSSL Software Foundation (OSF), this is wrong. If you're one of those people, stop reading this blog, take a deep breath, and go pound sand.
Open source powers the world, it's a fact and we move on. Moreover, open source projects are great places for security libraries to live. The openness means vulnerabilities are more easily and quickly patched plus the transparency of the code means the projects' provenance is guaranteed and auditable. If you're worried about big brother peaking into your secure systems, anything proprietary is likely already cracked and in many cases with the compensated help of the provider/vendor.
I don't defend all the choices of the guys at OSF, there are definitely some issues with their commercial plan (they're no RedHat). But they fight the good fight and have provided a serious amount of value over the life of the project across all geographies. Regardless of the fallout from this bug, we'll still be net positive as a result efforts as a whole.
It's ECON 3545-001 Environmental Economics at CU-Boulder all over again (yes, I'm a Buffalo). The market has failed to assign appropriate value to the OpenSSL project. There some some ridiculous estimates about OpenSSL market share (see apache and nginx) yet it's "4 guys and a dog" struggling to keep up with the code base and some commercial projects. Unfortunately the weeks immediately following a major security hole disclosure isn't the best time to ask for $. Maybe I'll followup with a post in a couple months to solicit corporate contributions to OSF. (I am a OpenSSL contributor).
NFV can be Lifesaving Nitro*
What will be the nitroglycerin for old man Internet's recent heart attack? The Heartbleed bug is real, the fallout is and will be real, the NSA is real and has been listening. Now what?
It's time to apply what we've learned. In the event VNS3 was affected by the bug, we could have built and delivered a new image (for all our supported public cloud and virtual environments) in a matter of hours. Our customers would have simply swapped in the new image with total potential downtime limited to minutes, if any. Wait what? Yeah instance based NFV is the future for a number of reasons but let's focus on the fact that is software not hardware given the topic of conversation... Bugs.
Do some hardware providers really burn in OpenSSL in to their chips? Maybe, but I hope not. Bruce Schneier, a cybersecurity researcher and cryptographer, was recently quote in the Wall Street Journal seemed to think so when he said, "the upgrade path is going to involve a trash can, a credit card, and a trip to Best Buy." That's a little sensationalist especially given that Cisco and Juniper are saying software patches will be out soon as opposed to saying new replacement hardware devices will be shipping in early Q2. But the fact is customers are still waiting for some patches and the all clear.
Exploring the hypothetical is fun when you're sitting around a table with some whiskey and some buddies, but it's less useful when talking enterprise IT strategy. Typically a real world event is needed to help us look at our decisions from a different point of view - enter the Heartbleed. I argue that this type of exploit makes the case for running a tight, limited, high performance and isolated VMware stack in the corporate DMZ. Just run a rack of vSphere with it's own dedicated set of switches. Run all your DMZ edge network devices as NFV appliances. Make your DMZ a micro cloud. If anything on the DMZ edge is compromised or vulnerable, the fact that it's an instance-based NFV appliance means it's quickly and easily replaced. Turn on aes-ni support on the Intel chips (why public cloud don't do this already is straight silliness) running your DMZ micro cloud and you'll even have some nice savings on any encryption overhead.
The bottom line is this - as an NFV vendor, I'm incentivized to get my customer new images. Hardware vendors are "fire and forget." The major difference is additional unit manufacturing cost (and delivery cost). I as a virtual appliance vendor have a unit and delivery cost of $0 and that means every time I have a new version, I want to get it into my customer's hands ASAP. Latest and greatest means I'm happy because my customers are happy keeping up with my new hotness.
*Added Bonus - NFV won't kill you if you're using Viagra