e-Discovery Team

The Negligence of One Open Source Software Contributor Put Millions of Internet Users’ Confidential Data At Risk.

Is Open Source Software Appropriate for Major Security Applications?

It was small oversight. Segglemann forgot to add a single line of code limiting the size of memory access to a feature called heartbeat (thus the nickname for the bug, heartbleed). Oops. These things can happen. Easy to understand. Hey, it was, after all, one minute before midnight on New Years eve 2011 when he submitted his work. I kid you not. Segglemann knew that another expert was going to check his work anyway, so why should  he be too concerned? Too bad the supervising expert missed the error too. Oops again. Oh well, that’s open source for you. Segglemann did not get paid for his work, and there may be no legal consequences for his gift to the world, a gift that many security experts call the worst thing to ever happen to Internet security.

Bruce Schneier, a leading digital security analyst that I follow, says that “‘Catastrophic’ is the right word. On the scale of one to 10, this is an 11.” Brean, How a programmer’s small error created Heartbleed — a secret back door to supposedly secure sites (National Post, 4/11/14). For more on Schneir’s thoughts on Heartbleed, see the Harvard Business Review interview of him by Scott Berinato. 

Rusty Foster wrote in The New Yorker that: “In the worst-case scenario, criminal enterprises, intelligence agencies, and state-sponsored hackers have known about Heartbleed for more than two years, and have used it to systematically access almost everyone’s encrypted data. If this is true, then anyone who does anything on the Internet has likely been affected by the bug.” Forbes cybersecurity columnist Joseph Steinberg wrote, “Some might argue that [Heartbleed] is the worst vulnerability found (at least in terms of its potential impact) since commercial traffic began to flow on the Internet.”

For details on Heartbleed see the Hacker News article by Mohit Kumar, HeartBleed Bug Explained – 10 Most Frequently Asked Questions (Hacker News, 4/14/14). Hacker News includes this good video explanation of the details by Fierce Outlaws:

Bottom line, this little programming error, which some experts refer to as a buffer over-read bug, has huge implications for the security of the Internet, and Android phones, which use the same protocol. It could affect almost every Internet user in the world, depending on who else knew about the mistake, and for how long. At this point, nobody knows. So far only one 19-year old Canadian hacker has been arrested for exploiting the bug, and one online railroad payment system in Russia has discovered it had been hacked. If all that were not bad enough, the security firm Mandiant claims to have found evidence of heartbleed based attacks on one of its client’s virtual private networks (outside of the Internet). Heartbleed vulnerability was used to get pass the firewall and gain access to the VPN.

The Heatbleed catastrophe has dramatically revealed that our current system of Internet security is primarily based  on an open source software called OpenSSL. Heartbleed has shown that the whole security of the Internet can depend on one unpaid volunteer like Segglemann, a lone PhD student in Münster, Germany, who had nothing better to do on New year’s eve than finish a freebie software coding project. No doubt he thought it would help his resume. Bad decision.

Something is terribly wrong when the whole Internet is vulnerable due to the mistake of one math student. This mistake should be a wake up call to change the system. I conclude this blog with a call for dialogue among security experts, open source experts, white-hat hackers, lawyers, the FBI, consumer advocates, and others, to come up with serious reforms to our current Internet security infrastructure, including especially reforms of OpenSSL as an organization, and to do so by the end of this year. The public trust in the security of the Internet cannot withstand another Heartbleed, especially if it turns out that thousands, perhaps millions, have been injured. (We already have reports that the hack of the Russian railroad website allowed 10,000 credit card accounts to be stolen.)

Seggelmann claims it was just a mistake. In his words, a trivial error. He seems kind of blasé about it in his only interview to date, a short talk with an Australian journalist. The interview is quoted in full below. In fairness, I do not think Seggelmann realized the implications of his error at the time he spoke. (He has stopped talking now, no doubt on advice of legal counsel, and his current employer, Deutsche Telekom.)

Seggelmann denies that he was paid to do this by the NSA or anyone else. Most of the articles on him written to date just take his word for it. Oddly enough, most writers even express sympathy for Segglemann. The response you see in Huffington Post is typical: “You could blame the author, but he did this work for free, for the community, and with the best of intentions. ” Oh really. How do you know that? Because he said so? I am tempted to say something about naive bleeding heart liberals, but have been accused of being one myself, and besides, it is a bad pun, so I will not.

I hope Robin Seggelmann is telling the truth true too, but have been a lawyer far too long to believe anything a person in his position now says. Plus, the circumstance of posting such important code just a minute before New Years is clearly indicative of carelessness. If the NSA was behind it, and Bloomberg has reported that they have known about the defect for years, then I expect we will know that soon enough from Snowden. If someone else was, we may never know.

Are Seggelmann or OpenSSL Liable for any Damages that Heartbleed May Cause? 

Even if this was just a mistake, not fraud, major errors like this have consequences, but for whom? Innocent users of websites operating this code for over two years may already have been victimized. We do not know yet what damages this mistake may cause, but we already have had reports on one arrest in Canada, and one theft of credit card numbers in Russia.

That is just the tip of the iceberg. Who will be responsible for the damages caused to so many? Will it be Seggelmann himself, or perhaps the not-for-profit open source group, OpenSSL, that he did this work for as an unpaid volunteer? Although I have no sympathy for a person whose negligence has caused such havoc, I doubt Seggelmann will ever be forced to reimburse anyone for the harm he has caused. 

Perhaps the operators of websites who told their users that their website was secure? Probably not them either, but it may be a closer question. This may be a rare situation where there is no remedy for people damaged by another’s negligence. It will depend on the facts, the as yet unknown details. But, rest assured, much more of the truth will come out in due time. I fully expect some lawyer, somewhere, will file suit when damaged victims appear, or maybe even before.

It will probably be difficult to hold OpenSSL liable for a number of reasons. First or all, who or what is OpenSSL? It appears to be a type of legal entity that we would call in the U.S. an unincorporated association. It is often treated something like a partnership in U.S. law.

According to the Washington Post, OpenSSL‘s headquarters — to the extent one exists at all — is the home the group’s only employee, a part timer at that, located on Sugarloaf Mountain, Maryland. He lives and works amid racks of servers and an industrial-grade Internet connection. Craig Timberg, Heartbleed bug puts the chaotic nature of the Internet under the magnifying glass (Washington Post, 4/9/14).

You cannot make up stuff like this. Truth is always stranger than fiction. Tineberg’s article also reports that the software that serves as the backbone for security on the Internet has, due to the lack of personnel and funds of OpenSSL, never been though a security audit, a meticulous process that involves testing the software for vulnerabilities.

Here is what OpenSSL has to say about themselves:

The OpenSSL Project is a collaborative effort to develop a robust, commercial-grade, full-featured, and Open Source toolkit implementing the Secure Sockets Layer … managed by a worldwide community of volunteers that use the Internet to communicate, plan, and develop the OpenSSL toolkit and its related documentation.  . . .  [Y]ou are free to get and use it for commercial and non-commercial purposes.  . . .

The OpenSSL project is volunteer-driven. We do not have any specific requirement for volunteers other than a strong willingness to really contribute while following the projects goal. The OpenSSL project is formed by a development team, which consists of the current active developers and other major contributors. Additionally a subset of the developers form the OpenSSL core team which globally manages the OpenSSL project.

This is the way most open source software works. As open source software goes, OpenSSL is one of the most successful in the world (well, it was, until this whole catastrophe thing). Their product, OpenSSL, was, and still is, the world’s most popular open source cryptographic library. It is used to encrypt most of the traffic on the Internet. About two-thirds of web sites with an “S” at the end of the HTTP address use this freebie software. 

Seggelmann had an accomplice of sorts at OpenSSL, although I do not mean to imply any type of conspiracy by use of this word. There is no evidence of that. But I am sure people will look into that possibility, not only government investigators, but also private eyes, especially if and when they are motivated by the kind of mixed greed and fear incentives that only law suits can bring. The appearances now all suggest that the double-checker just happened to miss the trivial error too. OpenSSL, like most good open source projects, has quality control procedures. Proposed code contributions are double checked for mistakes by a senior contributor to OpenSSL before they are accepted.

In this case Seggelmann’s work was checked by Stephen Henson. He is a freelance crypto consultant (his words) based in the U.K. who has a PhD in Mathematics. He is still listed by OpenSSL as one of only four core team members of this open source group. That’s right, four people; one of them the part-time employee who works out his home on a mountain that serves as the group’s headquarters.

So after two people looked over the new code contribution, way too casually as we now know, the code was approved. Soon thereafter millions of websites started using it and so made themselves susceptible to attack.

I could only find one legal disclaimer in the OpenSSL website, but I bet this changes soon as the academics in charge of this non-profit association start to wake up to legal realities:

THIS AREA IS PROVIDED BY THE OPENSSL PROJECT “AS IS” AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OPENSSL PROJECT OR ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

I suspect the enforceability of this language may get tested in some court somewhere in the world, probably in the U.S., but to what end? I doubt OpenSSL has any assets, much less insurance, and, even if you could prove proximate causation, how deep could the pockets of Segglemann, Henson, and other contributors be? It is more likely the primary targets for restitution will be the companies who used the defective open source software in their servers, thus exposing their users’ confidential information.

Was it negligence for commercial sites to rely on Open Source software?

Was it negligence for commercial sites to rely on Open Source software? Free software donated by all-too-human expertse? I do not think so. I will not go that far, but I’m willing to bet some lawyer somewhere will go that far. With a mistake this size it is almost inevitable that a class action suit will eventually get filed against somebody.

With the facts I have seen to date I do not think there was adequate notice to the adopters of this free software as to its unreliability to support a cause of action against them for negligent use of OpenSSL. But can the same be said now after the Heartbleed disaster has come to light? Now that we know the mistakes of only two men can put everyone at risk? Maybe not.

Is this the Beginning of the End for Open Source?

This may spell the beginning of the end of widespread commercial adoption of free software, at least for security purposes. After all, it took a for-profit company, Google, to discover the error in the software that many of its websites were using too. What we do not know is how many hackers, government sponsored or free lance, had previously discovered this mistake. How many had previously exploited this flaw to discover the supposedly secret passwords of the hundreds of millions persons potentially impacted. What makes this even worse is that we will never know. The biggest programming error in history made it possible for hackers to steal your data without ever leaving a trace.

Many believe the NSA has been exploiting this flaw for years. Who knows what criminal enterprises and foreign governments may have done the same?

There are two rational responses to this open source security scandal. One, stop using the Internet for anything that you want to keep secure, like all financial information. Or two, stop using Open Source, and instead use paid software, software with real safeguards, and with an entity or entities who will stand behind their products, and insurers who will stand behind them.

Society today relies heavily on the Internet. Commerce relies heavily on the Internet. If security is at risk, our current way of life is at risk. It is that important. So the first alternative is out.

This means we have to stop reliance on Open Source software for security, at least the way it is run now. We need the safety of big corporations who will have a direct economic incentive to take responsibility for their work. We need paid employees, not volunteers. Ones who will get paid bonuses for doing great work, and fired if the make Heartbleed type errors that put us all at risk. Either that, or we need major reform of this open source non-profit so that they are accountable. We are way beyond the hobbyist beginnings of the Internet, a time I remember well, and yet we still delegate major Internet responsibilities to small, unregulated groups of independent associations.

The Heartbleed disaster shows that reliance on open source software for commerce is a risky proposition when it comes to security. It may save users some money, but the risk of error may be too high. Consumers will demand that companies pay up and protect their personal data security. As Chris Williams put it in his article for The Register, OpenSSL Heartbleed: Bloody nose for open-source bleeding hearts:

Open source or open sores? The crux of the matter is that OpenSSL is used by millions and millions of people, even if they don’t know it, but this vital encryption software – used to secure online shopping and banking, mobile apps, VPNs and much more – has a core developer team of just four volunteers who rely on donations and sponsorship. The library code is free and open source, and is used in countless products and programs, but Seggelmann and others point out that the project receives little help.

The use of open source software for everything was a fine experiment, an idealistic one based on the notion that crowdsourcing provided a better alternative to free enterprise, that capitalism could be replaced by a volunteer society of dedicated altruists. Personally, I was always skeptical. I think that competition is a good thing and helps build better products. Heartbleed confirms the skepticism was warranted. Heartbleed has exposed the dark side of crowdsourcing, the inherent weaknesses of volunteerism. The dark side of crowdsourcing is that the crowds will not come, or will stop coming. Here the crowd that checked a critical update to the code consisted of two people only, Robin Seggelmann and Stephen Henson. Two is never a crowd. In fact, a jury may some day be called upon to decide whether it was reasonable to release security code after only two people looked at it.

Robin Seggelmann’s Side of the Story

Ben Grubb is the only journalist so far to get an interview of Robin Seggelmann, published in Man who introduced serious ‘Heartbleed’ security flaw denies he inserted it deliberately (Sydney Morning Herald, 4/11/14). Here are the key excerpts and quotes from Grubb’s article, but I suggest you read the entire article and Grubb’s followup articles too. He has an interesting perspective, including criticism of Google’s handling of the release of information of the bug’s discovery.

Dr. Seggelmann, of Münster in Germany, said the bug which introduced the flaw was “unfortunately” missed by him and a reviewer when it was introduced into the open source OpenSSL encryption protocol over two years ago. “I was working on improving OpenSSL and submitted numerous bug fixes and added new features,” he said. “In one of the new features, unfortunately, I missed validating a variable containing a length.” After he submitted the code, a reviewer “apparently also didn’t notice the missing validation”, Dr Seggelmann said, “so the error made its way from the development branch into the released version.” Logs show that reviewer was Dr Stephen Henson. Dr Seggelmann said the error he introduced was “quite trivial”, but acknowledged that its impact was “severe”.

Conspiracy theories. A number of conspiracy theorists have speculated the bug was inserted maliciously. Dr Seggelmann said it was “tempting” to assume this, especially after the disclosure by Edward Snowden of the spying activities conducted by the US National Security Agency and others. “But in this case, it was a simple programming error in a new feature, which unfortunately occurred in a security relevant area,” he said. “It was not intended at all, especially since I have previously fixed OpenSSL bugs myself, and was trying to contribute to the project.” Despite denying he put the bug into the code intentionally, he said it was entirely possible intelligence agencies had been making use of it over the past two years. “It is a possibility, and it’s always better to assume the worst than best case in security matters, but since I didn’t know [about] the bug until it was released and [I am] not affiliated with any agency, I can only speculate.”

Benefits of discovery If anything had been demonstrated by the discovery of the bug, Dr Seggelmann said it was awareness that more contributors were needed to keep an eye over code in open source software. “It’s unfortunate that it’s used by millions of people, but only very few actually contribute to it,” he said. “The benefit of open source software is that anyone can review the code in the first place. “The more people look at it, the better, especially with a software like OpenSSL.”

Future Heartbleed prevention. Asked how OpenSSL would make sure something like Heartbleed didn’t happen in the future, OpenSSL core team member Ben Laurie, who just happens to work at Google, said no promises could be made. “No one knows how to write completely secure code,” he said, speaking on behalf of OpenSSL. “However, a better job could be done of reducing the risk. For example, code audit, more review of changes. These things take more manpower, which can either come from donated time or donated money.”

Call for Dialogue and Reforms

From Ben Grubb’s article it seems that even OpenSSL agrees that some change is now needed to open source Internet security code. Not surprisingly, their answer is to give them more money, a lot more. According to a NY Times Bits article, OpenSSL has only been able to raise $2,000 per year. Nicole Perlroth, OpenSSL and Linux: A Tale of Two Open-Source Projects (NYT Bits, 4/18/14). Sorry, but that is beyond pathetic. Is a catastrophe really a good fundraising strategy? I think much more fundamental reforms are now required to protect the security of the Internet. Heartbleed has proven that.

I do not have the answers. But I do have a proposal. I call for real dialogues between security experts, and a broad range of other interested parties, to come up with ideas for serious Internet security reform, and then to act on them. This should be completed before the end of this year, 2014.

I suggest that Jim Zemlin, the executive director of the highly successful open source project, Linux, assume at least part of the lead on this, but not take control, and not limit the agenda to open source. The NYT Bits article by Perlroth suggests that Zemlin, and other open source leaders, think that better funding for OpenSSL is all that is needed to fix the problem and reassure the public after the Heartbleed catastrophe.

This entry was posted on Tuesday, April 22nd, 2014 at 2:28 pm and is filed under Lawyers Duties, Security, Technology. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.