The Negligence of One Open Source Software Contributor Put Millions of Internet Users’ Confidential Data At Risk.
Is Open Source Software Appropriate for Major Security Applications?
It was small oversight. Segglemann forgot to add a single line of code limiting the size of memory access to a feature called heartbeat (thus the nickname for the bug, heartbleed). Oops. These things can happen. Easy to understand. Hey, it was, after all, one minute before midnight on New Years eve 2011 when he submitted his work. I kid you not. Segglemann knew that another expert was going to check his work anyway, so why should he be too concerned? Too bad the supervising expert missed the error too. Oops again. Oh well, that’s open source for you. Segglemann did not get paid for his work, and there may be no legal consequences for his gift to the world, a gift that many security experts call the worst thing to ever happen to Internet security.
Bruce Schneier, a leading digital security analyst that I follow, says that “‘Catastrophic’ is the right word. On the scale of one to 10, this is an 11.” Brean, How a programmer’s small error created Heartbleed — a secret back door to supposedly secure sites (National Post, 4/11/14). For more on Schneir’s thoughts on Heartbleed, see the Harvard Business Review interview of him by Scott Berinato.
Rusty Foster wrote in The New Yorker that: “In the worst-case scenario, criminal enterprises, intelligence agencies, and state-sponsored hackers have known about Heartbleed for more than two years, and have used it to systematically access almost everyone’s encrypted data. If this is true, then anyone who does anything on the Internet has likely been affected by the bug.” Forbes cybersecurity columnist Joseph Steinberg wrote, “Some might argue that [Heartbleed] is the worst vulnerability found (at least in terms of its potential impact) since commercial traffic began to flow on the Internet.”
For details on Heartbleed see the Hacker News article by Mohit Kumar, HeartBleed Bug Explained – 10 Most Frequently Asked Questions (Hacker News, 4/14/14). Hacker News includes this good video explanation of the details by Fierce Outlaws:
Bottom line, this little programming error, which some experts refer to as a buffer over-read bug, has huge implications for the security of the Internet, and Android phones, which use the same protocol. It could affect almost every Internet user in the world, depending on who else knew about the mistake, and for how long. At this point, nobody knows. So far only one 19-year old Canadian hacker has been arrested for exploiting the bug, and one online railroad payment system in Russia has discovered it had been hacked. If all that were not bad enough, the security firm Mandiant claims to have found evidence of heartbleed based attacks on one of its client’s virtual private networks (outside of the Internet). Heartbleed vulnerability was used to get pass the firewall and gain access to the VPN.
The Heatbleed catastrophe has dramatically revealed that our current system of Internet security is primarily based on an open source software called OpenSSL. Heartbleed has shown that the whole security of the Internet can depend on one unpaid volunteer like Segglemann, a lone PhD student in Münster, Germany, who had nothing better to do on New year’s eve than finish a freebie software coding project. No doubt he thought it would help his resume. Bad decision.
Something is terribly wrong when the whole Internet is vulnerable due to the mistake of one math student. This mistake should be a wake up call to change the system. I conclude this blog with a call for dialogue among security experts, open source experts, white-hat hackers, lawyers, the FBI, consumer advocates, and others, to come up with serious reforms to our current Internet security infrastructure, including especially reforms of OpenSSL as an organization, and to do so by the end of this year. The public trust in the security of the Internet cannot withstand another Heartbleed, especially if it turns out that thousands, perhaps millions, have been injured. (We already have reports that the hack of the Russian railroad website allowed 10,000 credit card accounts to be stolen.)
Seggelmann claims it was just a mistake. In his words, a trivial error. He seems kind of blasé about it in his only interview to date, a short talk with an Australian journalist. The interview is quoted in full below. In fairness, I do not think Seggelmann realized the implications of his error at the time he spoke. (He has stopped talking now, no doubt on advice of legal counsel, and his current employer, Deutsche Telekom.)
Seggelmann denies that he was paid to do this by the NSA or anyone else. Most of the articles on him written to date just take his word for it. Oddly enough, most writers even express sympathy for Segglemann. The response you see in Huffington Post is typical: “You could blame the author, but he did this work for free, for the community, and with the best of intentions. ” Oh really. How do you know that? Because he said so? I am tempted to say something about naive bleeding heart liberals, but have been accused of being one myself, and besides, it is a bad pun, so I will not.
I hope Robin Seggelmann is telling the truth true too, but have been a lawyer far too long to believe anything a person in his position now says. Plus, the circumstance of posting such important code just a minute before New Years is clearly indicative of carelessness. If the NSA was behind it, and Bloomberg has reported that they have known about the defect for years, then I expect we will know that soon enough from Snowden. If someone else was, we may never know.
Are Seggelmann or OpenSSL Liable for any Damages that Heartbleed May Cause?
Even if this was just a mistake, not fraud, major errors like this have consequences, but for whom? Innocent users of websites operating this code for over two years may already have been victimized. We do not know yet what damages this mistake may cause, but we already have had reports on one arrest in Canada, and one theft of credit card numbers in Russia.
That is just the tip of the iceberg. Who will be responsible for the damages caused to so many? Will it be Seggelmann himself, or perhaps the not-for-profit open source group, OpenSSL, that he did this work for as an unpaid volunteer? Although I have no sympathy for a person whose negligence has caused such havoc, I doubt Seggelmann will ever be forced to reimburse anyone for the harm he has caused.
Perhaps the operators of websites who told their users that their website was secure? Probably not them either, but it may be a closer question. This may be a rare situation where there is no remedy for people damaged by another’s negligence. It will depend on the facts, the as yet unknown details. But, rest assured, much more of the truth will come out in due time. I fully expect some lawyer, somewhere, will file suit when damaged victims appear, or maybe even before.
It will probably be difficult to hold OpenSSL liable for a number of reasons. First or all, who or what is OpenSSL? It appears to be a type of legal entity that we would call in the U.S. an unincorporated association. It is often treated something like a partnership in U.S. law.
According to the Washington Post, OpenSSL‘s headquarters — to the extent one exists at all — is the home the group’s only employee, a part timer at that, located on Sugarloaf Mountain, Maryland. He lives and works amid racks of servers and an industrial-grade Internet connection. Craig Timberg, Heartbleed bug puts the chaotic nature of the Internet under the magnifying glass (Washington Post, 4/9/14).
You cannot make up stuff like this. Truth is always stranger than fiction. Tineberg’s article also reports that the software that serves as the backbone for security on the Internet has, due to the lack of personnel and funds of OpenSSL, never been though a security audit, a meticulous process that involves testing the software for vulnerabilities.
Here is what OpenSSL has to say about themselves:
The OpenSSL Project is a collaborative effort to develop a robust, commercial-grade, full-featured, and Open Source toolkit implementing the Secure Sockets Layer … managed by a worldwide community of volunteers that use the Internet to communicate, plan, and develop the OpenSSL toolkit and its related documentation. . . . [Y]ou are free to get and use it for commercial and non-commercial purposes. . . .
The OpenSSL project is volunteer-driven. We do not have any specific requirement for volunteers other than a strong willingness to really contribute while following the projects goal. The OpenSSL project is formed by a development team, which consists of the current active developers and other major contributors. Additionally a subset of the developers form the OpenSSL core team which globally manages the OpenSSL project.
This is the way most open source software works. As open source software goes, OpenSSL is one of the most successful in the world (well, it was, until this whole catastrophe thing). Their product, OpenSSL, was, and still is, the world’s most popular open source cryptographic library. It is used to encrypt most of the traffic on the Internet. About two-thirds of web sites with an “S” at the end of the HTTP address use this freebie software.
Seggelmann had an accomplice of sorts at OpenSSL, although I do not mean to imply any type of conspiracy by use of this word. There is no evidence of that. But I am sure people will look into that possibility, not only government investigators, but also private eyes, especially if and when they are motivated by the kind of mixed greed and fear incentives that only law suits can bring. The appearances now all suggest that the double-checker just happened to miss the trivial error too. OpenSSL, like most good open source projects, has quality control procedures. Proposed code contributions are double checked for mistakes by a senior contributor to OpenSSL before they are accepted.
In this case Seggelmann’s work was checked by Stephen Henson. He is a freelance crypto consultant (his words) based in the U.K. who has a PhD in Mathematics. He is still listed by OpenSSL as one of only four core team members of this open source group. That’s right, four people; one of them the part-time employee who works out his home on a mountain that serves as the group’s headquarters.
So after two people looked over the new code contribution, way too casually as we now know, the code was approved. Soon thereafter millions of websites started using it and so made themselves susceptible to attack.
I could only find one legal disclaimer in the OpenSSL website, but I bet this changes soon as the academics in charge of this non-profit association start to wake up to legal realities:
THIS AREA IS PROVIDED BY THE OPENSSL PROJECT “AS IS” AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OPENSSL PROJECT OR ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
I suspect the enforceability of this language may get tested in some court somewhere in the world, probably in the U.S., but to what end? I doubt OpenSSL has any assets, much less insurance, and, even if you could prove proximate causation, how deep could the pockets of Segglemann, Henson, and other contributors be? It is more likely the primary targets for restitution will be the companies who used the defective open source software in their servers, thus exposing their users’ confidential information.
Was it negligence for commercial sites to rely on Open Source software?
Was it negligence for commercial sites to rely on Open Source software? Free software donated by all-too-human expertse? I do not think so. I will not go that far, but I’m willing to bet some lawyer somewhere will go that far. With a mistake this size it is almost inevitable that a class action suit will eventually get filed against somebody.
With the facts I have seen to date I do not think there was adequate notice to the adopters of this free software as to its unreliability to support a cause of action against them for negligent use of OpenSSL. But can the same be said now after the Heartbleed disaster has come to light? Now that we know the mistakes of only two men can put everyone at risk? Maybe not.
Is this the Beginning of the End for Open Source?
This may spell the beginning of the end of widespread commercial adoption of free software, at least for security purposes. After all, it took a for-profit company, Google, to discover the error in the software that many of its websites were using too. What we do not know is how many hackers, government sponsored or free lance, had previously discovered this mistake. How many had previously exploited this flaw to discover the supposedly secret passwords of the hundreds of millions persons potentially impacted. What makes this even worse is that we will never know. The biggest programming error in history made it possible for hackers to steal your data without ever leaving a trace.
Many believe the NSA has been exploiting this flaw for years. Who knows what criminal enterprises and foreign governments may have done the same?
There are two rational responses to this open source security scandal. One, stop using the Internet for anything that you want to keep secure, like all financial information. Or two, stop using Open Source, and instead use paid software, software with real safeguards, and with an entity or entities who will stand behind their products, and insurers who will stand behind them.
Society today relies heavily on the Internet. Commerce relies heavily on the Internet. If security is at risk, our current way of life is at risk. It is that important. So the first alternative is out.
This means we have to stop reliance on Open Source software for security, at least the way it is run now. We need the safety of big corporations who will have a direct economic incentive to take responsibility for their work. We need paid employees, not volunteers. Ones who will get paid bonuses for doing great work, and fired if the make Heartbleed type errors that put us all at risk. Either that, or we need major reform of this open source non-profit so that they are accountable. We are way beyond the hobbyist beginnings of the Internet, a time I remember well, and yet we still delegate major Internet responsibilities to small, unregulated groups of independent associations.
The Heartbleed disaster shows that reliance on open source software for commerce is a risky proposition when it comes to security. It may save users some money, but the risk of error may be too high. Consumers will demand that companies pay up and protect their personal data security. As Chris Williams put it in his article for The Register, OpenSSL Heartbleed: Bloody nose for open-source bleeding hearts:
Open source or open sores? The crux of the matter is that OpenSSL is used by millions and millions of people, even if they don’t know it, but this vital encryption software – used to secure online shopping and banking, mobile apps, VPNs and much more – has a core developer team of just four volunteers who rely on donations and sponsorship. The library code is free and open source, and is used in countless products and programs, but Seggelmann and others point out that the project receives little help.
The use of open source software for everything was a fine experiment, an idealistic one based on the notion that crowdsourcing provided a better alternative to free enterprise, that capitalism could be replaced by a volunteer society of dedicated altruists. Personally, I was always skeptical. I think that competition is a good thing and helps build better products. Heartbleed confirms the skepticism was warranted. Heartbleed has exposed the dark side of crowdsourcing, the inherent weaknesses of volunteerism. The dark side of crowdsourcing is that the crowds will not come, or will stop coming. Here the crowd that checked a critical update to the code consisted of two people only, Robin Seggelmann and Stephen Henson. Two is never a crowd. In fact, a jury may some day be called upon to decide whether it was reasonable to release security code after only two people looked at it.
Robin Seggelmann’s Side of the Story
Ben Grubb is the only journalist so far to get an interview of Robin Seggelmann, published in Man who introduced serious ‘Heartbleed’ security flaw denies he inserted it deliberately (Sydney Morning Herald, 4/11/14). Here are the key excerpts and quotes from Grubb’s article, but I suggest you read the entire article and Grubb’s followup articles too. He has an interesting perspective, including criticism of Google’s handling of the release of information of the bug’s discovery.
Dr. Seggelmann, of Münster in Germany, said the bug which introduced the flaw was “unfortunately” missed by him and a reviewer when it was introduced into the open source OpenSSL encryption protocol over two years ago. “I was working on improving OpenSSL and submitted numerous bug fixes and added new features,” he said. “In one of the new features, unfortunately, I missed validating a variable containing a length.” After he submitted the code, a reviewer “apparently also didn’t notice the missing validation”, Dr Seggelmann said, “so the error made its way from the development branch into the released version.” Logs show that reviewer was Dr Stephen Henson. Dr Seggelmann said the error he introduced was “quite trivial”, but acknowledged that its impact was “severe”.
Conspiracy theories. A number of conspiracy theorists have speculated the bug was inserted maliciously. Dr Seggelmann said it was “tempting” to assume this, especially after the disclosure by Edward Snowden of the spying activities conducted by the US National Security Agency and others. “But in this case, it was a simple programming error in a new feature, which unfortunately occurred in a security relevant area,” he said. “It was not intended at all, especially since I have previously fixed OpenSSL bugs myself, and was trying to contribute to the project.” Despite denying he put the bug into the code intentionally, he said it was entirely possible intelligence agencies had been making use of it over the past two years. “It is a possibility, and it’s always better to assume the worst than best case in security matters, but since I didn’t know [about] the bug until it was released and [I am] not affiliated with any agency, I can only speculate.”
Benefits of discovery If anything had been demonstrated by the discovery of the bug, Dr Seggelmann said it was awareness that more contributors were needed to keep an eye over code in open source software. “It’s unfortunate that it’s used by millions of people, but only very few actually contribute to it,” he said. “The benefit of open source software is that anyone can review the code in the first place. “The more people look at it, the better, especially with a software like OpenSSL.”
Future Heartbleed prevention. Asked how OpenSSL would make sure something like Heartbleed didn’t happen in the future, OpenSSL core team member Ben Laurie, who just happens to work at Google, said no promises could be made. “No one knows how to write completely secure code,” he said, speaking on behalf of OpenSSL. “However, a better job could be done of reducing the risk. For example, code audit, more review of changes. These things take more manpower, which can either come from donated time or donated money.”
Call for Dialogue and Reforms
From Ben Grubb’s article it seems that even OpenSSL agrees that some change is now needed to open source Internet security code. Not surprisingly, their answer is to give them more money, a lot more. According to a NY Times Bits article, OpenSSL has only been able to raise $2,000 per year. Nicole Perlroth, OpenSSL and Linux: A Tale of Two Open-Source Projects (NYT Bits, 4/18/14). Sorry, but that is beyond pathetic. Is a catastrophe really a good fundraising strategy? I think much more fundamental reforms are now required to protect the security of the Internet. Heartbleed has proven that.
I do not have the answers. But I do have a proposal. I call for real dialogues between security experts, and a broad range of other interested parties, to come up with ideas for serious Internet security reform, and then to act on them. This should be completed before the end of this year, 2014.
I think they are wrong about this. The public does not care at all about the survival of open source. All they care about is the survival and security of the Internet. After all, their bank account and refrigerator are connected to the Internet today; tomorrow it could be their pacemaker. It is a key part of their life. They do not care if Microsoft or other companies profit from keeping it secure. They want their personal data secure from criminals. They do not want their bank accounts drained or their identity stolen. They want security. They want insurance.
I hope that Zemlin and other outsourcing leaders get this, and will consider other, deeper reforms than better open source fund raising. The input of security experts, ones not tied to the open source movement, including its commercial competitors, should be considered. This is not an open source problem, this is a security problem. Opponents to the open source movement should also be invited so all sides can be heard, so too should open source neutrals and outsiders, which, for the record, is my position. I am not an open source fanboy, but, on the other hand, I do use open source software, WordPress, for my blog and most websites. Others who should be invited to the conference include all shades of security experts, white-hat hackers, lawyers, consumer advocates, and others. Even the government, including the FBI and NSA. They should all be invited to dialogue and come up with serious reforms to our current Internet security infrastructure, including especially reforms of OpenSSL, and to do so by the end of this year.
Like it or not, the views of law and lawyers must be considered. Lawsuits are not the answer. But still, they will come. Proposed reforms should take legal consequences into consideration. Real people, innocent people, may already have been harmed by these security errors. It will take years to find out what damages have been caused by OpenSSL‘s major blooper. Some courts may find that they are entitled to restitution.
The Internet is not a no-mans-land of irresponsibility. It has laws and is subject to laws. I first pointed that out in my 1996 book for MacMillan, Your Cyber Rights and Responsibilities: The Law of the Internet, Chapter 3 of Que’s Special Edition Using the Internet. Persons committing crimes on the Internet must and will be prosecuted no matter where their bodies are located. The same goes for negligent actors, be they human, corporate, or robot. Responsibility for our actions must always be considered in any human endeavor, even online. Not-for-profit status is not a get out of jail free card. That is one reason why lawyers must have a seat at the table and participate in the Internet security dialogue. Law and cyber liability issues must be considered.
From my perspective as a lawyer I expect that any real reform of Internet security will include the development of new rules. They will likely be focused on mandatory procedures to safeguard quality. The rules will try to prevent the reoccurrence of another major screwup like Heartbleed. For instance, if there is no bona fide crowd sourcing, say a minimum of 10 to 20 experts reviewing each line of code, not just two, then other safeguards should be required. In that event, perhaps deep-pocket corporations should be hired to audit everything. They should be made to vouch for the code, to stand behind it.
All alternatives should be considered, not just better fundraising and publicity for OpenSSL. (Frankly, I think it is too late for publicity to ever help OpenSSL.) Maybe private enterprise should take over OpenSSL, at least in part? Or maybe some kind of quasi-governmental entity should get involved in Internet security. For example, maybe it should be a part of ICANN’s duties?
Maybe private or public insurance should be required for any software like this, and so spread the risk among all users. Although this may offend open source fanatics, but the reality is, as Heartbleed proves, Free is not necessarily a good thing when you are looking for quality. Perhaps providers should pay for at least part of all Open Source. Most are, after all, profiting from it in one way or another. Although I hate to say it, since most politicians are technically clueless, perhaps new laws should also be considered? Laws that place incentives for quality, that impose both carrot and stick consequences. I would put everything on the table for discussion. More of the same is too risky.
I invite this dialogue to begin here and now. Email me or leave a comment below. If that dialogue is already happening elsewhere, please let me know. In any event, feel free to forward this call for dialogue. I will report on it all here, no matter where and how it occurs, so long as it is real dialogue, people really listening to what each other have to say, and not just posturing and win/lose debate.
If this happens, I will report on the parts that I can understand, the aspects that are not overly technical, and aspects that are somewhat legal in nature. If someone or organization wants to volunteer to convene a Congress to conclude the dialogue and facilitate consensus decisions, then I will assist in publicity and report on that too. I will also be happy to attend, if at all possible. If I have anything to say on issues, I will also do that, and not just report. But for now, aside from the few general suggestions already provided here, my message at this time is to sound an alarm on the need to take action, and to suggest that the action be preceded by dialogue. I would like to know what you think about all of this?
[…] via HEARTBLEED: A Lawyer’s Perspective On Cyber Liability and the Biggest Programming Error in History… […]
[…] I could about it. My research is ongoing, but I have already published a detailed report on my personal blog. I have also been pondering policy changes, and changes in the laws governing the Internet that be […]
[…] was going to write a nice, calm post today when I came across Ralph Losey’s piece on the Heartbleed bug. It is a long piece and you can tell it was written by a lawyer. I have nothing against lawyers as […]
[…] resolution. He wants open debate, dialogue. And he’s getting it. In his most recent post, HEARTBLEED: A Lawyer’s Perspective On Cyber Liability and the Biggest Programming Error in History, he provides (as usual) a detailed perspective. Which was quickly countered by Laurence Hart in his […]
[…] As we used to say in the Sixties, my consciousness has been raised recently by a new book on cybersecurity: Cybersecurity and Cyberwar: What everyone needs to know by P.W. Singer and Allan Friedman (Oxford, 2014). I recommend this as a must read on the big picture of cybersecurity. We are living in a dangerous world where cybersecurity is a constant threat. The latest Hearbleed security flaw scandal is a good example of this. HEARTBLEED: A Lawyer’s Perspective On Cyber Liability and the Biggest Programming Error in Histor…. […]
This is a pretty silly article. The problem isn’t that a programmer made a mistake in a free, open source piece of code. The problem is so many for-profit companies, companies like Cisco, use free, open source security software in their products. Don’t you find that incredible?
As for the biggest programming mistake ever, I’d take Y2K in sheer size and cost to remediate over this.
Y2K was not an error made by one specific programmer. It was more of a historical mistake by an industry arising out of limited space when programming first started. Heartbleed was an error by one person who submitted a patch to open source a minute before New Years. Two thirds of Internet sites were compromised for over two years. I am sure the Chinese thought it was very silly indeed. NSA did too, I’m sure, but the criminal gangs and Chinese are the ones that worry me much more.
[…] appeals to commentators who believe that OpenSSL contributors, as unpaid volunteers, are simply under-incentivized to check for Heartbleed-like errors – an allegedly grueling and monotonous task, which earns them […]
[…] Step 4: Conduct a Risk Assessment. You next analyze the operational environment of your computer systems and evaluate the likelihood of attacks of various components. This includes a through consideration the impact that a successful intrusion could have. Here it is important to include emerging risks and threats, and the vulnerability of your data. For instance, any risk assessments going on right now must include consideration of the OpenSSL Heartbleed vulnerability. See eg. HEARTBLEED: A Lawyer’s Perspective On Cyber Liability and the Biggest Programming Error in Histor…. […]
Okay, so you have a decent starting portfolio of ‘glamour photography’ shots you’ve taken and assembled to show the possibilities of your range and depth of photographic artistry.
Laughter, smiles and playfulness were the norm for his photos, but it was not in fashion photography.
You will be excited watching your kids in pastel
tones especially when summer comes.