A well-known Google security researcher discovered that Cloudflare was exposing chat messages, encryption keys, cookies, password manager data, hotel bookings and more. The content delivery network quickly confirmed the finding, traced it to a coding error involving just a single wrong character and put related remediations in place.
But the leaked data had been cached by major search engines, and the discovery triggered a frantic effort to remove the cached data before the flaw was publicized. Much of the exposed data would have normally been protected by SSL/TLS, but the nature of the vulnerability caused it to be exposed to the internet in unencrypted form.
It’s unknown how much data may have been leaked, which may make it difficult for companies and users to decide what their most prudent reaction to this bug report should be.
Cloudflare specializes in improving the performance and redundancy of websites, as well as offering protection against attacks such as distributed denial-of-service. The discovery shows how a weak link in just a single widely used cloud service can have a vast impact on data security downstream.
The sensitive data was exposed for “months,” writes Google’s Tavis Ormandy, a researcher with the company’s Project Zero, who found the bug. He jokingly dubbed it Cloudbleed, a portmanteau that recalls the Heartbleed OpenSSL vulnerability.
Cloudflare has not released a list of affected domains. But Nick Sweeting, the co-founder and CTO of Blitzka Software, has created a list of 4.3 million websites that use Cloudlfare, and he aims to eventually narrow the list to only display sites left at risk by the coding error.
So far, Ormandy has found data on the web from Uber, 1Password, FitBit and OKCupid. 1Password, a widely used password manager, says the data that was exposed was encrypted in two other ways, thus making the Cloudflare bug of little consequence for its users.
Cloudflare have been leaking customer HTTPS sessions for months. Uber, 1Password, FitBit, OKCupid, etc. https://t.co/wjwE4M3Pbk
— Tavis Ormandy (@taviso) February 23, 2017
“At the moment, we want to ensure and remind everyone that we designed 1Password with the expectation that SSL/TLS can fail,” Jeffrey Goldberg, a security architect with AgileBits – which develops 1Password – writes in a blog post.
Cloudflare, in a detailed postmortem of what went wrong, says the error amounted to the inadvertent use of a single character – “>” instead of “=” – that caused a buffer overrun, which lead to the memory leak spotted by Ormandy.
Cloudflare accidentally introduced the bug into an HTML parser, John Graham-Cumming, the company’s CTO, says in a blog post. The HTML parser modifies web pages that come through Cloudflare’s edge servers for security reasons, such as obscuring email addresses from known scraping bots.
So far, it does not appear the bug was discovered by anyone else or exploited, Graham-Cumming says. Although SSL keys used by Cloudflare’s customers were not leaked, the company itself has had a private key exposed as a result of the breach, as well as “a small number of secrets used internally at Cloudflare for authentication,” Graham-Cumming says.
Cloudflare began using the private key to encrypt connections between its own machines, following former National Security Agency contractor Edward Snowden’s leaks describing global government surveillance.
Other information that may have been disclosed includes cookies and authentication information, such as API keys and OAuth tokens.
Cloudflare customers were most impacted from Feb. 13 to Feb. 18, Graham-Cumming writes, noting that one in 3.3 million HTTP requests running through Cloudflare potentially resulted in a memory leak.
Incident report on memory leak caused by Cloudflare parser bug – https://t.co/rTZ4bFw3uJ
— Cloudflare (@Cloudflare) February 23, 2017
Funny Data, Then Whoa
Ormandy’s first indication something was wrong came Feb. 17 as he was working on a project analyzing large public data sets. Some data he encountered was at first a mystery, he writes, but he notes that it isn’t uncommon to come across malformed or garbage data.
“We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major Cloudflare-hosted sites from other users,” he writes. “Once we understood what we were seeing and the implications, we immediately stopped and contacted Cloudflare security.”
It was an unusual situation – and one that posed a great threat to personally identifiable information. “Seconds mattered here, emails to support on a Friday evening were not going to cut it,” he writes.
Ormandy first reached Cloudflare via Twitter. As anyone in the security industry knows, if Ormandy wants to reach you urgently, something bad has happened. But fixing the damage involved much more than simply fixing the bug itself.
By Feb. 21, with Cloudflare attempting to scrub spilled data that had been indexed by search engines such as Google, Yahoo and Bing, Ormandy says he was still discovering new sensitive data that had been leaked as a result of the coding error.
“I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings,” he said. “We’re talking full HTTPS requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.”
Now, however, Cloudflare says it’s confident that there’s no more cached data lurking. “We are disclosing this problem now as we are satisfied that search engine caches have now been cleared of sensitive information,” Graham-Cumming writes.
All told, Cloudflare reports that it discovered 770 unique uniform resource identifiers, or URIs, that contained data from the memory leak, affecting 161 domains. All search engines have now removed the content, Graham-Cumming says.
Ormandy, however, has taken issue with the tone of Cloudflare’s blog post. While he says it’s an excellent postmortem, he believes that it “severely downplays the risk to customers.”