RSA-CRT Leak Report for January/February 2016
Florian WeimerIn January and February 2016, I continued to run RSA-CRT key leak detectors.
Crawler status
I continued to run the crawler throughout most of January and February, importing new Censys
dumps as they became available. I also analyzed the Censys dumps for RSA-CRT key leaks by replaying the TLS handshakes which the Censys infrastructure marked as failed.
The main instance continued to run on a container hosting platform. It was my first choice because it was dirt-cheap, provided faster-than-Fast Ethernet networking, and sufficient RAM. Gigabit network speed is interesting for importing the Censys dumps, to extract IP addresses and scan for RSA-CRT leaks, and the container environment has excellent single-TCP stream network performance, typically downloading Censys dumps at 700 Mbps from the Google storage service the Censys project uses (so that JSON processing became the bottleneck).
In February, I set up additional crawler instances on fully virtualized hosting environments. One instance proved extremely helpful for validating the crawler performance. The container environment I used before did not support running the perf tool. Basic performance monitoring from within the container indicated that most of the time was spent in the kernel, but it was hard to tell whether this was accurate, or just the container environment telling me that the system was oversubscribed. Being able to use perf showed me that the crawler was not CPU-bound at all (neither in kernel nor userspace), and was waiting on the network most of the time.
In both the virtualized environment I used for my perf measurements, and the original container environment, the crawler was not able to match the network bandwidth which the Censys downloads showed was avaiable to the system. I suspect that due to the many TCP connections it creates, the crawler is not suited well to either hosting environment. (Although the second hosting provider was quite helpful to make sure the crawler was able to run on their infrastructure.)
A second virtualized hosting provider fairly quickly blocked access to the server. This also happened to the main crawler instance towards the end of February. In both cases, it was difficult to get a full rationale for their decision to block the server, and they remain blocked. Surprisingly, neither provider canceled the contract, or suggested alternative products which would better cope with the network load (with higher monthly charges, obviously).
The challenges in finding a suitable hosting environment look like a topic for a separate technical report. This experience could be relevant those who try to operate emerging applications which do not match the network traffic patterns supported by existing hosting environments.
Dell/Sonicwall software update
Dell (under the Sonicwall brand) released a software update for their Network Security Appliance product line which address RSA-CRT private key leaks:
This is the leak attributed to a “network security appliance” in the September/October 2015 report.
Devices of this kind have been identified as responsible for multiple key leaks. One affected a certificate in the browser PKI, which has since been revoked.
ZTE software update
ZTE provides updated software for their ZXSEC US products to address RSA-CRT key leaks:
This is the security gateway mentioned in the November/December 2015 report. Since then, another leak from a different device has been observed.
Noteworthy key leaks
I already mentioned that a Dell Sonicwall Network Security Appliance leaked its private key, which belonged to a X.509 certificate in the browser PKI.
A leak from a LANCOM 1811n Wireless device was observed. LANCOM indicated that they addressed this issue silently in firmware version 8.84 (which appears to have been relased in 2014).
There was a single leak from a different Viprinet device. Viprinet told me they had released fixed software to their customers, and it is likely that the customer simply did not apply this update. However, this second leak shows that the vulnerability was not something that happened to just a single piece of hardware, due to some unique physical property. (The load balancer leak mentioned in the September/October 2015 report still did not reoccur, by the way.)
A third German security device vendor was tentatively identified as the maker of a device which apparently acted as a TLS-terminating reverse proxy. This key leak is still under investigation. It is noteworthy because the X.509 certificate whose private key leaked is part of the browser PKI.
Why Germany?
One has to wonder what is up with the Germans and their cryptography. The first key leak I ever encountered (in January 2015, described in the original report) was from a German web site, and the report lists another German TLS implementation which was affected (in addition to Viprinet discussed above).
This could be the result of conducting the majority of the TLS handshakes from crawlers in Germany, and a timing-related preference for nearby hosts, similar to the The case of the 500-mile email (Trey Harris, November 2002, retrieved 2016-03-06). I hope to address issue in a subsequent report.
Revisions
2016-03-06: published