DEEP PACKET SURVEILLANCE —

Deep packet inspection device purged of flaw reported by TOR users (Updated)

Security devices have been updated after warnings they could expose TOR traffic.

Dan Goodin - Jul 9, 2012 10:20 pm UTC

Cyberoam, a maker of appliances designed to secure sensitive networks, said it has issued an update to fix a flaw that was reported by members of TOR anonymity network.

Cyberoam issued the hotfix on Monday to a variety of its unified threat management tools. The devices, which are used to inspect individual packets entering or exiting an organization's network, previously used the same cryptographic certificate. Researchers with the TOR network recently reported the flaw and said it caused a user to seek a fake certificate for thetorproject.org when one of the DPI (or deep packet inspection) devices was being used to monitor his connection.

"Examination of a certificate chain generated by a Cyberoam DPI device shows that all such devices share the same CA certificate and hence the same private key," TOR researcher Runa A. Sandvik wrote in a blog post published last Tuesday. "It is therefore possible to intercept traffic from any victim of a Cyberoam device with any other Cyberoam device—or to extract the key from the device and import it into other DPI devices, and use those for interception." Someone commenting on the post went on to publish the purported private key used by the Cyberoam certificate.

Monday's post announcing the Cyberoam hotfix suggested competing DPI devices may contain the same vulnerability. "We, at Cyberoam, do understand the critical nature of this issue though we have been singled out and have been put into a situation that requires us to react urgently, keeping our customers' best interest in mind," the post stated. "We think that the industry needs to react to this on an urgent basis so that a deeper crisis is averted." The post didn't name specific devices or manufacturers who may also be vulnerable.

In a post published last week, Cyberoam officials said: "Cyberoam UTM either accepts or rejects, but does not store HTTPS Deep Scan Inspection data, as processing is done in real-time. The possibility of data interception between any two Cyberoam appliances is hence nullified."

Monday's over-the-air fix forcefully generates a unique certificate on each UTM appliance. Customers will know the device has been successfully updated when it displays a "positive alert."

Article updated on July 17, 2012, to correct details about the vulnerability.

Promoted Comments

Rabbit0864 | Smack-Fu Master, in training
jump to post

cdillon wrote:
rmcrowley2000 wrote:
Max_Patlick wrote:
I don't understand, I thought the idea of deep packet inspection was to catch/stop stuff like TOR.

This. I'm rather confused about this is well; it seems more like this is a weakness of TOR? Some clarification on this would be helpful.

The only way a DPI appliance is able to inspect the cleartext of SSL traffic is by terminating the SSL session (and thus decrypting the data) from the server to the appliance and then re-encrypting the data in a brand-new SSL session between the appliance and the end-user. Due to the nature of SSL, the new SSL session that is created between the appliance and the end-user cannot use the same SSL certificate that the original server used, it must instead use its own "faked" certificate with its own private key. Normally this is going to throw up an error on the client side, because this is literally a "man-in-the-middle attack", unless the client has been pre-configured to trust the DPI appliance's certificate, which is usually the case in an enterprise environment where most of these DPI appliances are getting used.

The problem here is that all Cyberroam appliances were using the SAME "FAKE" CERTIFICATE shipped from the factory. This means anybody who gets their hands on a Cyberroam appliance (or even just a firmware image) has access to the same private key that is encrypting every other Cyberoam customer's data that has been inspected and re-encrypted by the DPI appliance. It would be a very simple matter to sniff all of that re-encrypted data and decrypt it because you've got the private key. What a stupid, bone-headed mistake by Cyberoam. I'm curious now what other vendors have made the same mistake.

Rule #1 for Security Appliance Self-Security -- Don't ship your shit with the same private keys or passwords as ANY OTHER SYSTEM YOU ARE SHIPPING! Ship it with unique keys and passwords across the board. Unfortunately, VERY few vendors get this right. One of the few examples of "doing it right" that I see every day is with HP iLO, where every server is shipped with a unique default iLO password that is printed on a removable tag attached to the server, and unique SSL and SSH keys are generated on first-use. I wish every appliance took this approach.

Edit: Also, you DO NOT need to have another Cyberoam appliance to take advantage of this vulnerability. The simple fact of the matter is you've got a whole mess of SSL encrypted data leaving the DPI appliance encrypted with ONE private key. If you have your hands on that private key then you can decrypt all of that SSL data with your tool of choice.

The major problem with what you describe is that the easily decrypted data will only be found on the LAN side of the device. Sniffing would not be possible outside of the LAN as that uses the original cert of the site / service visited (ie on the internet side) and would require either physical access or some kind of malware infecting one of the PCs on the LAN.
Last edited by Rabbit0864 on Mon Jul 09, 2012 5:41 pm

79 posts | registered Jun 18, 2009
cdillon | Smack-Fu Master, in training
jump to post

Rabbit0864 wrote:
The major problem with what you describe is that the easily decrypted data will only be found on the LAN side of the device. Sniffing would not be possible outside of the LAN as that uses the original cert of the site / service visited (ie on the internet side) and would require either physical access or some kind of malware infecting one of the PCs on the LAN.

Correct. And that IS a major problem for some people. Any malicious user on the LAN side could decrypt every other LAN user's SSL traffic which was being terminated by the DPI appliance. Hey, let's read the boss's emails! Corporate espionage, anyone? Also, as you mentioned, malware... the malicious user wouldn't even have to be on the LAN as long as he could infect a machine on the LAN with some packet-sniffing malware.

26 posts | registered Jul 18, 2010
KpaBap | Ars Scholae Palatinae
jump to post

This is a really amateur mistake to have made for what's supposedly a seasoned security company and I'm surprised it has taken this long for anyone to blow the whistle on the SSL private key issue. People with the know-how and influence in purchasing decisions for the types of appliances Cyberoam sells should likely consider what other poor design choices were made.

1073 posts | registered May 6, 2001
KpaBap | Ars Scholae Palatinae
jump to post

Devin wrote:
How can the Cyberoam device decrypt the SSL contents if the client initiated the session? Wouldn't the device have to be snooping on the outbound connection as well to figure out how to decrypt it?

I'm not much of a SSL guy but it seems that if cryptography is working properly the only two devices with the key to decrypt the traffic would be the endpoints.

Web servers (and other publicly accessible applications) are not immune from man in the middle attacks, which, as mentioned previously is exactly what these and many similar appliances do.

Very simply put, SSL currently (with a man in the middle) works like this:

1) The client (your web browser or other application) asks the server for a certificate
2) The server responds with its certificate and the client establishes an SSL session with it - in this case, the "server" is actually the "man in the middle" appliance in question. Since the client has no choice but to either accept the certificate, or close the connection, at this point the appliance has the ability to decrypt communications intended for the public web server.

Normally, this would present a browser warning about an untrusted certificate. However, in corporate environments, the man in the middle certificate is usually installed as trusted on all employee workstations.

3) The client encrypts data sent to the server with the man in the middle's certificate
4) The man in the middle establishes a separate SSL connection to the intended web server
5) The man in the middle decrypts the data from the web server, and re-encrypts in the separate SSL session it has with the client

The "servers" can decrypt the messages sent to them by using their private keys which as their name implies, should be kept private. If either of the "server" private keys are compromised, then the data can be decrypted for that particular SSL session.

SSL does provide for client authentication on the server side. However, this would involve every person that wants to use any site or application having to send in their personal public key to the site/application in question. I wouldn't hold my breath for that, just by the sheer logistics and horrible privacy concerns involved in positively identifying every web user.

1073 posts | registered May 6, 2001

Promoted Comments

Rabbit0864 | Smack-Fu Master, in training
jump to post

cdillon wrote:
rmcrowley2000 wrote:
Max_Patlick wrote:
I don't understand, I thought the idea of deep packet inspection was to catch/stop stuff like TOR.

This. I'm rather confused about this is well; it seems more like this is a weakness of TOR? Some clarification on this would be helpful.

The only way a DPI appliance is able to inspect the cleartext of SSL traffic is by terminating the SSL session (and thus decrypting the data) from the server to the appliance and then re-encrypting the data in a brand-new SSL session between the appliance and the end-user. Due to the nature of SSL, the new SSL session that is created between the appliance and the end-user cannot use the same SSL certificate that the original server used, it must instead use its own "faked" certificate with its own private key. Normally this is going to throw up an error on the client side, because this is literally a "man-in-the-middle attack", unless the client has been pre-configured to trust the DPI appliance's certificate, which is usually the case in an enterprise environment where most of these DPI appliances are getting used.

The problem here is that all Cyberroam appliances were using the SAME "FAKE" CERTIFICATE shipped from the factory. This means anybody who gets their hands on a Cyberroam appliance (or even just a firmware image) has access to the same private key that is encrypting every other Cyberoam customer's data that has been inspected and re-encrypted by the DPI appliance. It would be a very simple matter to sniff all of that re-encrypted data and decrypt it because you've got the private key. What a stupid, bone-headed mistake by Cyberoam. I'm curious now what other vendors have made the same mistake.

Rule #1 for Security Appliance Self-Security -- Don't ship your shit with the same private keys or passwords as ANY OTHER SYSTEM YOU ARE SHIPPING! Ship it with unique keys and passwords across the board. Unfortunately, VERY few vendors get this right. One of the few examples of "doing it right" that I see every day is with HP iLO, where every server is shipped with a unique default iLO password that is printed on a removable tag attached to the server, and unique SSL and SSH keys are generated on first-use. I wish every appliance took this approach.

Edit: Also, you DO NOT need to have another Cyberoam appliance to take advantage of this vulnerability. The simple fact of the matter is you've got a whole mess of SSL encrypted data leaving the DPI appliance encrypted with ONE private key. If you have your hands on that private key then you can decrypt all of that SSL data with your tool of choice.

The major problem with what you describe is that the easily decrypted data will only be found on the LAN side of the device. Sniffing would not be possible outside of the LAN as that uses the original cert of the site / service visited (ie on the internet side) and would require either physical access or some kind of malware infecting one of the PCs on the LAN.
Last edited by Rabbit0864 on Mon Jul 09, 2012 5:41 pm

79 posts | registered Jun 18, 2009
cdillon | Smack-Fu Master, in training
jump to post

Rabbit0864 wrote:
The major problem with what you describe is that the easily decrypted data will only be found on the LAN side of the device. Sniffing would not be possible outside of the LAN as that uses the original cert of the site / service visited (ie on the internet side) and would require either physical access or some kind of malware infecting one of the PCs on the LAN.

Correct. And that IS a major problem for some people. Any malicious user on the LAN side could decrypt every other LAN user's SSL traffic which was being terminated by the DPI appliance. Hey, let's read the boss's emails! Corporate espionage, anyone? Also, as you mentioned, malware... the malicious user wouldn't even have to be on the LAN as long as he could infect a machine on the LAN with some packet-sniffing malware.

26 posts | registered Jul 18, 2010
KpaBap | Ars Scholae Palatinae
jump to post

This is a really amateur mistake to have made for what's supposedly a seasoned security company and I'm surprised it has taken this long for anyone to blow the whistle on the SSL private key issue. People with the know-how and influence in purchasing decisions for the types of appliances Cyberoam sells should likely consider what other poor design choices were made.

1073 posts | registered May 6, 2001
KpaBap | Ars Scholae Palatinae
jump to post

Devin wrote:
How can the Cyberoam device decrypt the SSL contents if the client initiated the session? Wouldn't the device have to be snooping on the outbound connection as well to figure out how to decrypt it?

I'm not much of a SSL guy but it seems that if cryptography is working properly the only two devices with the key to decrypt the traffic would be the endpoints.

Web servers (and other publicly accessible applications) are not immune from man in the middle attacks, which, as mentioned previously is exactly what these and many similar appliances do.

Very simply put, SSL currently (with a man in the middle) works like this:

1) The client (your web browser or other application) asks the server for a certificate
2) The server responds with its certificate and the client establishes an SSL session with it - in this case, the "server" is actually the "man in the middle" appliance in question. Since the client has no choice but to either accept the certificate, or close the connection, at this point the appliance has the ability to decrypt communications intended for the public web server.

Normally, this would present a browser warning about an untrusted certificate. However, in corporate environments, the man in the middle certificate is usually installed as trusted on all employee workstations.

3) The client encrypts data sent to the server with the man in the middle's certificate
4) The man in the middle establishes a separate SSL connection to the intended web server
5) The man in the middle decrypts the data from the web server, and re-encrypts in the separate SSL session it has with the client

The "servers" can decrypt the messages sent to them by using their private keys which as their name implies, should be kept private. If either of the "server" private keys are compromised, then the data can be decrypted for that particular SSL session.

SSL does provide for client authentication on the server side. However, this would involve every person that wants to use any site or application having to send in their personal public key to the site/application in question. I wouldn't hold my breath for that, just by the sheer logistics and horrible privacy concerns involved in positively identifying every web user.

1073 posts | registered May 6, 2001

Dan Goodin Dan Goodin is Senior Security Editor at Ars Technica, where he oversees coverage of malware, computer espionage, botnets, hardware hacking, encryption, and passwords. In his spare time, he enjoys gardening, cooking, and following the independent music scene.

Channel Ars Technica

← Previous story Next story →

Promoted Comments

Promoted Comments

reader comments

Channel Ars Technica