The OTR protocol uses a more interactive method of communication. The advantage to this approach is the ability to create more anonymous messages. The disadvantage, of course, is its increased complexity. Therefore, we will first show a sample run of the protocol.
The first step in the protocol is the establishing of a shared secret between the correct parties. You might recall this is done using the Diffie Hellman key exchange. Again, first we define the assumptions.
Which states that they both have a public and private key, they have each others' public key, and they believe these are suitable for communication between them. If they didn't have each others' keys, an attack similar to the one described in the previous section would be possible. The Diffie Hellman exchange can be written as follows:
Now, Alice and Bob have a shared secret gxy that they can use to send a message. Furthermore, by sending their identities, it is clear to whom they want to talk. A message sent using the OTR protocol is defined as follows:
Where MAC is the Message Authentication Code used to authenticate the message (which includes the new Diffie Hellman parameter, and the encrypted message) using key H(kij). Remember that a MAC(X,K) is capable of confirming message X when the receiver also knows secret K. kij is a key created by taking the hash of the value gxiyj, which both Alice and Bob should know now. gxi+i was sent in order to create a new key, so the older one can be released.
Now, we can define what Bob has learned this moment:
That is, Bob knows the shared secret, the key derived from it, and the message authentication key, which is the hash of the key. This key is taken as the hash of the shared key, so both parties know it, and it won't compromise the shared key if it is released to the public. Also, Bob believes the shared key is a suitable to communicate with Alice, since only he and Alice know the secret created by Diffie Hellman.
After this, Bob receives Alice's message:
With this information, Bob is able to authenticate the message and read it:
However, Bob does not yet know Alice actually sent the message. In order to come to this conclusion, Bob needs to authenticate the message. This is done using the verifation rule V2:
That is, if Bob receives the MAC of a message X, and he owns a key of which he thinks it is known only to him and Alice, and he is in possession of the message X, he can conclude the message X was written by Alice. This is like the same rule as V1, however in this case we make use of the shared key K, of which Bob knows it is a good key between him and Alice, but which the outside world does not know. Now Bob knows something more:
That is, Bob knows that Alice sent a new part of the Diffie-Hellman key, and also sent a message encrypted with key kij, which is in his possession. Bob can now safely assume Alice was the sender of the message.
The reason Bob uses the message authentication code for verification will now be demonstrated. Because the message is authenticated using a different key than the encryption key, it won't matter if this key is released after it isn't used anymore. In fact, this is precisely what Bob is going to do. Let's suppose Eve was listening in on the conversation the whole time.
That is, Eve hears a past message, and after this, she also hears the key with which the authenticity was established. Now, she is able to verify that the message was indeed authenticated with the key H(kij). However, she is not able to conclude that Alice sent the message, since Eve does not know if Alice and Bob were the only ones with the secret key. And, even then, she cannot tell for sure whether it was Bob or Alice who sent the message (since they're using the same key). Furthermore, she cannot prove to anyone that either Bob or Alice were sending the message, since she is in possession of the key. Thus, she might as well create her own encrypted message, and create a MAC that was suitable for the key.
To make the point more clear, Eve was not able to do this until she received the key. Before the key was released, only Bob and Alice knew the secret and thus only they were able to authenticate the message. After the release of the key, Eve is able to verify the authentication, but does not know who it came from. Also, no-one will believe her because, since she was able to check the hash, she was also capable of creating the hash herself.
This point also proves the non-reputability of the authentication code: even if Bob wants to, he cannot show anyone else Alice sent a particular message. He knows for himself a message came from Alice, since he didn't send it, but he cannot prove this to anyone else, since he might have created the message himself.
Because of the use of Diffie Hellman key exchange, public keys are not used for encryption in this protocol. This means that, even if the private key of Bob is compromised, a perpetrator cannot decrypt past messages (though he can start a message with Alice under Bob's name). This can be shown using GNY. Let's first show Eve's knowledge:
Now, the same authentication messages are sent:
As we can see, the public key is not used for encryption. Thus, Eve can't do more with the private key than she could before, unless she initiates a session:
Because Bob's private key is used, A will be under the impression that she is talking to Bob. This demonstrates the point that even though past conversations will not be decodable, future sessions can be compromised when the private key is exposed. This feature is called perfect forward secrecy.
We now have demonstrated the non-reputability and perfect forward secrecy of the OTR protocol. However, the protocol itself does have some weaknesses, which we will show now.
First of all, OTR is susceptible to the same key responsibility weakness as OpenPGP. When the same channel is used to verify the authenticity of the public keys, a third party may intercept the messages and act as a proxy. This is inherent to the use of public keys however, so might not be considered a flaw.
More seriously, OTR is susceptible to the Diffie - Hellman authentication flaw which has become famous in the cryptography world. This flaw was detected by Diffie and Hellman themselves, and the result of the flaw is that one party thinks he is talking to someone else. It shows that the intricacies of anonymous key exchange are more subtle than one might think. We will demonstrate this flaw using the Diffie-Hellman key exchange part of the OTR-protocol.
The flaw is executed by Eve running two sessions at the same time. Using these two sessions to Alice and Bob, Eve is able to trick Alice into thinking she is communicating with Bob, and Bob thinking he is communicating with Alice. While Eve does not know the key, and is thus not able to send messages herself, she can forward messages between the two sessions:
Now, messages sent from Bob to Eve can be sent by Eve as if coming from Bob, and messages sent from Alice to Bob can be sent as if they are coming from Eve, even if Eve cannot read the messages herself. The solution to this flaw is to include the identitiy of the person you wish to talk to in the initial hash. However, using this, the non-reputability part of the protocol will disappear (since we have a message from Alice saying she wants to speak to Bob.)
A third problem in OTR is the possibility of session hijacking. The use of public and private keys to initiate a session should mean that, as long as the private keys of Alice and Bob aren't compromised, one cannot create a new session. Using the OTR protocol, however, a new session can be started between Alice and a perpetrator when this attacker knows only a single Diffie-Hellman value once used by Bob. We will demonstrate this shortly. First, a normal key establishment is created. We will first define our assumptions, as usual:
That is, Bob and Alice have each other's keys and trust them, but Eve also has the public keys (which shouldn't be hard to get), and knows Alice and Bob trust each other in using them. Now, a standard initialization session commences:
And Eve snoops in on the conversation. Now, she knows gx and gy. Suppose later on she also learns the value of x from Alice somehow. Now, Eve is able to create a session with Bob under Alice's name, by sending the same message:
The solution to this problem is the use of freshness. That is, Alice not only has to provide a partial key, she has to provide proof that it has been generated recently and hasn't been used in a session before. Typically this is done using some form of timestamp. Of course, Bob will have to do the same.