Understanding SSL Handshake Mechanism Between Webgate & OAM Server

Currently Webgate supports 3 modes of communication with OAM Server:

1. Open
2. Simple
3. Cert

With the SIMPLE & CERT Mode SSL comes in picture, but why at all we require it;

• The answer lies in the question it self i.e. security. OAM combined with Webgate is a security product that provides SSO & other features thus itself requires to be secure in talking terms.

From webgate-oam perspective we will be talking about NAP over SSL.

With SSL we ensure that the data transamitted b/w 2 parties is safe & sound as;

• Both the parties knows one another, but how because they do handshake before they start communication.
•  Once they start communicating, the packets or the data been transferred b/w them is secure as it is encrypted.

The 2 SSL modes of communication described above performs handshake process before they are sending data to each other; The handshake mechanism involves following steps;

Remember: In webgate & oam conversation, it is always the webgate who sends the request & server responds to it. So the request-response model is not vice-versa in OAM/Webgate case.   

1. In SSL Handshake first it is require for both client & server to ensure that they are talking to right or intended party only. So how webgate & oam do this;

1. It is the first INITIALIZE NAP message that is sent between both the parties;
2. This init_nap msg comprises of set of messages that they both exchange. And all this is done to ensure that the listener & receiver are not evil.
3. During this exchange webgate ask the server for the communication mode, if it is not open mode than webgate starts the SSL Handshake process. Describe in step 2.

2. At this step it is required to establish a secure connection between the webgate & oam. For this webgate prepare a CLIENT HELLO Message, which contains the following;

• Supported set of Cipher-suite like
  SSL_RSA_WITH_AES_256_CBC_SHA , SSL_RSA_WITH_3DES_EDE_CBC_SHA
• Which TLS Version to be supported; TLSv1.0, TLSv1.1, TLSv1.2; 
• Now webgate with r2ps3 also supports TLSv1.2 as well.

• Random Number - this number is of 32 bytes out of which 4 bytes contains date & time and rest are randomly generated. This random number will be used to prepare the master key (which is combination of client generated random number & server generated random number). This master key will be used for encrypting the data transferred b/w client & server once handshake is done.
• Session ID: a null session id is sent. If this is not the first time, than a valid value is been sent here as a session already exist b/w them.

 Cipher-suite: it is the algo that need to be used for encrypting the data.
 TLS version - it is the SSL supported version, latest one is TLSv1.2

3. Based on the Client Hello, server responds with SERVER HELLO, & in this server responds with;

• Supported cipher suite, that will be used in b/w webgate & oam for data encryption;
• Supported TLS version;
• And server sends its own certificate to webgate; And this is required to authenticate the server. Webgate does so by checking the cert in webgate truststore i.e. cwallet.sso. If webgate is able to verify the process is proceeded to next step, else it "TLS HANDSHAKE ERROR" is thrown.
• Possible reason for this:
• There is no trusted cert loaded in cwallet.sso, so webgate is unable to verify the server cert.
• Else the server provided cert is not a valid cert, hence server need to provide a valid cert.

• Random Number: server generated number same as client did while sending hello message.
• Session ID: server sends this newly generated id, that will be used in further communications. By this server can detect whether a session exist b/w webgate & server or not.

4. Key Exchange Phase: 

•  This phase is divided into 2 messages exchanged b/w client & server:
• SERVER KEY EXCHANGE
• CLIENT KEY EXCHANGE

Server key exchange - During this message exchange you will notice that while sending server hello, you see one more message along with it i.e. SERVER KEY EXCHANGE. This message is encrypted with server's private key and client decrypts it with the server's public key received with server's certificate.

• The SERVER KEY EXCHANGE Message is not mandatory to be sent by server, as it depends on the cipher suite selected for communication.
• DH_ANNON - for this cipher it will be sent
• RSA/MD5 - it will not be sent
• After this server key exchange, server hello is done. immediately after this from webgate side it is required to send CLIENT KEY EXCHANGE message. The data in this message is encrypted with server's public key.

Client key exchange -

• The purpose of sending this message is to share the master key, which client generates using the (client random number + server random number) encrypts it using the cipher suite selected & encrypts the whole message with public key shared by server in its certificate.

Note: 

• This above key exchange step is required so that to generate a master key that will be used as symmetric key b/w server & client for further data exchange.
• And to achieve this step we use asymmetric key, as we saw client used server public key for sending CLIENT KEY EXCHANGE Message while server used its private key to send SERVER KEY EXCHANGE Message.

5. FINISHED: at this point handshake process is done & now onwards client & server exchange the messages encrypted using the master key.


Note:

• It is possible that server will respond with error, if the list of cipher suite provided by webgate is not all supported not any one of it is supported or;
• The tls version provided by webgate is not supported by server.

6. Once client & server handshake is done successfully, after remaining INIT_NAP messages are excahnged b/w webgate & oam server.

7) Once above step is done successfully, than NAP channel is initialized b/w webgate & server. Basically this NAP channel in simple terms means a socket connection b/w webgate & server is established that will transmit the data over SSL.

Note: Once a NAP channel is initialized than for the requests been sent via this channel need not to do the SSL handshake again and again. It is a one time process per connection until unless that connections is to be refreshed or gets expired.


Hence the WEBGATE-OAM SSL Handshake finishes here.....



More Info: Related to SSL (nice article)
http://robertheaton.com/2014/03/27/how-does-https-actually-work/


Enjoy :-)