>>How to diagnose TCP Session Closing and disconnections ?

How to diagnose TCP Session Closing and disconnections ?

By | 2017-11-22T16:20:35+00:00 April 26th, 2017|Categories: Performance Troubleshooting|

TCP Series #2: TCP Closing a SESSION

Here is a second article of a series covering all you need to know to troubleshoot performance issues impacting applications relying on the TCP Protocol.

After studying how TCP sessions are established we will now see what can go wrong when closing a TCP sessions.

Paradoxically, closing a TCP connection is more complex than creating one! This is due to the fact that resources must be correctly released on both side.

standard_tcp_end.png Fig1 – Simplified TCP closing with FIN.

The standard way to close a TCP session is to send a FIN packet, then waiting a FIN response from the other party.

  1. A sends a FIN packets and wait for a response, it can releases some resources but wait the response of the other part (Fin Wait)
  2. B receives it and must release resources (wait for closing application level – Close Wait)
  3. B can now send a FIN to A and is waiting the acknowledgement (Last Ack wait).
  4. A can now fully close its job, but must wait for network collision (?) (Time Wait), it may have to send the final Ack another time.
  5. B eventually receives the final Ack and destroyes (kills) the connection.

This works fine in a perfect world. But what happens when one part of the conversasion is broken ? That’s why the Reset (RST) packet exists.

standard_tcp_rst.png

Fig2 – RST sent to force the end of a TCP session.

Those abnormal terminations (either aborted set up or disconnection) could appear for:

  • Lack of resources or network interruption,
  • A crash / bug during the session,
  • While one pair has already closed its part of the connexion but the other part continues to send data,
  • The server refuses to open a connection to the client.

PV provides metrics to see if TCP connections ends properly or not. By selecting the theme “TCP Events” you get the count of FIN and RST packets in both directions.

tcp_table_tcp_rst.pngFig3 – PV showing client IPs sorted by server RST.

If the SYN rate per connection and the server RST are both high (Fig3 – 1st row), this means that the server refuses the client connection demands. With a drilldown to the conversations (Fig4), you will have the precise server ports and applications that cause this issue. Hereunder, we see some attempts to connect to a VPN from an IP address using port `1194`

tcp_table_fin_rst_details.png

Fig4 – PV showing who the cause of RST without any Connections.

For Advanced Users of PerformanceVision
If you want to filter on data with or without
RST or
FIN packets,
PV provides some custom filters. In the previous article, we already saw how to filter on connections.

 

  • FIN: fin.count, fin.count.srv, fin.count.clt
  • RST: rst.count, rst.count.srv, rst.count.clt

 

Examples:
  • Flows with only RST and no FIN:
    rst.count > 0 and fin.count = 0
  • Flows with RST but without connections:
    rst.count > 0 and ct.count = 0

Sometimes, RST packets are quite “normal”. For example when the user manualy interrupts a huge data transfert. The TCP session is sending packets as fast as possible, so when the client sends the FIN and closes its part, the server is still sending lots of data for a moment. In this case, the client sends RST packets until the server stops sending data. In this case, the is as client FIN (than server FIN), but in addition you will see some RST packets.

It is also important to note that some applications do not close session properly and simply use a RST to close every session. It is not a good practice, but you must be aware that some applications are developed that way.

It could also be relevant to graph some RST or FIN metrics over time. PV provides a metric to graph the rate of RST per connexion over time using Custom Graphs (Fig5).

rst_over_time.png

Fig5 – PV Custom Chart showing TCP RST over time in both direction (from Client and from Server).

CONCLUSION

We saw that closing a TCP connection can be more complex than opening. The session can be closed by a double FIN, by a mix FIN + RST, or only by RST packets. But RST packets can also be sent without any connection.

PerformanceVision helps to diagnose session issues by reporting statistics about FIN, RST, SYN and Connections. It is also able to graph all the metrics over time, especially the RST per Connection.

As we saw in the first article, we can use pages like Top Client Zone and Top Servers to analyse where the issue is from.

In a next article, we will have a look at how TCP retransmission works, and when it can be an issue or not.

In the meantime; if you would like to give it a try, just download our evaluation virtual appliance or download our troubleshooting guide

TRY PERFORMANCE VISION FOR FREE

About the Author: