StoneBlog.stonesoft.com

One of the features I use often, and especially in cases when there is some sort of trouble, is the ability to actually see what traffic passes the firewall.

Most admins don’t feel comfortable using the console (over ssh), and ofcourse it is not as trivial as it seems – especially remembering the exact commands. So, for the community, and for my own personal use, I’ll document a small issue I just had, and how I “solved” it.

A customer called, saying: “I use the StoneGate VPN to connect to my server with RDP, and all I get is a black screen”.  Now, that’s something that’s (unfortunately) not too uncommon. Google for “MTU”, “Path MTU Discovery” and “Black Hole Detection”, and you’ll get tons of info, which all come down to:

Single packets in ethernet networks have a maximum size of 1500 bytes (RFC 879). 1460 bytes of data + 40 bytes header (ip-addresses, ports, settings etc.). All tunneling protocols (VPN, PPTP,PPPoE, etc.) add some bytes to the header part. This means less room for the data part.

Both “client” and “server”  agree to send packets with max. 1460 bytes of data. The first few packets of the connection aren’t large, perhaps 1000 bytes max, and fit through perfectly. Client and server agree to communicate, draw a frame of the correct size, etc. Then however, comes the Windows Logo, a picture that is over 3000 bytes of size.  That means,  2  large packets are sent.  Somewhere on the connection from server to client, these packets do not fit. So, the picture the server sent, does not reach the client. A black screen of the wanted size just sits there, and waits… and waits…. and waits…..

Since I do not want to discuss what causes this,  but just want to know if it IS an MTU issue, I do following:

• check if both sides agree to use 1460 bytes of data
• reduce the packet size on either client or server side to 1310 bytes of data
• test whether RDP works again

So, lets see what happens (this is where we get technical, sorry non techies).

I’ll explain what the commands do later on. I start my Putty SSH Client, and logon to the firewall with root/mypassword. Since I want to see the traffic going from firewall to the server, I put in the server-interface, filter for the server and the RDP port 3389, as well as not to show my own IP:

root@fw-zentrale1:~# tcpdump -i eth1 host 192.168.101.103 and tcp port 3389 and not host 259.79.49.139

I tell the customer to connect, and the first 3 lines show:

14:23:36.348613 IP 192.168.101.142.1326 > 192.168.101.103.3389: S 1913531291:1913531291(0) win 65535 <mss 1460,nop,wscale 1,nop,nop,sackOK>
14:23:36.348803 IP 192.168.101.103.3389 > 192.168.101.142.1326: S 2569746333:2569746333(0) ack 1913531292 win 16384 <mss 1460,nop,wscale 0,nop,nop,sackOK>
14:23:36.517536 IP 192.168.101.142.1326 > 192.168.101.103.3389: . ack 1 win 65535

Yes, dear readers, that’s what packets actually look like. I highligted the part where the data size is set. It seems both sides agree to send 1460 bytes of data. I ask the customer to lower the MTU on his laptop from 1500 to 1350 bytes, using “drtcp021.exe”, one of the ancient tools in my toolbox. You can get it from serious sources easily. The customer reboots his laptop for the MTU to be set, I press “enter” a few times, to get a few newlines and ask the customer to connect once more.

14:32:58.634817 IP 192.168.101.142.1114 > 192.168.101.103.3389: S 720573675:720573675(0) win 65535 <mss 1310,nop,wscale 1,nop,nop,sackOK>
14:32:58.634903 IP 192.168.101.103.3389 > 192.168.101.142.1114: S 858372081:858372081(0) ack 720573676 win 16384 <mss 1460,nop,wscale 0,nop,nop,sackOK>
14:32:58.825297 IP 192.168.101.142.1114 > 192.168.101.103.3389: . ack 1 win 65535

Aha! The client now want to send a maximum of 1310 bytes per packet, the server still wants to send 1460 bytes per packet. Ofcourse, both agree to use the lower value, so they can speak together. the Windows image is now sent in slightly smaller portions, and arrives at the client. The logo appears, the customer can login, and is astonished.

Now starts the long discussion as of why this happens, what can be done against it, and why the heck this problem bugs him alone! (it doesn’t). That has been discussed a lot on lots of sites and platforms, but does not belong here. I will just end it here with a short explanation of the command, and some nice tips to further explore the possibilities of actually looking at the packets themselves.

• tcpdump : available on virtually all unix/linux based environments
• -i eth1 : show packets on ethernet port 1. use “ip addr” as a command for an overview of interfaces and ip’s
• host 192.168.101.103 : show packets going to or coming from this ip-address
• and tcp port 3389 : but only packets on the RDP (Remote Desktop) port
• and not host 259.79.49.139 : but not the packets from my own session to the server

you can also use “and not icmp” or “and not udp port 23” instead of showing tcp packets. If you like your filter, but things go too fast for your eyes, use “-w jebdump1.dmp” to write the packets on the firewall, where you can get them using scp or sftp (hint: winscp and root/password). On windows, you can then use “wireshark” or “packetyzer” to have a (colorized) view at your own speed. Sometimes you’ll want to add “-s 0“(that’s zero) to get the full 1460 (or 1310) bytes of the packets that pass by. By default tcpdump doesn’t record the full data part of the packets, but just the start of it.

Try not to be scared of the console, it can really help you out sometimes!

written by jebATpop-i - 3,404 views \\ tags: CLI, Examples, stonegate firewall, Tips & Tricks