dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
23
share rss forum feed

woody1950

join:2007-01-19
Decatur, GA
reply to koitsu

Re: SSH connection puzzle

OK, here's what I did:
Rather than messing around with running tcpdump on the Win7 PC, I booted up an old PC I have that runs Ubuntu Linux. That machine is networked to my router the same as the Win7 PC and Pogoplug/Linux server. Connections to the Mac are successful from the Ubuntu PC. (Note that I masked the IP address of the Mac for security reasons.)

Here are tcpdumps from the Pogoplug and from the Ubuntu PC:

Pogoplug/Linux server (connection fails)

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 8192 bytes
13:07:09.179656 IP 192.168.123.58.53412 > xx.xx.74.168.22: Flags [S], seq 2144397977, win 14600, options [mss 1460,sackOK,TS val 15715341 ecr 0,nop,wscale 2], length 0
13:07:10.174524 IP 192.168.123.58.53412 > xx.xx.74.168.22: Flags [S], seq 2144397977, win 14600, options [mss 1460,sackOK,TS val 15715441 ecr 0,nop,wscale 2], length 0
13:07:12.174519 IP 192.168.123.58.53412 > xx.xx.74.168.22: Flags [S], seq 2144397977, win 14600, options [mss 1460,sackOK,TS val 15715641 ecr 0,nop,wscale 2], length 0
13:07:16.184517 IP 192.168.123.58.53412 > xx.xx.74.168.22: Flags [S], seq 2144397977, win 14600, options [mss 1460,sackOK,TS val 15716042 ecr 0,nop,wscale 2], length 0
13:07:24.204540 IP 192.168.123.58.53412 > xx.xx.74.168.22: Flags [S], seq 2144397977, win 14600, options [mss 1460,sackOK,TS val 15716844 ecr 0,nop,wscale 2], length 0
13:07:40.244523 IP 192.168.123.58.53412 > xx.xx.74.168.22: Flags [S], seq 2144397977, win 14600, options [mss 1460,sackOK,TS val 15718448 ecr 0,nop,wscale 2], length 0


Ubuntu/Linux PC (connection succeeds)

listening on eth0, link-type EN10MB (Ethernet), capture size 8192 bytes
13:28:30.876493 IP 192.168.123.100.37901 > xx.xx.74.168.22: Flags [S], seq 2522428520, win 14600, options [mss 1460,sackOK,TS val 62950 ecr 0,nop,wscale 4], length 0
13:28:30.898250 IP xx.xx.74.168.22 > 192.168.123.100.37901: Flags [S.], seq 168479137, ack 2522428521, win 65535, options [mss 1452,nop,wscale 3,nop,nop,TS val 992953978 ecr 62950,sackOK,eol], length 0
13:28:30.898305 IP 192.168.123.100.37901 > xx.xx.74.168.22: Flags [.], ack 1, win 913, options [nop,nop,TS val 62955 ecr 992953978], length 0
13:28:30.917691 IP xx.xx.74.168.22 > 192.168.123.100.37901: Flags [.], ack 1, win 65535, options [nop,nop,TS val 992953978 ecr 62955], length 0
13:28:30.934346 IP xx.xx.74.168.22 > 192.168.123.100.37901: Flags [P.], seq 1:22, ack 1, win 65535, options [nop,nop,TS val 992953978 ecr 62955], length 21
13:28:30.934404 IP 192.168.123.100.37901 > xx.xx.74.168.22: Flags [.], ack 22, win 913, options [nop,nop,TS val 62964 ecr 992953978], length 0
13:30:30.929714 IP xx.xx.74.168.22 > 192.168.123.100.37901: Flags [F.], seq 22, ack 1, win 65535, options [nop,nop,TS val 992955177 ecr 62964], length 0
13:30:30.930046 IP 192.168.123.100.37901 > xx.xx.74.168.22: Flags [F.], seq 1, ack 23, win 913, options [nop,nop,TS val 92963 ecr 992955177], length 0
13:30:30.948355 IP xx.xx.74.168.22 > 192.168.123.100.37901: Flags [.], ack 2, win 65535, options [nop,nop,TS val 992955177 ecr 92963], length 0


So, what I get from this is that both machines are initiating the connection the same way, but in one case, the Mac machine responds and in the other case, it doesn't.

As far as running tcpdump on the Mac, it does not belong to me and I'm not comfortable mucking around with my friend's machine to extent that I want to install packages on it and do that kind of testing. The Comcast cable modem/router that the Mac is attached to has very limited software on it. I don't believe there's any way to install and run any kind of software on it or even to get command line access to it.

At this point, I'm prepared to believe that this is indeed something happening in the cable modem/router. It is strange, though that the problem is not consistent; until yesterday, the Pogoplug server was connecting and the Win7 machine was not. Now, the situation is reversed. And some of the devices have been able to consistently connect.

I'm still open to any other theories that anybody might have...



koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

The first packet capture (from the Pogoplug/Linux server) indicates TCP SYN is being sent out across the wire, but SYN+ACK is never seen from the server (xx.xx.74.168).

This could be for many different reasons:

1. Layer 1 (physical) or layer 2 (ARP/MAC) issues. You can rule these out by trying to initiate a SSH session to a host on the Internet other than xx.xx.74.168.
2. Router in front of Pogoplug/Linux server, which is what's doing NAT, is dropping the packet (i.e. the SSH client's TCP SYN packet never gets NAT'd and sent out across the Internet)
3. Comcast-provided router is dropping inbound packet -- possibilities/reasons for this are many, I listed a few in my previous post
4. Mac is dropping inbound packet -- OS firewall, etc..

As I said in my previous post: you need to do tcpdumps at all 4 locations in the network path to figure out 1) if the TCP SYN is even reaching the Comcast-provided router or the Mac and 2) if TCP SYN is seen, where in the same topology the TCP SYN+ACK ("response") is being dropped.

P.S. -- Please do not edit/change data in tcpdump output, it will only lead to confusion. You aren't ensuring any form of security by removing two octets of the destination IP address. Brute-force attacks are being initiated against TCP port 22 of all IP addresses 24x7x365; you accomplish nothing by doing this other than risk potentially messing up the output in some fashion and making it impossible for someone to troubleshoot things.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


woody1950

join:2007-01-19
Decatur, GA

said by koitsu:

The first packet capture (from the Pogoplug/Linux server) indicates TCP SYN is being sent out across the wire, but SYN+ACK is never seen from the server (xx.xx.74.168).

This could be for many different reasons:

1. Layer 1 (physical) or layer 2 (ARP/MAC) issues. You can rule these out by trying to initiate a SSH session to a host on the Internet other than xx.xx.74.168.
2. Router in front of Pogoplug/Linux server, which is what's doing NAT, is dropping the packet (i.e. the SSH client's TCP SYN packet never gets NAT'd and sent out across the Internet)
3. Comcast-provided router is dropping inbound packet -- possibilities/reasons for this are many, I listed a few in my previous post
4. Mac is dropping inbound packet -- OS firewall, etc..

1. I'm able to initiate SSH connections to other destinations from this machine without any problems.

2. I tested by putting the Pogoplug in the DMZ of the router, which should take the firewall, at least, out of the picture. No change. Also, my router is using Tomato firmware, which is pretty widely used, so doesn't seem like it should have a flaw such as that.

At this point, the Mac or the Comcast router look like the best culprit. I will experiment to see if I can put the Comcast router in bridge mode and put another router behind it. That would, at least eliminate the router. Since the Mac is some distance from me, it will probably be a while before I have a chance to test at that end.


koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

1 recommendation

said by woody1950:

1. I'm able to initiate SSH connections to other destinations from this machine without any problems.

Okay, that's a great piece of information -- I would say that more or less rules out issues local to your own network. If you want to be completely sure, please telnet into your router and download the tcpdump binary I listed above, then issue the same command as before (except for -i eth0 please use -i vlan2 (Tomato-specific thing)).

said by woody1950:

2. I tested by putting the Pogoplug in the DMZ of the router, which should take the firewall, at least, out of the picture. No change.

This has no bearing on the situation. DMZ applies only for new inbound packets (i.e. TCP SYN directed at your WAN IP); it has no bearing on outbound packets.

There is virtually no outbound firewall rules on Tomato routers (rather not get into the explanation of how the NAT rules work and so on). The firewall applies only to inbound traffic. Your SSH connection is an outbound connection, i.e. TCP SYN is sent from 192.168.123.58 on a dynamic port number, which gets NAT'd to {yourwanip}:{adifferentdynamicport}, with a destination of {comcastrouterwanip}:22.

Firewalls are not the only things that can drop packets; the kernel itself does a lot of other packet inspection and validation outside of the iptables layer (that's the firewalling layer).

said by woody1950:

Also, my router is using Tomato firmware, which is pretty widely used, so doesn't seem like it should have a flaw such as that.

I do not think your router is the issue here, but just to educate/clarify: if you think the Tomato firmware is flawless, you're sadly mistaken. :-) I myself run TomatoUSB (Toastman RT-N builds), and I spend a lot of time on linksysinfo.org's forum as well as here. Take a peek over there sometime; you might be surprised at the large number of bugs. The ability to fix some of them is greatly limited by the fact that the wireless driver is a binary blob, which means upgrading from Linux 2.66.22.19 isn't plausible. Sad but true.

There are some situations where the Linux kernel can/will dispose of packets intended for NAT rewriting + forwarding out the WAN. The situation gets more complex if PPPoE is used on the router itself (when I was an SBC/AT&T ADSL customer, I had my ADSL bridge configured to do PPPoE while blindly passing all other types of traffic, because of how unreliable PPPoE on consumer and third-party firmwares is).

said by woody1950:

At this point, the Mac or the Comcast router look like the best culprit. I will experiment to see if I can put the Comcast router in bridge mode and put another router behind it. That would, at least eliminate the router. Since the Mac is some distance from me, it will probably be a while before I have a chance to test at that end.

I would suggest replacing the Comcast-provided router with something else (i.e. something that has a working IP stack), or check to see if there are firmware updates for the Comcast-provided router. Most consumer routers, and even some SOHO or small business routers (costing multiple hundreds of dollars), are utter crap -- this is one of the reasons I buy very specific models and run very specific firmwares on them (and you too it seems! :-) ). I do not like "black box" network devices because troubleshooting problems -- case in point -- is virtually impossible with them.

However, in the meantime, you can use tcpdump on the Mac to watch for inbound connections to TCP port 22. The syntax there should be: tcpdump -p -i en0 -l -n -s 8192 "tcp and port 22 and host {wanipofyourtomatorouter}". You'll need to run this as root or via sudo. Be aware if you're already SSH'd into the Mac remotely and issuing tcpdump there, you're going to see tons of packets for your existing SSH connection (all TCP ACK and/or PSH+ACK). So you might have to change your tcpdump line to only look for TCP SYN, which would be: tcpdump -p -i en0 -l -n -s 8192 "tcp and port 22 and host {wanipofyourtomatorouter} and tcp[tcpflags] & (tcp-syn) != 0". If OS X's libpcap is fairly old it may complain about the tcp[tcpflags] syntax, in which case try using tcp[13] & 2 != 0 instead (specifically: looks at bit 2 (SYN) of the 13th byte of the TCP header and checks to see if it's non-zero).
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.

woody1950

join:2007-01-19
Decatur, GA

Today, for no good reason, the Tomato router (running Tomato_USB) is connecting, even though it wasn't connecting yesterday. So, at the moment, all of my devices are connecting except for the Pogoplug/Linux server. Nothing has changed at my end since yesterday. Very weird.

Thanks for all your help! You are a wealth of good information. I'll try your suggestions and see what I can figure out.



clarknova

join:2010-02-23
Grande Prairie, AB
kudos:7
Reviews:
·TekSavvy DSL

I'm starting to wonder if this isn't an MTU problem. You're only seeing a problem connecting to the Mac, which is not on your LAN, right? Could you try lowering the MTU on the router?

Note that tcpdump should be installed on the Mac already, so you should be able to just run it (assuming you can ssh to it from some device) without modifying software. The problem with running tcpdump via ssh on the Mac is that you will effectively create a storm on port 22 unless you can connect to it on a port or from an IP address other than the one you will be testing, such that you can filter packets from the control client out of the dump, and only capture those from the test client. Make sense?

If you can confirm that syn packets are leaving the client computer, but not arriving at the server, then doing packet dumps on both LAN and WAN interfaces of your router (and the remote router, if possible), will help to identify where the packets aren't getting through.
--
db



koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

1 recommendation

said by clarknova:

Note that tcpdump should be installed on the Mac already, so you should be able to just run it (assuming you can ssh to it from some device) without modifying software. The problem with running tcpdump via ssh on the Mac is that you will effectively create a storm on port 22 unless you can connect to it on a port or from an IP address other than the one you will be testing, such that you can filter packets from the control client out of the dump, and only capture those from the test client. Make sense?

Already covered this in my last paragraph. Also, not to nitpick, but the term "storm" here is highly out-of-context; such a term is almost always associated with DoS or more literally "a large amount of unsolicited packets" -- that is not the case here.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


clarknova

join:2010-02-23
Grande Prairie, AB
kudos:7
Reviews:
·TekSavvy DSL

said by koitsu:

such a term is almost always associated with DoS or more literally "a large amount of unsolicited packets" -- that is not the case here.

If you tcpdump while connected via ssh without filtering those ssh packets, you are effectively DoSsing yourself. Whether or not it's unsolicited is another question I guess, but the positive feedback loop will certainly grow to make the dump useless.

But you're right, I did miss the fact that you already addressed the issue.
--
db