dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
2296
woody1950
join:2007-01-19
Decatur, GA

woody1950

Member

SSH connection puzzle

I use SSH to connect to a remote Mac machine. In the past, I was able to connect to it without problem from several different machines. Now, I'm finding that I can connect from some machines but not from others.

The Mac is served by Comcast Cable using a Comcast-provided cable modem/router. The router has port 22 opened up in the firewall. The settings on this router are very limited, nothing fancy.

I tried connecting from various machines and ssh clients. Here's the results:
From behind the same router:
2 devices running OpenSSH client can connect
1 device running Dropbear SSH client on linux cannot connect
OpenSSh/Cygwin on Win7 cannot connect
PuTTy on Win 7 cannot connect

From other machines on other networks:
OpenSSH on linux can connect
PuTTy on Win7 cannot connect.

Nmap scans from these locations show that the port is "open" from some machines and "filtered" from other machines.

I'm really stumped about why it accepts connections from some sources and denies them from other ones. The issue seems to be specific to certain machines or ssh clients. Anybody have any thoughts on this?

GILXA1226
MVM
join:2000-12-29
Dayton, OH

GILXA1226

MVM

can you get additional logging from one of the clients that cannot connect? The OpenSSH/Cygwin client should be able to make use of the -vvv command switch to go to debug level three. Might give you some additional information.
woody1950
join:2007-01-19
Decatur, GA

woody1950

Member

I've tried using SSH in verbose mode. When the connection is successful, I get the whole stream of negotiations going on between client and server. When it's not successful, the connection times out as if the SSH client never gets a response from the server. So it looks like the connection is being blocked somewhere.

GILXA1226
MVM
join:2000-12-29
Dayton, OH

GILXA1226

MVM

I wonder if the Mac you are connecting too has implemented something like the hosts.deny that can be found on other *nix systems. This could be used to block all requests.

On my server I use denyhosts to populate the hosts.deny file if too many invalid connection attempts come in. Extra layer of security.
pablo
MVM
join:2003-06-23

pablo to woody1950

MVM

to woody1950
Hi,

Given what you've described, it sounds like a firewall is blocking the connection between the end-points. You can also always test without ssh by using telnet: telnet some-machine sshd-port

Cheers,
-pablo
woody1950
join:2007-01-19
Decatur, GA

woody1950

Member

Very bizarre. Yesterday, I rebooted my router (which is running Tomato). After that, I tested from 4 different devices behind the router:
1. Win7 PC. Previously could not connect from this machine, now I can
2. Linux device running Dropbear client: Could not connect before or after
3. Linux device running OpenSSH client: was able to connect before, but now can't.
4. Android phone: Can consistently connect from this device
5. Remote Linux server (not behind this router): Can consistently connect from this device

At this point, I'm really stumped to figure out why these connections are so variable. I'm thinking that it's some kind of routing issue, but I can't figure out where the connection is being blocked. I don't know if rebooting my router was a coincidence, or if I maybe got a new dynamic IP address from my ISP and if that made a difference.
pablo
MVM
join:2003-06-23

pablo

MVM

Hi woody1950 See Profile,

How about a simple network topology (draw it in ASCII) of your set up. Once we have the topology, we should be able to start simplifying to identify the root-cause.

Cheers,
-pablo
woody1950
join:2007-01-19
Decatur, GA

1 edit

woody1950

Member



                             ISP(AT&T)
                                 |
                           |DSL Modem|
                                 |
                |Tomato Router/Linux Opware|
                                 |
         --------------------------------------------
        |                        |                   |
   |Win 7 PC|           |Linux server|     |Android Phone|
 

I have Telnet and SSH clients on all of these devices, including the Tomato router.

As I said in my last post, previously, the Win7 PC and the Tomato router were not able to connect, but the Linux Server and the Android phone could. Now, the Win7 PC and Android phone can connect, but the Tomato router and the Linux server cannot connect. In the cases where the devices can connect, I'm able to successfully connect using Telnet as well as SSH. In the cases where I can't connect, both Telnet and SSH fail.
pablo
MVM
join:2003-06-23

pablo

MVM

Hi,

Please use the "code" tags to format your post.

Thx!
-pablo
woody1950
join:2007-01-19
Decatur, GA

woody1950

Member

OK, I fixed the post using block code tags. Thanks for the tip!
pablo
MVM
join:2003-06-23

pablo

MVM

Hi,

Where's the Mac Server in the above scheme? I don't see it in the schematic. Is it on the Internet? Does it have tcpdump on it?

Does your Tomato router have tcpdump?

We'll use the currently failing box, the Linux server for our testing. Let me know on the above.

Thx!
-pablo

Brano
I hate Vogons
MVM
join:2002-06-25
Burlington, ON

Brano to woody1950

MVM

to woody1950
In addition to above questions
- are those LAN devices on static IPs or DHCP?
- how does your port forwarding (22) on the router look like?

koitsu
MVM
join:2002-07-16
Mountain View, CA
Humax BGW320-500

1 edit

koitsu to pablo

MVM

to pablo
tcpdump is available for Tomato-based routers. The easiest way to get a working tcpdump binary is to use a statically-linked version. For most routers (MIPSR1 or MIPSR2-based) you can use this binary:

»multics.minidns.net/toma ··· /tcpdump

That comes from rhester72's utilities site where he makes some of these things available to folks. I personally prefer to use Entware, but for a quick-and-dirty "I don't have time or the space to deal with Entware I just need tcpdump!" situaiton, the above works.

telnet/ssh into the router, wget the above URL, chmod 755 the binary, go to town. I'm not going to provide a "how to use tcpdump" write-up. Note: this binary does not do IPv6.

P.S. -- I wouldn't be surprised if this turns out to be an IPv6 thing. :P
woody1950
join:2007-01-19
Decatur, GA

woody1950

Member

I appreciate all of the feedback. The linux server referenced above is a Pogoplug plug computer running ArchLinux Arm. I just installed tcpdump on it to do these tests.

To answer some of the questions asked:
1. The Mac is connected to a Comcast cable modem/router. It has port 22 opened in the router and pointed to the Mac. That computer belongs to a friend who is not tech-savvy. It's some distance from me, so I don't have quick physical access to it.

2. All of the devices mentioned above (including the Mac) have static IP addresses on the LAN.

3. Port 22 is not open in my router (for incoming connections). I use higher port numbers for all of my SSH incoming connections.

I haven't used tcpdump before, so I'll have to read up on it. I will run some tasks later today.

koitsu
MVM
join:2002-07-16
Mountain View, CA
Humax BGW320-500

koitsu

MVM

tcpdump -p -i eth0 -l -n -s 8192 "tcp and port 22"

I'm assuming your LAN Ethernet interface is eth0.

Be aware that this will capture packets for your own SSH session on that box itself. Distinguishing the difference is as easy as using host to match the IP address of the system you're trying to SSH to, i.e.:

In window #1: tcpdump -p -i eth0 -l -n -s 8192 "tcp and port 22 and host 1.2.3.4"
In window #2: ssh user@1.2.3.4

This also assumes IPv4.

Doing this on your Linux server is only going to give 1/4th of the picture however. What you actually need to do is run the equivalent of tcpdump on the following 4 points in the network:

1. Your Linux server
2. Your Tomato-based router
3. The Comcast-provided router which the Mac is connected to
4. The Mac itself

And I can assure you Comcast does not filter TCP port 22.

Please make sure you're using IPv4 exclusively across the board; like I said, I wouldn't be surprised if this turned out to be an IPv6 thing, where certain utilities are resolving a hostname (blah.xx.comcast.com) to an IPv6 address which has priority over IPv4 in most stacks, and the port forward on the router which the Mac is connected to might only support IPv4 forwarding.

You can rule this out by explicitly/exclusively connecting to the IPv4 address rather than using DNS (this is best for testing anyway).

I also wouldn't be surprised if this turns out to be a quirk/bug in the Comcast-provided router's firmware, or possibly some kind of SPI or "malformed IP packet" protection that has gone awry. Seen it many times over.
woody1950
join:2007-01-19
Decatur, GA

woody1950

Member

OK, here's what I did:
Rather than messing around with running tcpdump on the Win7 PC, I booted up an old PC I have that runs Ubuntu Linux. That machine is networked to my router the same as the Win7 PC and Pogoplug/Linux server. Connections to the Mac are successful from the Ubuntu PC. (Note that I masked the IP address of the Mac for security reasons.)

Here are tcpdumps from the Pogoplug and from the Ubuntu PC:

Pogoplug/Linux server (connection fails)

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 8192 bytes
13:07:09.179656 IP 192.168.123.58.53412 > xx.xx.74.168.22: Flags [S], seq 2144397977, win 14600, options [mss 1460,sackOK,TS val 15715341 ecr 0,nop,wscale 2], length 0
13:07:10.174524 IP 192.168.123.58.53412 > xx.xx.74.168.22: Flags [S], seq 2144397977, win 14600, options [mss 1460,sackOK,TS val 15715441 ecr 0,nop,wscale 2], length 0
13:07:12.174519 IP 192.168.123.58.53412 > xx.xx.74.168.22: Flags [S], seq 2144397977, win 14600, options [mss 1460,sackOK,TS val 15715641 ecr 0,nop,wscale 2], length 0
13:07:16.184517 IP 192.168.123.58.53412 > xx.xx.74.168.22: Flags [S], seq 2144397977, win 14600, options [mss 1460,sackOK,TS val 15716042 ecr 0,nop,wscale 2], length 0
13:07:24.204540 IP 192.168.123.58.53412 > xx.xx.74.168.22: Flags [S], seq 2144397977, win 14600, options [mss 1460,sackOK,TS val 15716844 ecr 0,nop,wscale 2], length 0
13:07:40.244523 IP 192.168.123.58.53412 > xx.xx.74.168.22: Flags [S], seq 2144397977, win 14600, options [mss 1460,sackOK,TS val 15718448 ecr 0,nop,wscale 2], length 0


Ubuntu/Linux PC (connection succeeds)

listening on eth0, link-type EN10MB (Ethernet), capture size 8192 bytes
13:28:30.876493 IP 192.168.123.100.37901 > xx.xx.74.168.22: Flags [S], seq 2522428520, win 14600, options [mss 1460,sackOK,TS val 62950 ecr 0,nop,wscale 4], length 0
13:28:30.898250 IP xx.xx.74.168.22 > 192.168.123.100.37901: Flags [S.], seq 168479137, ack 2522428521, win 65535, options [mss 1452,nop,wscale 3,nop,nop,TS val 992953978 ecr 62950,sackOK,eol], length 0
13:28:30.898305 IP 192.168.123.100.37901 > xx.xx.74.168.22: Flags [.], ack 1, win 913, options [nop,nop,TS val 62955 ecr 992953978], length 0
13:28:30.917691 IP xx.xx.74.168.22 > 192.168.123.100.37901: Flags [.], ack 1, win 65535, options [nop,nop,TS val 992953978 ecr 62955], length 0
13:28:30.934346 IP xx.xx.74.168.22 > 192.168.123.100.37901: Flags [P.], seq 1:22, ack 1, win 65535, options [nop,nop,TS val 992953978 ecr 62955], length 21
13:28:30.934404 IP 192.168.123.100.37901 > xx.xx.74.168.22: Flags [.], ack 22, win 913, options [nop,nop,TS val 62964 ecr 992953978], length 0
13:30:30.929714 IP xx.xx.74.168.22 > 192.168.123.100.37901: Flags [F.], seq 22, ack 1, win 65535, options [nop,nop,TS val 992955177 ecr 62964], length 0
13:30:30.930046 IP 192.168.123.100.37901 > xx.xx.74.168.22: Flags [F.], seq 1, ack 23, win 913, options [nop,nop,TS val 92963 ecr 992955177], length 0
13:30:30.948355 IP xx.xx.74.168.22 > 192.168.123.100.37901: Flags [.], ack 2, win 65535, options [nop,nop,TS val 992955177 ecr 92963], length 0


So, what I get from this is that both machines are initiating the connection the same way, but in one case, the Mac machine responds and in the other case, it doesn't.

As far as running tcpdump on the Mac, it does not belong to me and I'm not comfortable mucking around with my friend's machine to extent that I want to install packages on it and do that kind of testing. The Comcast cable modem/router that the Mac is attached to has very limited software on it. I don't believe there's any way to install and run any kind of software on it or even to get command line access to it.

At this point, I'm prepared to believe that this is indeed something happening in the cable modem/router. It is strange, though that the problem is not consistent; until yesterday, the Pogoplug server was connecting and the Win7 machine was not. Now, the situation is reversed. And some of the devices have been able to consistently connect.

I'm still open to any other theories that anybody might have...

koitsu
MVM
join:2002-07-16
Mountain View, CA
Humax BGW320-500

koitsu

MVM

The first packet capture (from the Pogoplug/Linux server) indicates TCP SYN is being sent out across the wire, but SYN+ACK is never seen from the server (xx.xx.74.168).

This could be for many different reasons:

1. Layer 1 (physical) or layer 2 (ARP/MAC) issues. You can rule these out by trying to initiate a SSH session to a host on the Internet other than xx.xx.74.168.
2. Router in front of Pogoplug/Linux server, which is what's doing NAT, is dropping the packet (i.e. the SSH client's TCP SYN packet never gets NAT'd and sent out across the Internet)
3. Comcast-provided router is dropping inbound packet -- possibilities/reasons for this are many, I listed a few in my previous post
4. Mac is dropping inbound packet -- OS firewall, etc..

As I said in my previous post: you need to do tcpdumps at all 4 locations in the network path to figure out 1) if the TCP SYN is even reaching the Comcast-provided router or the Mac and 2) if TCP SYN is seen, where in the same topology the TCP SYN+ACK ("response") is being dropped.

P.S. -- Please do not edit/change data in tcpdump output, it will only lead to confusion. You aren't ensuring any form of security by removing two octets of the destination IP address. Brute-force attacks are being initiated against TCP port 22 of all IP addresses 24x7x365; you accomplish nothing by doing this other than risk potentially messing up the output in some fashion and making it impossible for someone to troubleshoot things.
woody1950
join:2007-01-19
Decatur, GA

woody1950

Member

said by koitsu:

The first packet capture (from the Pogoplug/Linux server) indicates TCP SYN is being sent out across the wire, but SYN+ACK is never seen from the server (xx.xx.74.168).

This could be for many different reasons:

1. Layer 1 (physical) or layer 2 (ARP/MAC) issues. You can rule these out by trying to initiate a SSH session to a host on the Internet other than xx.xx.74.168.
2. Router in front of Pogoplug/Linux server, which is what's doing NAT, is dropping the packet (i.e. the SSH client's TCP SYN packet never gets NAT'd and sent out across the Internet)
3. Comcast-provided router is dropping inbound packet -- possibilities/reasons for this are many, I listed a few in my previous post
4. Mac is dropping inbound packet -- OS firewall, etc..

1. I'm able to initiate SSH connections to other destinations from this machine without any problems.

2. I tested by putting the Pogoplug in the DMZ of the router, which should take the firewall, at least, out of the picture. No change. Also, my router is using Tomato firmware, which is pretty widely used, so doesn't seem like it should have a flaw such as that.

At this point, the Mac or the Comcast router look like the best culprit. I will experiment to see if I can put the Comcast router in bridge mode and put another router behind it. That would, at least eliminate the router. Since the Mac is some distance from me, it will probably be a while before I have a chance to test at that end.

koitsu
MVM
join:2002-07-16
Mountain View, CA
Humax BGW320-500

1 recommendation

koitsu

MVM

said by woody1950:

1. I'm able to initiate SSH connections to other destinations from this machine without any problems.

Okay, that's a great piece of information -- I would say that more or less rules out issues local to your own network. If you want to be completely sure, please telnet into your router and download the tcpdump binary I listed above, then issue the same command as before (except for -i eth0 please use -i vlan2 (Tomato-specific thing)).
said by woody1950:

2. I tested by putting the Pogoplug in the DMZ of the router, which should take the firewall, at least, out of the picture. No change.

This has no bearing on the situation. DMZ applies only for new inbound packets (i.e. TCP SYN directed at your WAN IP); it has no bearing on outbound packets.

There is virtually no outbound firewall rules on Tomato routers (rather not get into the explanation of how the NAT rules work and so on). The firewall applies only to inbound traffic. Your SSH connection is an outbound connection, i.e. TCP SYN is sent from 192.168.123.58 on a dynamic port number, which gets NAT'd to {yourwanip}:{adifferentdynamicport}, with a destination of {comcastrouterwanip}:22.

Firewalls are not the only things that can drop packets; the kernel itself does a lot of other packet inspection and validation outside of the iptables layer (that's the firewalling layer).
said by woody1950:

Also, my router is using Tomato firmware, which is pretty widely used, so doesn't seem like it should have a flaw such as that.

I do not think your router is the issue here, but just to educate/clarify: if you think the Tomato firmware is flawless, you're sadly mistaken. :-) I myself run TomatoUSB (Toastman RT-N builds), and I spend a lot of time on linksysinfo.org's forum as well as here. Take a peek over there sometime; you might be surprised at the large number of bugs. The ability to fix some of them is greatly limited by the fact that the wireless driver is a binary blob, which means upgrading from Linux 2.66.22.19 isn't plausible. Sad but true.

There are some situations where the Linux kernel can/will dispose of packets intended for NAT rewriting + forwarding out the WAN. The situation gets more complex if PPPoE is used on the router itself (when I was an SBC/AT&T ADSL customer, I had my ADSL bridge configured to do PPPoE while blindly passing all other types of traffic, because of how unreliable PPPoE on consumer and third-party firmwares is).
said by woody1950:

At this point, the Mac or the Comcast router look like the best culprit. I will experiment to see if I can put the Comcast router in bridge mode and put another router behind it. That would, at least eliminate the router. Since the Mac is some distance from me, it will probably be a while before I have a chance to test at that end.

I would suggest replacing the Comcast-provided router with something else (i.e. something that has a working IP stack), or check to see if there are firmware updates for the Comcast-provided router. Most consumer routers, and even some SOHO or small business routers (costing multiple hundreds of dollars), are utter crap -- this is one of the reasons I buy very specific models and run very specific firmwares on them (and you too it seems! :-) ). I do not like "black box" network devices because troubleshooting problems -- case in point -- is virtually impossible with them.

However, in the meantime, you can use tcpdump on the Mac to watch for inbound connections to TCP port 22. The syntax there should be: tcpdump -p -i en0 -l -n -s 8192 "tcp and port 22 and host {wanipofyourtomatorouter}". You'll need to run this as root or via sudo. Be aware if you're already SSH'd into the Mac remotely and issuing tcpdump there, you're going to see tons of packets for your existing SSH connection (all TCP ACK and/or PSH+ACK). So you might have to change your tcpdump line to only look for TCP SYN, which would be: tcpdump -p -i en0 -l -n -s 8192 "tcp and port 22 and host {wanipofyourtomatorouter} and tcp[tcpflags] & (tcp-syn) != 0". If OS X's libpcap is fairly old it may complain about the tcp[tcpflags] syntax, in which case try using tcp[13] & 2 != 0 instead (specifically: looks at bit 2 (SYN) of the 13th byte of the TCP header and checks to see if it's non-zero).
woody1950
join:2007-01-19
Decatur, GA

woody1950

Member

Today, for no good reason, the Tomato router (running Tomato_USB) is connecting, even though it wasn't connecting yesterday. So, at the moment, all of my devices are connecting except for the Pogoplug/Linux server. Nothing has changed at my end since yesterday. Very weird.

Thanks for all your help! You are a wealth of good information. I'll try your suggestions and see what I can figure out.

clarknova
join:2010-02-23
Grande Prairie, AB

clarknova

Member

I'm starting to wonder if this isn't an MTU problem. You're only seeing a problem connecting to the Mac, which is not on your LAN, right? Could you try lowering the MTU on the router?

Note that tcpdump should be installed on the Mac already, so you should be able to just run it (assuming you can ssh to it from some device) without modifying software. The problem with running tcpdump via ssh on the Mac is that you will effectively create a storm on port 22 unless you can connect to it on a port or from an IP address other than the one you will be testing, such that you can filter packets from the control client out of the dump, and only capture those from the test client. Make sense?

If you can confirm that syn packets are leaving the client computer, but not arriving at the server, then doing packet dumps on both LAN and WAN interfaces of your router (and the remote router, if possible), will help to identify where the packets aren't getting through.

koitsu
MVM
join:2002-07-16
Mountain View, CA
Humax BGW320-500

1 recommendation

koitsu

MVM

said by clarknova:

Note that tcpdump should be installed on the Mac already, so you should be able to just run it (assuming you can ssh to it from some device) without modifying software. The problem with running tcpdump via ssh on the Mac is that you will effectively create a storm on port 22 unless you can connect to it on a port or from an IP address other than the one you will be testing, such that you can filter packets from the control client out of the dump, and only capture those from the test client. Make sense?

Already covered this in my last paragraph. Also, not to nitpick, but the term "storm" here is highly out-of-context; such a term is almost always associated with DoS or more literally "a large amount of unsolicited packets" -- that is not the case here.

clarknova
join:2010-02-23
Grande Prairie, AB

clarknova

Member

said by koitsu:

such a term is almost always associated with DoS or more literally "a large amount of unsolicited packets" -- that is not the case here.

If you tcpdump while connected via ssh without filtering those ssh packets, you are effectively DoSsing yourself. Whether or not it's unsolicited is another question I guess, but the positive feedback loop will certainly grow to make the dump useless.

But you're right, I did miss the fact that you already addressed the issue.

houkouonchi
join:2002-07-22
Ontario, CA

houkouonchi to woody1950

Member

to woody1950
It seems to me from your very first post the issue had to be a firewall issue on the machine running the SSH server, the router, or your ISP.

Nmap showing filtered ports like that pretty much has to be a firewall blocking it. If not it would be closed or open.