dslreports logo
 
    All Forums Hot Topics Gallery
spc
uniqs
133

koitsu
MVM
join:2002-07-16
Mountain View, CA
Humax BGW320-500

1 recommendation

koitsu

MVM

Re: TomatoUSB and Comcast IPv6 -- bugs found

For those following this thread (and I imagine there is many since I posted it everywhere relevant, haha ) I thought I'd provide a status update as to where we are, what's being discussed, ideas, etc... In no particular order:

1. Toastman and I have talked privately on the linksysinfo.org forum. During our discussion another user showed up and posted something that looked like a key piece of information (keep reading). Toastman at this point is willing to build firmwares for people to download/test, assuming there are folks who have good ideas as to what the bugs/issues are and what to change.

The sad part of our conversation is that IPv6 support in TomatoUSB in general is neglected big time. There was a really key person who was IPv6-savvy and part of the Tomato/TomatoUSB project who has since disappeared (hasn't responded to Emails, etc.), which is a bummer given his extensive knowledge of the protocol. (I looked the guy up myself and he even provided patches/code to the Linux iptables folks to fix problems). Toastman and I both wish we could find this guy.

2. An individual on the tomatousb.org and linksysinfo.org forums named armarkey mentioned that the "spurious default route" actually isn't so spurious after all, and that it might be related to metrics. Metrics define what route (or interface, depending on context) get preferred over another; think of it as a preferencing order. Two routes with an identical metric result in the route which got defined first taking precidence.

armarkey's theory is that because the IPv6 ICMP-based RA (route announcement) default metric is 1024, and that the DHCPv6 client's default route metric is 1024 (which is created the instant the interface is brought up), this causes a situation where the latter route was defined first, thus gets used when in fact the ICMP-based RA route should be being used.

I tried solving the problem by removing the metric 1024 route (same route you see in my first post) and inserting the same route with a metric of 2048, but to no avail.

So we're still working on that to figure out what the cause is, if the metric is actually responsible for the problem, yadda yadda.

3. armarkey also pointed out that the IA-NA workaround which was proposed by Comcast forum user morrildl may actually be incorrect in general. All this does is, in addition to Comcast handing out a /64 network to the DHCPv6 client in TomatoUSB, it also hands a /128 (single IPv6 address).

The theory is that the /128 isn't needed at all. That and the default route fix do make it so you can talk to the Internet via IPv6 directly inside TomatoUSB, but technically one should be able to do that anyway since br0 is assigned an address as part of the /64.

I am in agreement with this theory -- but of course, in practise, it doesn't work and we're still not sure why.

So for now, generally speaking, the IA-NA workaround is probably not needed.

I can absolutely confirm, hands down, no questions asked, that if you run TomatoUSB without the IA-NA workaround that you do in fact get handed a /64 network from Comcast. This means the DHCPv6 request in TomatoUSB is in fact working correctly, and that the problem may be elsewhere.

That leads me to the final item...

4. Most everyone I've been hearing from (privately and from reading public forums) has the exact same problem when using TomatoUSB's "DHCPv6 with Prefix Delegation" mode -- specifically it's that IPv6-capable machines on the LAN segment simply don't work.

I spent some time tonight poking at this (see my long post a few pages up) and didn't get anywhere. The IA_NA fix and default route fix don't resolve this problem. All I'm able to confirm at this point is that packets from a machine on the LAN segment do get sent to the TomatoUSB router correctly, and the TomatoUSB router does in fact send the packets out the WAN interface. However, no responses are ever seen. My guess is that the response packets never arrive because of the routing loop ordeal I mentioned a few posts up.

So we're still researching that issue as well.

HunterZ
join:2003-07-16
Kent, WA

1 recommendation

HunterZ

Member

koitsu, thanks a ton for keeping everyone posted on this. Hopefully it will be ironed out by the time Comcast turns on ipv6 in my area

Morac
Cat god
join:2001-08-30
Riverside, NJ

Morac to koitsu

Member

to koitsu
I've been trying to read up on how this is all supposed to work ( »manpages.ubuntu.com/manp ··· f.5.html, »www.ietf.org/rfc/rfc3315.txt, »www.ietf.org/rfc/rfc3633.txt ) and from my understanding the delegating router (i.e. Comcast) is supposed to assign the requesting router (Tomato) a prefix. It's up to the requesting router to divvy that up into subnets and addresses and give it to local clients.

If Tomato is getting the prefix, which it sounds like it is, then Comcast should know to route any responses back to addresses that falls in that Prefix to Tomato, unless Tomato is mangling the outbound requests, but you already tested trying to do an inbound traceroute (which failed) so that seems unlikely. As such it sounds like the initial DHCPv6 setup is screwy since the delegating router doesn't know where the requesting router is.

Has anyone dumped out the actual DHCPv6 requests going back and forth?

The thing I find a bit odd is that Tomato does not appear to use the prefix to assign itself an iPv6 address on the WAN side. You'd think it would need one in order to talk for itself (as opposed to LAN clients) on the WAN side.

Speaking of the later, it sounds like the router is supposed to assign addresses to the LAN using the prefix with stateless address autoconfiguration and then announce those using router advertisements. That could be why routing is failing.

One thing I did notice looking at the code is that there is a "debug_ipv6" nvram setting which appears to enable debugging for both dhcp and radvd (but not dhcpv6).

Most of the code regarding DHCP and router advertisements is in dhcp.c and services.c in »repo.or.cz/w/tomato.git/ ··· outer/rc

DHCPv6 is implemented in »repo.or.cz/w/tomato.git/ ··· r/dhcpv6

ipV6 is implemented as part of Linux in »repo.or.cz/w/tomato.git/ ··· net/ipv6

koitsu
MVM
join:2002-07-16
Mountain View, CA
Humax BGW320-500

koitsu

MVM

said by Morac:

I've been trying to read up on how this is all supposed to work ( »manpages.ubuntu.com/manp ··· f.5.html, »www.ietf.org/rfc/rfc3315.txt, »www.ietf.org/rfc/rfc3633.txt ) and from my understanding the delegating router (i.e. Comcast) is supposed to assign the requesting router (Tomato) a prefix. It's up to the requesting router to divvy that up into subnets and addresses and give it to local clients.

If Tomato is getting the prefix, which it sounds like it is, then Comcast should know to route any responses back to addresses that falls in that Prefix to Tomato, unless Tomato is mangling the outbound requests, but you already tested trying to do an inbound traceroute (which failed) so that seems unlikely. As such it sounds like the initial DHCPv6 setup is screwy since the delegating router doesn't know where the requesting router is.

Has anyone dumped out the actual DHCPv6 requests going back and forth?

Yes, I have done this. Let me explain the annoyance however:

The problem with using tcpdump on TomatoUSB is that it (and libpcap) is extremely old and does not have decent IPv6 support. I'm using the one available via ipkg, which come from »ipkg.nslu2-linux.org/fee ··· /stable/ --

libpcap - 1.0.0-2 - PCAP Library
tcpdump - 4.0.0-1 - tcpdump dumps the traffic on a network
 

Old. Ancient. Horrible. The problem is that these can detect IPv6 packets (based on protocol header) but it cannot decode any of the IPv6 header data/etc.. This requires one to do the following:

kill `cat /var/run/dhcp6c.pid`
tcpdump -p -l -n -s 0 -i vlan2 -w /tmp/filename.pcap
 

Then in another window...

dhcp6c -T LL vlan2
 

Then wait a while, let things settle (it can sometimes take up to 120 seconds for DHCPv6 to fully do its thing; sometimes Comcast's servers don't respond to the request immediately). Then once you decide "I hope things are done", you have to do:

scp -p /tmp/filename.pcap user@someboxthatrunssshd:
 

And then look at the file using Wireshark.

It would be a lot easier if I had an inline tap which could look at packets flowing across the WAN port of my RT-N16, but I'd rather not go down to my garage and have to dig out my rack-mounted HP ProCurve switches just to get port mirroring. I do not have any hubs (god forbid anyone! ;-) ).
said by Morac:

The thing I find a bit odd is that Tomato does not appear to use the prefix to assign itself an iPv6 address on the WAN side. You'd think it would need one in order to talk for itself (as opposed to LAN clients) on the WAN side.

And that is exactly what the IA-NA problem/fix is about. By enabling IA-NA in dhcp6c.conf, you do get a single IPv6 address (/128) on vlan2. However, as arkmarkley and other people have stated, this shouldn't be needed.

What should be happening is that Comcast should be delegating a /64 prefix to the customer, which then gets assigned to the br0 interface on TomatoUSB. That part works.

TomatoUSB should then be forwarding any outbound packets (from any IPv6 member on the /64 LAN to the Internet) to the default IPv6 gateway on the router itself.

Likewise, Comcast's routers should know that packets destined to the /64 prefix should be sent to the address delegated with that /64. Let me explain what I mean by that:

Comcast DHCPv6 hands me 2601:9:4600:4f:e2cb:4eff:fec0:c4/64, which gets assigned to br0.

In English (using wildcard syntax), that means 2601:9:4600:4f:* is mine.

Likewise, since Comcast has delegated me 2601:9:4600:4f:e2cb:4eff:fec0:c4 -- which is a single IPv6 address -- their routers should know to send packets destined for 2601:9:4600:4f::/64 to 2601:9:4600:4f:e2cb:4eff:fec0:c4.

Thus, the whole IA-NA fix ("getting a /128 IPv6 WAN IP on vlan2") really shouldn't be required. The TomatoUSB router which gets 2601:9:4600:4f:e2cb:4eff:fec0:c4 should be able to talk to the world using that address, and clients (on the LAN) using that same /64 prefix should be able to talk to the world as well.
said by Morac:

Speaking of the later, it sounds like the router is supposed to assign addresses to the LAN using the prefix with stateless address autoconfiguration and then announce those using router advertisements. That could be why routing is failing.

But that wouldn't explain why in my case (see Linksysinfo.org forum posts) with statically-assigned IPv6 it isn't working.

To me, this almost looks like something on the Comcast side is not configured, route-wise, to forward packets destined to the /64 prefix to the IPv6 address they delegate.

The official Comcast forum thread with people testing IPv6 using a web browser isn't helpful because nobody is showing what actually gets delegated / announced / etc.. The only post I can find that has even remotely useful information is this picture, which is from a customer's router, where he also decided to edit the picture and hide his MAC prefix section:

»forums.comcast.com/t5/im ··· -1&px=-1

VERY IMPORTANT: Note that in that photo, he has a "WAN IPv6 Address" with a /128. The only way to get that is to enable the IA-NA fix I mentioned in this thread (first post), but doing that DOES NOT solve the issue of packets from the /64 prefix LAN going out the Internet and coming back.

I wish that guy would have provided his router's routing table!!

starsilk
@comcast.net

starsilk

Anon

there is an ipv6 enabled version of tcpdump that works fine in tomato here:

»code.google.com/p/wl500g ··· can=2&q=

koitsu
MVM
join:2002-07-16
Mountain View, CA
Humax BGW320-500

koitsu

MVM

said by starsilk :

there is an ipv6 enabled version of tcpdump that works fine in tomato here:

»code.google.com/p/wl500g ··· can=2&q=

Thank you -- yeah, that one seems to have decent support for decoding packets, although it's still incredibly old. I found this to be much more useful and up-to-date:

»multics.minidns.net/blog ··· ilities/

The guy who does all these builds is user rhester72 on the linksysinfo.org forums.

You can download either a statically-linked version of the software you need from here:

»multics.minidns.net/toma ··· -static/

Or you can download dynamically-linked versions from here, but you'll need to know which libraries the binary is linked to (can determine that from ldd or just running the binary and seeing what libxxx.so.x files it complains about):

»multics.minidns.net/toma ··· OMPILED/