dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
17237
share rss forum feed


koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

4 edits

1 recommendation

[IPv6] TomatoUSB and Comcast IPv6 -- bugs found

(Edited as of 06/14, given new discoveries)

Anyone using TomatoUSB and wanting to use IPv6 via Comcast should read the below.

A problem was found which keeps native IPv6 (not 6to4 Anycast) from working on TomatoUSB builds. Shibby and Toastman builds are both confirmed to have this problem as of this writing; other IPv6-aware builds may have this issue too.

Details:

TomatoUSB appears to have a bug where a spurious default route is added for the WAN interface (usually vlan2), where all IPv6 packets going out the default route effectively go to /dev/null.

Permanent fix:

There is currently no permanent fix. The necessary code changes are being discussed with Toastman in this thread at linksysinfo.org. Its recommended that viewers of this thread start from the bottom posts (most recent) and work their way up (older posts).

This section will be updated when a new beta/test Toastman build is available with the fixes.

Workarounds:

Either of the below workarounds should suffice:

* »Re: TomatoUSB and Comcast IPv6 -- bugs found
* »www.linksysinfo.org/index.php?th···t-184504

Place the workaround script in your WAN Up scripts section (Administration -> Scripts -> WAN Up), then reboot your router.

Two DSLR/BBR users ( koitsu See Profile and LMoreland See Profile) have confirmed this fixes the issue for them.

Detailed discussions (highly technical):

* »tomatousb.org/forum/t-484538/com···h-tomato
* »www.linksysinfo.org/index.php?th···t.38006/
* »forums.comcast.com/t5/Home-Netwo···/1306317

History:

It was previously (erroneously) believed the issue was related to the lack of IA-NA bit being set in the DHCPv6 client configuration on TomatoUSB. Use of IA-NA resulted in a single IPv6 address (/128) being assigned to the WAN interface (usually vlan2) on TomatoUSB, thus IPv6 connectivity (only from within the router itself) worked.

This has since been debunked; a single /128 assigned to the WAN interface is unnecessary, since a full /64 prefix is delegated from Comcast via DHCPv6, which is then assigned to the LAN interface (usually br0 on TomatoUSB). That /64 prefix includes an IP address for the router itself, which is publicly-reachable.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.



aefstoggaflm
Open Source Fan
Premium
join:2002-03-04
Bethlehem, PA
kudos:7

Re: TomatoUSB and Comcast IPv6 -- bugs found

When will it be fixed, without using the workarounds?

Thanks.



koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23
reply to koitsu

Unknown at this time. I haven't received a response back from Toastman yet (he's probably at work), and Shibby hasn't commented on the Linksysinfo.org post. So, at this time, no ETR.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.



koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

3 edits
reply to koitsu

I wrote the following workaround which I've confirmed as 100% reliably working on my RT-N16 running Toastman's firmware (specifically tomato-K26USB-1.28.7499MIPSR2Toastman-RT-VPN.trx). TomatoUSB users can put this in their Administration - Scripts - WAN Up section:

#
# Workaround for modifying /etc/dhcp6c.conf to work with Comcast IPv6,
# requiring IA-NA DHCP option bit set.
#
# Note that this workaround intentionally messes up the spacing of
# the send ia-pd option; this is to ensure that if if the sed command
# is run multiple times it won't continue to append the send ia-na entry
# over and over.
#
# http://www.dslreports.com/forum/r27234575-TomatoUSB-and-Comcast-IPv6-bugs-found
#
sed -i -e's/ send ia-pd 0;/send ia-pd 0;\n send ia-na 0;/' /etc/dhcp6c.conf
kill `cat /var/run/dhcp6c.pid` && dhcp6c -T LL `nvram get wan_iface`
#
# Workaround for TomatoUSB bug where a spurious default IPv6 route is
# added for no justified reason, resulting in packets getting forwarded
# effectively to /dev/null.
#
# http://www.dslreports.com/forum/r27234575-TomatoUSB-and-Comcast-IPv6-bugs-found
#
route -A inet6 del default gw :: metric 1024 `nvram get wan_iface`
 

You can then test to make sure its working by rebooting your router or pasting the commands in to a shell (telnet/ssh) window directly. Afterwards, your WAN interface (in my case vlan2) should have an IPv6 address assigned to it labelled Scope:Global. Do not get confused by the Scope:Link entry; that's not the important one.

After that, ping6 ipv6.google.com should work and you should see ICMP responses.

A couple important things:

1. The above may not work correctly for users with complex routing environments (such as individuals using VPN tunnels or doing complex VLANing). In this case, the ping6 test may not work depending on your configuration.

To verify the above WAN Up script is working, all you need to look at is your WAN interface (in my case vlan2) and make sure there is an IPv6 address assigned labelled Scope:Global. If there is, then the WAN Up script I provided is working correctly. If there isn't, then something is amiss with the dhcp6c configuration, or Comcast hasn't deployed IPv6 to your area. See below for further verification.

2. One should copy-paste the WAN Up script into the WAN Up script window; there are backticks used to run subcommands (not apostrophes!) and so on. I hope anyone even reading this thread would know that though. :-)

Validation:

root@gw:/tmp/home/root# uptime
 15:19:26 up 0 min, load average: 0.15, 0.05, 0.01
root@gw:/tmp/home/root# ping6 ipv6.google.com
PING ipv6.google.com (2001:4860:4001:803::1010): 56 data bytes
64 bytes from 2001:4860:4001:803::1010: seq=0 ttl=56 time=56.647 ms
64 bytes from 2001:4860:4001:803::1010: seq=1 ttl=56 time=46.848 ms
64 bytes from 2001:4860:4001:803::1010: seq=2 ttl=56 time=14.746 ms
64 bytes from 2001:4860:4001:803::1010: seq=3 ttl=56 time=64.740 ms
64 bytes from 2001:4860:4001:803::1010: seq=4 ttl=56 time=15.879 ms
64 bytes from 2001:4860:4001:803::1010: seq=5 ttl=56 time=16.359 ms
 
--- ipv6.google.com ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 14.746/35.869/64.740 ms
 

Verifying the dhcp6c.conf changes are working -- note the line with Scope:Global in it:

root@gw:/tmp/home/root# ifconfig vlan2
vlan2      Link encap:Ethernet  HWaddr E0:CB:4E:C0:00:C5
           inet addr:67.180.84.87  Bcast:67.180.87.255  Mask:255.255.252.0
HERE -->   inet6 addr: 2001:558:6045:9b:114c:a6be:d64d:520c/128 Scope:Global
           inet6 addr: fe80::e2cb:4eff:fec0:c5/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:270068 errors:0 dropped:0 overruns:0 frame:0
           TX packets:88814 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:33258559 (31.7 MiB)  TX bytes:9207877 (8.7 MiB)
 

--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


LMoreland

join:2010-07-28
Littleton, CO

1 edit

Didn't work for me. I am running an E4200 with Tomato Firmware v1.28.0499 MIPSR2Toastman-RT-N K26 USB VPN
and I have my IPv6 settings to DHCPv6 with prefix delegation with only the accept RA from WAN ticked.



Morac
Cat god

join:2001-08-30
Riverside, NJ
kudos:1
Reviews:
·Comcast

1 edit
reply to koitsu

Just a FYI, but you should be able to replace this line:

kill `cat /var/run/dhcp6c.pid`

with

killall dhcp6c

You might want to also add a check to make sure dhcp6c is even running before just starting it.

Oh and if the WAN connection goes down, the script will run again and add another "send ia-na 0;" line to the dhcp6c.conf file, even if that line is already there. Though I think that file gets generated on WAN UP so that might be okay.
--
The Comcast Disney Avatar has been retired.



koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

said by Morac:

Just a FYI, but you should be able to replace this line:

kill `cat /var/run/dhcp6c.pid`

with

killall dhcp6c

No. There are repercussions in that case; this is a very, very bad habit to get into in shell scripting in general. Been doing this stuff since 1991. :-)

said by Morac:

You might want to also add a check to make sure dhcp6c is even running before just starting it.

I can do that. It would be as simple as:

kill `cat /var/run/dhcp6c.pid` && dhcp6c -T LL `nvram get wan_iface`
 

Checking to see if dhcp6c is already running is more or less pointless given the scope of who would be using this WAN Up script. TomatoUSB users who aren't using an IPv6 firmware and who have not enabled IPv6 would not need this script (nor should they use it). I thought this was implied, but oh well. I would rather not have to add if [ -r /var/run/dhcp6c.pid ]; then ... ; fi statements around things.

said by Morac:

Oh and if the WAN connection goes down, the script will run again and add another "send ia-na 0;" line to the dhcp6c.conf file, even if that line is already there.

Incorrect. Please take the time to read the comment that precedes the sed statement. The sed statement is written very carefully.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

1 edit
reply to LMoreland

said by LMoreland:

Didn't work for me. I am running an E4200 with Tomato Firmware v1.28.0499 MIPSR2Toastman-RT-N K26 USB VPN and I have my IPv6 settings to DHCPv6 with prefix delegation with only the accept RA from WAN ticked.

I need output from the following commands on the router:

* ifconfig -a
* ip -6 route show
* nvram get wan_iface
* nvram get script_wanup
* tail -100 /var/log/messages

Thanks.

EDIT: I'm working with LMoreland See Profile on this. We're discussing it privately via Email.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


LMoreland

join:2010-07-28
Littleton, CO

1 edit

can you send me your email address and I will send all of that to you. You can pm it to me from this or the linksys forum. I have replied to your thread pointing here in the linksys forum.

Thanks



HunterZ

join:2003-07-16
Kent, WA
reply to koitsu

I've heard that Toastman may be only communicating on linksysinfo and not tomatousb.org these days, so you may want to try to contact him over there.



koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

said by HunterZ:

I've heard that Toastman may be only communicating on linksysinfo and not tomatousb.org these days, so you may want to try to contact him over there.

Yeah, Linksysinfo.org is where I contacted him privately. Thanks for the tip though!
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


HunterZ

join:2003-07-16
Kent, WA

No problem. Looks like someone also already posted in the latest Toastman build thread on linksysinfo: »www.linksysinfo.org/index.php?th···t-184391



rgolds

join:2012-06-13
Philadelphia, PA
reply to koitsu

Thanks for all the work you've put into this, koitsu! Using your script, my router is now assigned a native IPv6 address (2001:558:...), and IPv6 connectivity from the router to WAN works perfectly (ping6 via telnet to ipv6.google.com works, etc).

However, for some reason, none of my LAN devices are being assigned valid/working IPv6 addresses. The assigned address prefix is 2601:b:..., and there is no IPv6 connectivity at all. I'm new to Tomato (and IPv6), so I feel like I'm just missing a setting or otherwise doing something stupid. One of the lines from my router logs:

unknown daemon.err httpd[3398]: bind: [2601:b:a780:31:9644:52ff:fea6:cdfc]:80: Cannot assign requested address

Any ideas?



koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

1 edit

The "LAN ordeal", meaning getting actual IPv6 packets from a machine on one's local network to talk to the Internet, is a separate problem that I still cannot figure out. I have spent the entire day working on this and I'm no where closer to where I was before. Other people have IM'd me and sent me Email complaining about the same thing -- I can't help.

I'd love to wrap my hands around the committees/individuals that designed IPv6. The amount of obfuscation and nonsense going on under the hoo3 is almost embarrassing. It's about the most non-KISS compliant protocol I've worked with yet, other than, say, IPsec. :P

The httpd error you see indicates that httpd (that's the web server that runs on the router itself, used for configuring the GUI bits, etc.) cannot bind to an IPv6 address. The reason for this is that, if I remember right, the httpd binary on TomatoUSB is not built properly with IPv6 support. All this would impact is being able to reach your TomatoUSB router's GUI natively via IPv6. I get the exact same message, but for my Internet-facing IPv6 address. I'm not concerned about httpd in the least bit.

At this point I have quite honestly given up on trying to get this crap to work. I will paste what I've got so far though:

Comcast is handing out a /128 prefix (read: single IPv6 address) for the "public Internet IP" portion, and also handing off a /64 prefix to each customer. On my router, the /128 prefix portion gets assigned to vlan2, and the /64 portion gets to assigned to br0. vlan2 = Internet, br0 = LAN. Here's the relevant ifconfig bits:

br0        Link encap:Ethernet  HWaddr E0:CB:4E:C0:00:C4
           inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
           inet6 addr: 2601:9:4600:4f:e2cb:4eff:fec0:c4/64 Scope:Global
           inet6 addr: fe80::e2cb:4eff:fec0:c4/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:136169 errors:0 dropped:0 overruns:0 frame:0
           TX packets:140120 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:12825633 (12.2 MiB)  TX bytes:34201252 (32.6 MiB)
 
vlan2      Link encap:Ethernet  HWaddr E0:CB:4E:C0:00:C5
           inet addr:67.180.84.87  Bcast:67.180.87.255  Mask:255.255.252.0
           inet6 addr: 2001:558:6045:9b:114c:a6be:d64d:520c/128 Scope:Global
           inet6 addr: fe80::e2cb:4eff:fec0:c5/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:395630 errors:0 dropped:0 overruns:0 frame:0
           TX packets:130505 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:45129459 (43.0 MiB)  TX bytes:13136168 (12.5 MiB)
 

The first thing to notice here -- and this is what confuses a lot of people, myself included -- is that there are two IPv6 addresses/prefixes assigned to each interface. Note the difference however: one is Scope:Link and the other is Scope:Global.

IPv6 is different than IPv4 in this respect. IPv6 provides what is called a "link-local" address to every network interface. This interface is supposed to represent a LAN segment, basically.

The Scope:Global address assigned to br0 -- in my case, 2601:9:4600:4f:e2cb:4eff:fec0:c4 -- is being handed to me from Comcast. The first 64 bits (so 2601:0009:4600:004f) are Comcast, while the last 64 bits are a combination of my MAC address for that interface. Wikipedia explains this in greater detail.

Based on this, one should be able to take a machine on a LAN segment (e.g. a Windows 7 box) and give it an address within that /64 prefix associated with br0. For example, give it 2601:9:4600:4f:230:48ff:fed2:22d0.

Then, give that same machine a default IPv6 route of the br0 interface's IPv6 address -- although I'm told by people more familiar with IPv6 that technically it should be using the link-local address for br0 (which doesn't work for me; I get "network unreachable" if I try that) -- so, 2601:9:4600:4f:e2cb:4eff:fec0:c4.

IPv6 packets from the Windows box should then go to the br0 interface on the router, then be sent out via Comcast. At least that's how I understand it to be (and I'm only familiar with IPv4).

Except that doesn't work.

LAN traffic works fine -- meaning 2601:9:4600:4f:e2cb:4eff:fec0:c4 and 2601:9:4600:4f:230:48ff:fed2:22d0 can talk just fine -- but packets going out across the Internet never get a response.

My colleague who has working IPv6, across the Internet, attempted to ping 2601:9:4600:4f:e2cb:4eff:fec0:c4 to see what he got back. This is the result:

PING 2601:9:4600:4f:e2cb:4eff:fec0:c4(2601:9:4600:4f:e2cb:4eff:fec0:c4) 56 data bytes
From 2001:558:82:a4::2 icmp_seq=1 Time exceeded: Hop limit
 

I immediately asked for a traceroute6, because to me that means a routing loop. Sure enough:

10  pos-0-12-0-0-ar01.sfsutro.ca.sfba.comcast.net (2001:558:0:f697::2)  50.183 ms  50.184 ms  50.213 ms
11  te-9-2-ur03.santaclara.ca.sfba.comcast.net (2001:558:80:178::2)  46.067 ms  46.294 ms  46.421 ms
12  * * *
13  te-0-0-0-12-ur05.santaclara.ca.sfba.comcast.net (2001:558:82:87::1)  45.498 ms  45.569 ms  45.672 ms
14  * * *
15  te-5-5-ur03.santaclara.ca.sfba.comcast.net (2001:558:82:87::1)  45.777 ms  45.535 ms  45.909 ms
16  * * *
17  te-0-0-0-12-ur05.santaclara.ca.sfba.comcast.net (2001:558:82:87::1)  45.923 ms  46.036 ms  46.084 ms
 
...repeat indefinitely...
 

The reason this is occurring seems to be that Comcast's routers either don't know what to do with packets destined to the network they delegated to me, or (and this is much more likely), TomatoUSB is very busted/broken and isn't properly handling the routing situation correctly, which causes Comcast's routers to punt the packet to another router, try again, rinse lather repeat.

Again, IPv6 is very complex. I realise everyone wants to look at it as a very simple "get an IP, get a network block, go to town" thing, but there is a lot under the hood that is different than IPv4. The rabbit hole quite honestly doesn't end.

I'm tempted at this point -- much like another user here on this forum -- to just use the 6to4 Anycast option instead and see if one can get that working with a LAN segment. There's a piece to this puzzle which I do not get, and honestly I would need a decent network engineer to assist in the matter -- but at this point I'm out of breath.

--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


rgolds

join:2012-06-13
Philadelphia, PA

1 edit

Wow, thanks for that exhaustive reply! I hope that people here smarter than I can pick up and run with it... and/or when you get a chance to sleep on it, some solutions may occur to you.

said by koitsu:

I'm tempted at this point -- much like another user here on this forum -- to just use the 6to4 Anycast option instead and see if one can get that working with a LAN segment.

I believe I have 6to4 Anycast Relay working properly. I just used the default settings: prefix length 64, static DNS blank, enable router advertisements checked, relay anycast address 192.88.99.1, tunnel MTU 0 (default MTU), tunnel TTL 255.

Obviously, my router doesn't get a WAN-facing IPv6 IP, but it assigns IPv6 addresses to all devices on my LAN, and they work. One device was assigned 2002:473a:21c:1:816e:6572:e33d:78aa, another was assigned 2002:473a:21c:1:5042:ce0f:466f:3cc6. I can successfully ping ipv6.google.com, etc. The tracert from the former to ipv6.google.com is as follows:

Tracing route to ipv6.l.google.com [2001:4860:800a::6a] over a maximum of 30 hops:
 
  1    <1 ms    <1 ms    <1 ms  2002:473a:21c:1::1
  2    39 ms    19 ms    19 ms  2002:c058:6301::
  3    37 ms    51 ms    62 ms  ge-2-41-ur03.ndceast.pa.bo.comcast.net [2001:558:fe14:1::1]
  4    57 ms    29 ms    23 ms  te-1-3-ar01.ndceast.pa.bo.comcast.net [2001:558:e0:14::2]
  5    20 ms    34 ms    43 ms  te-1-3-ar01.philadelphia.pa.bo.comcast.net [2001:558:e0:3::1]
  6    48 ms    23 ms    79 ms  te-0-0-0-9-cr01.newyork.ny.ibone.comcast.net [2001:558:0:f745::1]
  7    23 ms    30 ms    23 ms  pos-1-5-0-0-pe01.111eighthave.ny.ibone.comcast.net [2001:558:0:f5d8::2]
  8    54 ms    39 ms    19 ms  2001:559::366
  9    20 ms    21 ms    21 ms  2001:4860::1:0:755
 10    31 ms    20 ms    20 ms  2001:4860::8:0:2fc6
 11    29 ms    26 ms    26 ms  2001:4860::1:0:9ff
 12    55 ms    39 ms    32 ms  2001:4860::8:0:3005
 13    55 ms    40 ms    39 ms  2001:4860::8:0:2f04
 14    36 ms    39 ms    40 ms  2001:4860::2:0:a7
 15    39 ms    48 ms    37 ms  2001:4860:0:1::10b
 16    47 ms    55 ms    57 ms  yx-in-x6a.1e100.net [2001:4860:800a::6a]
 

If you'd like me to try anything else and copy the results here, let me know.

-Ryan


koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

1 recommendation

Thanks Ryan -- yeah, I had a feeling the 6to4 Anycast Relay would work. However, from everything I've read, the throughput is unbearably slow (I'm still trying to find the articles I read today on that problem; my browser history/cache is impossible to sift through). In fact, I can confirm that using the 6to4 Anycast Relay and pinging from the TomatoUSB router itself to ipv6.google.com intermittently sees packet loss, regardless of what address it gets back for that FQDN (it round-robins).

Given that Comcast is doing native IPv6 I would much rather use that, obviously. Otherwise I'm forced to set up 6to4 on all of my LAN machines (some are Windows, others are FreeBSD and Linux), so it's somewhat of an administrative nightmare.

Regardless of my situation, I hope the information you've provided is helpful to others who at least want to just get *something* IPv6-wise.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.



koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

1 recommendation

reply to koitsu

For those following this thread (and I imagine there is many since I posted it everywhere relevant, haha ) I thought I'd provide a status update as to where we are, what's being discussed, ideas, etc... In no particular order:

1. Toastman and I have talked privately on the linksysinfo.org forum. During our discussion another user showed up and posted something that looked like a key piece of information (keep reading). Toastman at this point is willing to build firmwares for people to download/test, assuming there are folks who have good ideas as to what the bugs/issues are and what to change.

The sad part of our conversation is that IPv6 support in TomatoUSB in general is neglected big time. There was a really key person who was IPv6-savvy and part of the Tomato/TomatoUSB project who has since disappeared (hasn't responded to Emails, etc.), which is a bummer given his extensive knowledge of the protocol. (I looked the guy up myself and he even provided patches/code to the Linux iptables folks to fix problems). Toastman and I both wish we could find this guy.

2. An individual on the tomatousb.org and linksysinfo.org forums named armarkey mentioned that the "spurious default route" actually isn't so spurious after all, and that it might be related to metrics. Metrics define what route (or interface, depending on context) get preferred over another; think of it as a preferencing order. Two routes with an identical metric result in the route which got defined first taking precidence.

armarkey's theory is that because the IPv6 ICMP-based RA (route announcement) default metric is 1024, and that the DHCPv6 client's default route metric is 1024 (which is created the instant the interface is brought up), this causes a situation where the latter route was defined first, thus gets used when in fact the ICMP-based RA route should be being used.

I tried solving the problem by removing the metric 1024 route (same route you see in my first post) and inserting the same route with a metric of 2048, but to no avail.

So we're still working on that to figure out what the cause is, if the metric is actually responsible for the problem, yadda yadda.

3. armarkey also pointed out that the IA-NA workaround which was proposed by Comcast forum user morrildl may actually be incorrect in general. All this does is, in addition to Comcast handing out a /64 network to the DHCPv6 client in TomatoUSB, it also hands a /128 (single IPv6 address).

The theory is that the /128 isn't needed at all. That and the default route fix do make it so you can talk to the Internet via IPv6 directly inside TomatoUSB, but technically one should be able to do that anyway since br0 is assigned an address as part of the /64.

I am in agreement with this theory -- but of course, in practise, it doesn't work and we're still not sure why.

So for now, generally speaking, the IA-NA workaround is probably not needed.

I can absolutely confirm, hands down, no questions asked, that if you run TomatoUSB without the IA-NA workaround that you do in fact get handed a /64 network from Comcast. This means the DHCPv6 request in TomatoUSB is in fact working correctly, and that the problem may be elsewhere.

That leads me to the final item...

4. Most everyone I've been hearing from (privately and from reading public forums) has the exact same problem when using TomatoUSB's "DHCPv6 with Prefix Delegation" mode -- specifically it's that IPv6-capable machines on the LAN segment simply don't work.

I spent some time tonight poking at this (see my long post a few pages up) and didn't get anywhere. The IA_NA fix and default route fix don't resolve this problem. All I'm able to confirm at this point is that packets from a machine on the LAN segment do get sent to the TomatoUSB router correctly, and the TomatoUSB router does in fact send the packets out the WAN interface. However, no responses are ever seen. My guess is that the response packets never arrive because of the routing loop ordeal I mentioned a few posts up.

So we're still researching that issue as well.
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.



rgolds

join:2012-06-13
Philadelphia, PA

1 edit

1 recommendation

reply to koitsu

I can confirm that IPv6 via a 6to4 relay is, indeed, unbearably slow. I just ran the test at »ipv6-test.com/speedtest/ - my IPv4 speed is 16.4 Mbit/s, and my IPv6 speed is 148 Kbit/s... over two orders of magnitude slower. I haven't done any long-term testing, so I can't speak to the reliability of the connection.

You mentioned that you'd be forced to set up 6to4 on your LAN machines with a 6to4 router setup; that's not the case. From the perspective of LAN devices, they have a native IPv6 connection. The router has a LAN-facing IPv6 address (2002:473a:21c:1::1), and it assigns valid and functional IPv6 addresses to the LAN devices with the same 2002:473a:21c:1 prefix. Several devices on my network with varying operating systems, all set to retrieve addresses via DHCP, were successfully assigned their own IPv6 addresses without any configuration after setting up 6to4 on the router.

Obviously, native IPv6 would be better, and that's what I've been trying to do too.

With the combination of a working WAN interface using your script and a working LAN interface using the 6to4 relay, I feel like we're getting close to a comprehensive solution!



RR Conductor
NWP RR Inc.,serving NW CA
Premium
join:2002-04-02
Redwood Valley, CA
kudos:1
reply to koitsu

I wonder if this is related to why Native IPv6 isn't working on the Asus RT-N66U? I'm working with Netdog on it, and a person over at the Asus Forums, but so far it just refuses to work. The only IPv6 I can get on it is using Comcast's 6to4 Relay and 6in4 Tunnel to Hurricane Electric.



RR Conductor
NWP RR Inc.,serving NW CA
Premium
join:2002-04-02
Redwood Valley, CA
kudos:1
reply to rgolds

said by rgolds:

I can confirm that IPv6 via a 6to4 relay is, indeed, unbearably slow. I just ran the test at »ipv6-test.com/speedtest/ - my IPv4 speed is 16.4 Mbit/s, and my IPv6 speed is 148 Kbit/s... over two orders of magnitude slower. I haven't done any long-term testing, so I can't speak to the reliability of the connection.

I just ran a test using Comcast's 6to4 Relay, and got this-



--
»www.amtrak.com
»www.freightrailworks.org
»www.isu.edu
»www.northcoastrailroad.org
»www.amtrakcalifornia.com
»www.cahighspeedrail.gov


RR Conductor
NWP RR Inc.,serving NW CA
Premium
join:2002-04-02
Redwood Valley, CA
kudos:1

My setup-

Zoom 5341J Modem
Asus RT-N66U Router running Merlin's custom build of Asus's 3.0.0.3.108 FW.

I was thinking of trying out Tomato USB on my router, I am glad now I held off.



whfsdude
Premium
join:2003-04-05
Washington, DC
Reviews:
·Comcast
reply to RR Conductor

said by RR Conductor:

I just ran a test using Comcast's 6to4 Relay, and got this-

Speed tests are going to be highly inconsistent on 6to4. Comcast's relay is only for outbound traffic on 6to4 and has nothing to do with your inbound 6to4 path. That's determined by the network you're trying to connect to. They use their nearest 6to4 relay to send to you.


whfsdude
Premium
join:2003-04-05
Washington, DC
Reviews:
·Comcast

2 edits
reply to koitsu

Can you post the output of these three commands. I've set DHCPv6 w/ DHCPv6-PD on linux before so I might be able to help.

route -6 -n
brctl show
ifconfig -a

My guess is you're trying to route the PD prefix to Comcast's gateway when in fact its gateway is your point to point /128.

Edit for clarity:
/64's default gateway is your /128.

So you shouldn't need an outbound route for the /64 link as it should be the same as a connected route. Inbound you will need a route pointing the /64 to the br interface.



koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

2 edits

1 recommendation

Output in question is here, sans brctl.

»tomatousb.org/forum/t-484538/com···-1479712

brctl output is plain and simple, nothing crazy, and doesn't involve vlan2 (you wouldn't want to bridge that!).

root@gw:/tmp/home/root# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.e0cb4ec000c4       no              vlan1
                                                        eth1
 

Edit for clarity:
/64's default gateway is your /128.

We don't get a /128 on vlan2 unless the IA-NA fix (see first post of this thread) is put in place. And even if that fix is put in place, the default IPv6 gateway (::/0) chosen per IPv6 RA ends up being fe80::201:5cff:fe32:6e41 (which is a link-local address completely unrelated to anything on the router itself). Proof of that is available in this thread.

Edit: Actually, that proof isn't quite valid given the preceding statement in the same paragraph. That proof is without the IA-NA fix and without the default route fix in place (you can see that from the fact that the vlan2 interface has no /128 and the "spurious ::/0" route).

Welcome to the mess. :-) You might consider taking part in this thread over on the linksysinfo.org forums where armarkey and I are trying to figure out what's actually going on under the hood:

»www.linksysinfo.org/index.php?th···t.38006/
--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


HunterZ

join:2003-07-16
Kent, WA

1 recommendation

reply to koitsu

koitsu, thanks a ton for keeping everyone posted on this. Hopefully it will be ironed out by the time Comcast turns on ipv6 in my area



Morac
Cat god

join:2001-08-30
Riverside, NJ
kudos:1
Reviews:
·Comcast
reply to koitsu

I've been trying to read up on how this is all supposed to work ( »manpages.ubuntu.com/manpages/har···f.5.html, »www.ietf.org/rfc/rfc3315.txt, »www.ietf.org/rfc/rfc3633.txt ) and from my understanding the delegating router (i.e. Comcast) is supposed to assign the requesting router (Tomato) a prefix. It's up to the requesting router to divvy that up into subnets and addresses and give it to local clients.

If Tomato is getting the prefix, which it sounds like it is, then Comcast should know to route any responses back to addresses that falls in that Prefix to Tomato, unless Tomato is mangling the outbound requests, but you already tested trying to do an inbound traceroute (which failed) so that seems unlikely. As such it sounds like the initial DHCPv6 setup is screwy since the delegating router doesn't know where the requesting router is.

Has anyone dumped out the actual DHCPv6 requests going back and forth?

The thing I find a bit odd is that Tomato does not appear to use the prefix to assign itself an iPv6 address on the WAN side. You'd think it would need one in order to talk for itself (as opposed to LAN clients) on the WAN side.

Speaking of the later, it sounds like the router is supposed to assign addresses to the LAN using the prefix with stateless address autoconfiguration and then announce those using router advertisements. That could be why routing is failing.

One thing I did notice looking at the code is that there is a "debug_ipv6" nvram setting which appears to enable debugging for both dhcp and radvd (but not dhcpv6).

Most of the code regarding DHCP and router advertisements is in dhcp.c and services.c in »repo.or.cz/w/tomato.git/tree/ref···outer/rc

DHCPv6 is implemented in »repo.or.cz/w/tomato.git/tree/ref···r/dhcpv6

ipV6 is implemented as part of Linux in »repo.or.cz/w/tomato.git/tree/ref···net/ipv6
--
The Comcast Disney Avatar has been retired.



koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

said by Morac:

I've been trying to read up on how this is all supposed to work ( »manpages.ubuntu.com/manpages/har···f.5.html, »www.ietf.org/rfc/rfc3315.txt, »www.ietf.org/rfc/rfc3633.txt ) and from my understanding the delegating router (i.e. Comcast) is supposed to assign the requesting router (Tomato) a prefix. It's up to the requesting router to divvy that up into subnets and addresses and give it to local clients.

If Tomato is getting the prefix, which it sounds like it is, then Comcast should know to route any responses back to addresses that falls in that Prefix to Tomato, unless Tomato is mangling the outbound requests, but you already tested trying to do an inbound traceroute (which failed) so that seems unlikely. As such it sounds like the initial DHCPv6 setup is screwy since the delegating router doesn't know where the requesting router is.

Has anyone dumped out the actual DHCPv6 requests going back and forth?

Yes, I have done this. Let me explain the annoyance however:

The problem with using tcpdump on TomatoUSB is that it (and libpcap) is extremely old and does not have decent IPv6 support. I'm using the one available via ipkg, which come from »ipkg.nslu2-linux.org/feeds/optwa···/stable/ --

libpcap - 1.0.0-2 - PCAP Library
tcpdump - 4.0.0-1 - tcpdump dumps the traffic on a network
 

Old. Ancient. Horrible. The problem is that these can detect IPv6 packets (based on protocol header) but it cannot decode any of the IPv6 header data/etc.. This requires one to do the following:

kill `cat /var/run/dhcp6c.pid`
tcpdump -p -l -n -s 0 -i vlan2 -w /tmp/filename.pcap
 

Then in another window...

dhcp6c -T LL vlan2
 

Then wait a while, let things settle (it can sometimes take up to 120 seconds for DHCPv6 to fully do its thing; sometimes Comcast's servers don't respond to the request immediately). Then once you decide "I hope things are done", you have to do:

scp -p /tmp/filename.pcap user@someboxthatrunssshd:
 

And then look at the file using Wireshark.

It would be a lot easier if I had an inline tap which could look at packets flowing across the WAN port of my RT-N16, but I'd rather not go down to my garage and have to dig out my rack-mounted HP ProCurve switches just to get port mirroring. I do not have any hubs (god forbid anyone! ;-) ).

said by Morac:

The thing I find a bit odd is that Tomato does not appear to use the prefix to assign itself an iPv6 address on the WAN side. You'd think it would need one in order to talk for itself (as opposed to LAN clients) on the WAN side.

And that is exactly what the IA-NA problem/fix is about. By enabling IA-NA in dhcp6c.conf, you do get a single IPv6 address (/128) on vlan2. However, as arkmarkley and other people have stated, this shouldn't be needed.

What should be happening is that Comcast should be delegating a /64 prefix to the customer, which then gets assigned to the br0 interface on TomatoUSB. That part works.

TomatoUSB should then be forwarding any outbound packets (from any IPv6 member on the /64 LAN to the Internet) to the default IPv6 gateway on the router itself.

Likewise, Comcast's routers should know that packets destined to the /64 prefix should be sent to the address delegated with that /64. Let me explain what I mean by that:

Comcast DHCPv6 hands me 2601:9:4600:4f:e2cb:4eff:fec0:c4/64, which gets assigned to br0.

In English (using wildcard syntax), that means 2601:9:4600:4f:* is mine.

Likewise, since Comcast has delegated me 2601:9:4600:4f:e2cb:4eff:fec0:c4 -- which is a single IPv6 address -- their routers should know to send packets destined for 2601:9:4600:4f::/64 to 2601:9:4600:4f:e2cb:4eff:fec0:c4.

Thus, the whole IA-NA fix ("getting a /128 IPv6 WAN IP on vlan2") really shouldn't be required. The TomatoUSB router which gets 2601:9:4600:4f:e2cb:4eff:fec0:c4 should be able to talk to the world using that address, and clients (on the LAN) using that same /64 prefix should be able to talk to the world as well.

said by Morac:

Speaking of the later, it sounds like the router is supposed to assign addresses to the LAN using the prefix with stateless address autoconfiguration and then announce those using router advertisements. That could be why routing is failing.

But that wouldn't explain why in my case (see Linksysinfo.org forum posts) with statically-assigned IPv6 it isn't working.

To me, this almost looks like something on the Comcast side is not configured, route-wise, to forward packets destined to the /64 prefix to the IPv6 address they delegate.

The official Comcast forum thread with people testing IPv6 using a web browser isn't helpful because nobody is showing what actually gets delegated / announced / etc.. The only post I can find that has even remotely useful information is this picture, which is from a customer's router, where he also decided to edit the picture and hide his MAC prefix section:

»forums.comcast.com/t5/image/serv···-1&px=-1

VERY IMPORTANT: Note that in that photo, he has a "WAN IPv6 Address" with a /128. The only way to get that is to enable the IA-NA fix I mentioned in this thread (first post), but doing that DOES NOT solve the issue of packets from the /64 prefix LAN going out the Internet and coming back.

I wish that guy would have provided his router's routing table!!

--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23
reply to koitsu

HOLY CRAP, I GOT IT WORKING! (packets via LAN)

$ uname -a
FreeBSD icarus.home.lan 9.0-STABLE FreeBSD 9.0-STABLE #0: Fri Jun  8 18:09:07 PDT 2012     root@icarus.home.lan:/usr/obj/usr/src/sys/X7SBA_RELENG_9_amd64  amd64
$ ping6 ipv6.google.com
PING6(56=40+8+8 bytes) 2601:9:4600:4f:230:48ff:fed2:22d0 --> 2001:4860:4001:800::1011
16 bytes from 2001:4860:4001:800::1011, icmp_seq=0 hlim=55 time=74.223 ms
16 bytes from 2001:4860:4001:800::1011, icmp_seq=1 hlim=55 time=78.878 ms
16 bytes from 2001:4860:4001:800::1011, icmp_seq=2 hlim=55 time=130.185 ms
^C
--- ipv6.l.google.com ping6 statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 74.223/94.429/130.185/25.355 ms
 

Still doing more testing though... please hold...

--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


koitsu
Premium,MVM
join:2002-07-16
Mountain View, CA
kudos:23

1 edit

2 recommendations

Okay, a follow-up to my own "IT WORKS!" post with some details.

I can confirm that at this point:

1. My TomatoUSB router is able to talk IPv6 to the Internet directly (e.g. ping6 ipv6.google.com from TomatoUSB natively works),

2. My FreeBSD box on my LAN is able to talk IPv6 to the Internet directly (e.g. ping6 ipv6.google.com from FreeBSD works). IMPORTANT: My FreeBSD box is statically configured for IPv6, it is not configured to dynamically get an IPv6 prefix/etc. from the TomatoUSB router.

I have rebooted my RT-N16 and done the manual fix-ups needed and it does in fact work reliably.

Now for the details:

1. The IA-NA fix (in dhcp6c.conf) ISN'T NEEDED. And that seems correct/logical given what I described in my earlier post. There is absolutely no need for a /128 address on the WAN interface (vlan2).

2. The "spurious default route" fix IS NEEDED. Removing the spurious route is absolutely required. Simply put: there should be only one default route for ::/0 and it should be an fe80::xxx address (negotiated via IPv6 RAs announced from Comcast).

3. Under Basic / IPv6, the WAN checkbox for "Accept RA from" must be checked. This is the only way to ensure that the TomatoUSB router gets a default IPv6 gateway from Comcast (DHCPv6 does not negotiate this; its announced via IPv6 RAs. This greatly differs from classic IPv6 DHCP, for those familiar with it).

4. Under Basic / IPv6, the "Enable Router Advertisements" checkbox is probably required for systems on a LAN which don't have statically configured IPv6 addresses and default IPv6 gateways. This checkbox makes it so that IPv6 RAs (from TomatoUSB to the LAN) are sent across the LAN. It has no bearing on the IPv6 RAs received via WAN from COmcast.

5. For statically-configured IPv6 machines on a LAN (like my FreeBSD box) only, it's very important that for the default gateway you ensure that you use the link-local IPv6 address of the TomatoUSB system (that would be the "Scope:Link" IPv6 address shown for interface br0), and that you use a zone index using the %index syntax as described here:

»en.wikipedia.org/wiki/IPv6_addre···_indices

Without use of the zone index, you cannot do something like "route add -inet6 default fe80::e2cb:4eff:fec0:c4" because the system has no idea what interface (zone index) is associated with the fe80::xxx address. On FreeBSD, without the zone index specifier, you get an error such as "Network unreachable" when trying to add the default route.

In the case of machines which are not statically routed, I imagine that IPv6 RAs (received from the TomatoUSB router across the LAN) should negotiate all of this stuff dynamically. I haven't gotten to that phase yet; I imagine that is the phase/methodology that most of the people on this forum will use, but for my setup I cannot use it at this point in time (has to do with issues/complexities with FreeBSD and what it does when recieving IPv6 RAs). Baby steps!

So, the WAN Up script I'm using now is the following, again, with 100% success (including after a reboot):

#
# Workaround for TomatoUSB bug where a spurious default IPv6 route is
# added for no justified reason, resulting in packets getting forwarded
# effectively to /dev/null.
#
# 1. Temporarily disable accepting IPv6 RAs on the WAN interface.  This
#    will stop the kernel from automatically adding a default IPv6 route
#    when such an RA is received via the WAN.
# 2. Delete ALL default IPv6 routes.  In effect this deletes the spurious
#    IPv6 default route, as well as any default IPv6 routes received via RA.
#    Sadly the "ip" command does not give you a way to differentiate between
#    the two, since the one we truly want to delete lacks "proto kernel".
# 3. Restore honouring IPv6 RAs via the WAN.  Within 60-120 seconds (often
#    within seconds on Comcast) a default IPv6 route should be added by the
#    kernel.  You can use "ip -6 route show default dev `nvram get wan_iface`"
#    to verify; you should have only one route ("default via fe80::xxx ...").
#
# http://www.dslreports.com/forum/r27234575-TomatoUSB-and-Comcast-IPv6-bugs-found
#
echo 0 > /proc/sys/net/ipv6/conf/`nvram get wan_iface`/accept_ra
ip -6 route flush default dev `nvram get wan_iface`
echo 2 > /proc/sys/net/ipv6/conf/`nvram get wan_iface`/accept_ra
 

And before criticising the script ( :-) ) please be sure to read the comments in the script; its written this way for a reason.

For those who have tried/used the IA-NA fix previously mentioned, please replace your WAN Up script with the one above and then reboot the router. (Yes, there is a way to do this without rebooting, but the instructions would be long and I'd rather not explain it. It involves editing /etc/dhcp6c.conf to remove the ia-na bit, restarting dhcp6c with its previous arguments, then running the above WAN Up script by hand)

--
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


NathanO

join:2008-08-21
Moorestown, NJ

Can you post your ifconfig output?