dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
2164
share rss forum feed


dennismurphy
Put me on hold? I'll put YOU on hold
Premium
join:2002-11-19
Parsippany, NJ
kudos:2
Reviews:
·Verizon FiOS

[Networking] Is 10.0.0.0/8 not blackholed?

What the heck is going on? I'm trying to troubleshoot my OpenVPN settings. I have my server set as 10.10.10.1 and the tunnel IPs elsewhere in the 10.10.10.0/24 subnet.

Even when the tunnel is down, I was getting ICMP replies from 10.10.10.1. That's very bizarre.... so I'm checking my LAN, and, well, nothing is up with 10.10.10.1 on my LAN. So FiOS is clearly routing the packets. The ping replies are coming from off my LAN ......

It's not in my arp table, it's tracerouteable, it's pingable... why the heck can I get there from here?

I'll readdress my openvpn config - not a big deal - but that's bothering me. 192.168.0.0/16, 172.16.0.0/12, and 10.0.0.0/8 should be blackholed on the FiOS network yet it isn't. Furthermore, the Actiontec should have a default rule to drop any packets from those source networks on the WAN port - those should be invalid.

Is anyone else seeing this behavior?

I'll have to add some rules to the Actiontec to drop the packets, but can anyone else verify they're seeing this as well?

If I can get someone else to verify they can get to 10.10.10.1, I'll make some phone calls.


dmurphy@macbookpro: ping -c 2 10.10.10.1
PING 10.10.10.1 (10.10.10.1): 56 data bytes
64 bytes from 10.10.10.1: icmp_seq=0 ttl=251 time=13.430 ms
64 bytes from 10.10.10.1: icmp_seq=1 ttl=251 time=12.207 ms

--- 10.10.10.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 12.207/12.819/13.430/0.612 ms

dmurphy@macbookpro: traceroute 10.10.10.1
traceroute to 10.10.10.1 (10.10.10.1), 64 hops max, 52 byte packets
1 dmurphy-router.home (172.16.0.1) 3.973 ms 3.719 ms 1.030 ms
2 l100.nwrknj-vfttp-101.verizon-gni.net (98.109.156.1) 5.406 ms 4.884 ms 4.879 ms
3 g0-9-2-6.nwrknj-lcr-22.verizon-gni.net (130.81.110.100) 7.306 ms 7.314 ms 7.623 ms
4 ae2-0.nwrk-bb-rtr2.verizon-gni.net (130.81.209.170) 7.378 ms
ae4-0.nwrk-bb-rtr2.verizon-gni.net (130.81.199.194) 9.298 ms
ae0-0.nwrk-bb-rtr2.verizon-gni.net (130.81.209.162) 6.822 ms
5 * 3.et-2-0-1.tl2.nyc1.alter.net (140.222.227.34) 11.670 ms 6.838 ms
6 gigabitethernet7-0.gw9.nyc4.alter.net (152.63.21.194) 12.561 ms * 14.750 ms

dmurphy@macbookpro: arp -a|grep 10.
dmurphy@macbookpro:


chrisb3127

join:2006-03-01
Manville, NJ

Yep, I am able to ping 10.10.10.1 and do a successful traceroute. I thought these networks were reserved for LAN use and should not be routable outside of the LAN?



dennismurphy
Put me on hold? I'll put YOU on hold
Premium
join:2002-11-19
Parsippany, NJ
kudos:2
Reviews:
·Verizon FiOS

said by chrisb3127:

Yep, I am able to ping 10.10.10.1 and do a successful traceroute. I thought these networks were reserved for LAN use and should not be routable outside of the LAN?

Correct. From RFC 1918:

Because private addresses have no global meaning, routing information
about private networks shall not be propagated on inter-enterprise
links, and packets with private source or destination addresses
should not be forwarded across such links. Routers in networks not
using private address space, especially those of Internet service
providers, are expected to be configured to reject (filter out)
routing information about private networks. If such a router receives
such information the rejection shall not be treated as a routing
protocol error.
Verizon REALLY should be dropping these packets. I'm also having trouble getting my Actiontec to drop them; I set an inbound rule on the "Broadband Connection" port to drop any packets from 10.0.0.0/8, but I'm still getting the ping replies. Not sure why yet... still digging.

chrisb3127

join:2006-03-01
Manville, NJ
reply to dennismurphy

Very odd they are routing it. I know some ISPs use 10.x.x.x as a first hop after your router/modem. Here's my successful trace:

2 5 ms 4 ms 4 ms l300.nwrknj-vfttp-102.verizon-gni.net [71.172.250.1]
3 7 ms 11 ms 6 ms g1-2-1-7.nwrknj-lcr-21.verizon-gni.net [100.41.194.58]
4 7 ms 6 ms 6 ms 130.81.199.14
5 7 ms 6 ms 7 ms 2.ae1.xt1.nyc4.alter.net [140.222.228.119]
6 8 ms 6 ms 9 ms 10.10.10.1


matthewh16

join:2010-05-21
Middletown, DE

I could be wrong, either partially or wholey, but I THINK I know whats going on.

Are any of you guys on REALLY old FiOS Installs? Where they still use PPPoE? Not sure if that is related or not.

Here's why I think 10.x.x.x responds, I'll show a traceroute as an example:
Tracing route to www.google.com [74.125.228.20]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms 192.168.1.1
2 6 ms 6 ms 6 ms 10.7.30.143
3 11 ms 12 ms 13 ms g0-10-1-1.phlapa-lcr-22.verizon-gni.net [130.81.
183.18]
4 29 ms 89 ms 9 ms ae12-0.phil-bb-rtr2.verizon-gni.net [130.81.163.
148]
5 17 ms 14 ms 14 ms 0.xe-11-1-1.xl2.iad8.alter.net [152.63.5.249]
6 14 ms 14 ms 14 ms 0.xe-11-0-2.gw9.iad8.alter.net [152.63.40.222]
7 * * * Request timed out.
8 28 ms 29 ms 21 ms 209.85.252.46
9 18 ms 19 ms 19 ms 72.14.238.173
10 18 ms 16 ms 16 ms iad23s05-in-f20.1e100.net [74.125.228.20]

Trace complete.

See how hop 1 is 192.168.1.1? Thats the Actiontec. Hop 2 is 10.7.30.143, now I believe that is the neighborhood fios distribution box. I'm thinking the 10.x.x.x network is used by VZ Internally for FiOS routing, before dumping onto the GNI 'external' network.


chrisb3127

join:2006-03-01
Manville, NJ

FiOS since 2007, but never see 10.x IPs in my traces.



dennismurphy
Put me on hold? I'll put YOU on hold
Premium
join:2002-11-19
Parsippany, NJ
kudos:2
Reviews:
·Verizon FiOS
reply to matthewh16

said by matthewh16:

I could be wrong, either partially or wholey, but I THINK I know whats going on.

Are any of you guys on REALLY old FiOS Installs? Where they still use PPPoE? Not sure if that is related or not.

Here's why I think 10.x.x.x responds, I'll show a traceroute as an example:
Tracing route to www.google.com [74.125.228.20]
over a maximum of 30 hops:

1

I'm not; 100% Ethernet here. Not sure what's going on, but either way, those packets should get dropped at the edge. I've sent off a note to someone who might know if this is normal or not.

edit: Oh, and 10.7.30.143 is not a "neighborhood fios distribution box" -- this is a completely passive optical network (PON) - there are no active components between your ONT & the central office.

matthewh16

join:2010-05-21
Middletown, DE

IDK then, the 10.x.x.x has been in my traceroutes ever since we got Fios around 2007ish.



birdfeedr
Premium,MVM
join:2001-08-11
Warwick, RI
kudos:9
reply to dennismurphy

10.x.x.x traces out, but goes nowhere if the last octet is not 1.
10.x.x.1 pings and replies, but last octet not 1 does not reply.



nycdave
Premium,MVM
join:1999-11-16
Melville, NY
kudos:14
reply to dennismurphy

You are pulling a PPPoE IP if you see traces with 10.x.x.x on the WAN side. Can you verify in your FiOS router? The Broadband connection type should state if your IP is PPPoE or DHCP.



More Fiber
Premium,MVM
join:2005-09-26
West Chester, PA
kudos:29
reply to dennismurphy

said by dennismurphy:

I'm not; 100% Ethernet here.

Whether you're connected to the ONT via coax or cat5 has nothing to do with whether you're PPPoE or DHCP.
--
There are 10 kinds of people in the world; those who understand binary and those who don't.


dennismurphy
Put me on hold? I'll put YOU on hold
Premium
join:2002-11-19
Parsippany, NJ
kudos:2
Reviews:
·Verizon FiOS

1 edit
reply to nycdave

Click for full size
Using DHCP
Click for full size
DHCP
said by nycdave:

You are pulling a PPPoE IP if you see traces with 10.x.x.x on the WAN side. Can you verify in your FiOS router? The Broadband connection type should state if your IP is PPPoE or DHCP.

I guarantee I'm on DHCP. No pppoe. Here's the screenshots from the Actiontec.

That's what strikes me as so strange. If we were pulling up PPP tunnels, I'd expect to see 10.0.0.0/8 traffic on the tunnel. Otherwise, something is advertising a route it shouldn't be ...


dennismurphy
Put me on hold? I'll put YOU on hold
Premium
join:2002-11-19
Parsippany, NJ
kudos:2
Reviews:
·Verizon FiOS
reply to More Fiber

said by More Fiber:

Whether you're connected to the ONT via coax or cat5 has nothing to do with whether you're PPPoE or DHCP.

Of course not, but it's a DHCP config. You can see from my trace route that what's answering as 10.10.10.1 is well within the core and not at my edge.

guppy_fish
Premium
join:2003-12-09
Lakeland, FL
kudos:1
Reviews:
·Verizon FiOS
reply to dennismurphy

This isn't a local to the user issue or brand of router, its internal to the Verizon network. A router will always forward any request that isn't in its subnet, that's what routers do, its perfectly acceptable to have different private networks that are routed to each other, as a matter of fact, this is used in the Cisco training

So the question is why is Verizon allowing an alternet router to respond to what to us is a WAN address, the answer is likely that its either misconfiguration or not considered the "internet"

VPN's shouldn't be an issue as the machine with the active VPN knows the subnet is tunneled and private IP is never seen by the local router ( its encapsulated )

Tracing route to 10.10.10.1 over a maximum of 30 hops

1 1 ms 1 ms 1 ms Wireless_Broadband_Router.home [192.168.1.9]
2 3 ms 2 ms 2 ms L100.TAMPFL-VFTTP-40.verizon-gni.net [96.252.135.1]
3 4 ms 7 ms 7 ms G0-14-2-6.TAMPFL-LCR-22.verizon-gni.net [130.81.218.0]
4 39 ms 4 ms 6 ms so-4-0-0-0.TPA01-BB-RTR2.verizon-gni.net [130.81.199.28]
5 41 ms 43 ms 43 ms 0.xe-2-1-4.XT2.NYC4.ALTER.NET [152.63.4.50]
6 44 ms 43 ms 42 ms 10.10.10.1

Trace complete.



nycdave
Premium,MVM
join:1999-11-16
Melville, NY
kudos:14
reply to dennismurphy

What does the main GUI page show, on the left side when you get the main page loaded? Broadband connection type will say DHCP or PPPoE. Your screen captures don't say which one.


McBane

join:2008-08-22
Plano, TX

I'm on a static IP bound directly on my server with a business account, this server traverses no router on my end:

deimos ~ # ping 10.10.10.1
PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data.
64 bytes from 10.10.10.1: icmp_seq=1 ttl=249 time=38.5 ms
64 bytes from 10.10.10.1: icmp_seq=2 ttl=249 time=38.3 ms
64 bytes from 10.10.10.1: icmp_seq=3 ttl=249 time=38.3 ms
64 bytes from 10.10.10.1: icmp_seq=4 ttl=249 time=38.2 ms
^C
--- 10.10.10.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3263ms
rtt min/avg/max/mdev = 38.217/38.359/38.536/0.115 ms

deimos ~ # traceroute 10.10.10.1
traceroute to 10.10.10.1 (10.10.10.1), 30 hops max, 60 byte packets
1 L100.DLLSTX-VFTTP-26.verizon-gni.net (71.170.175.1) 2.769 ms 2.765 ms 2.751 ms
2 G0-3-2-3.DLLSTX-LCR-22.verizon-gni.net (130.81.138.180) 7.593 ms 7.701 ms 7.706 ms
3 ae4-0.DFW9-BB-RTR2.verizon-gni.net (130.81.199.66) 11.921 ms 5.537 ms so-5-0-0-0.DFW9-BB-RTR2.verizon-gni.net (130.81.199.36) 5.265 ms
4 0.so-6-0-0.XT2.NYC4.ALTER.NET (152.63.17.97) 46.672 ms 46.663 ms 46.632 ms
5 GigabitEthernet7-0.GW9.NYC4.ALTER.NET (152.63.21.194) 41.601 ms * *


guppy_fish
Premium
join:2003-12-09
Lakeland, FL
kudos:1
Reviews:
·Verizon FiOS
reply to nycdave

Dave, the trace I just posted is DHCP, and its a non Verizon branded Actiontech as the primary router.

Regardless, the router is doing what its suppose to, it forwards the request and the Verizon network finds the alternet device, this has nothing to do with the users and our routers are doing what they are suppose to do. You should get your NOC's to find the router responding and fix it


norm

join:2012-10-18
Pittsburgh, PA
Reviews:
·Verizon FiOS
reply to dennismurphy

|---------------------------------------------------------------------------------- --------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|    L100.PITBPA-VFTTP-33.verizon-gni.net -    0 |   20 |   20 |    2 |    4 |    9 |    4 |
|  G0-1-0-5.PITBPA-LCR-21.verizon-gni.net -    0 |   20 |   20 |    3 |    5 |    8 |    4 |
| xe-15-1-1-0.RES-BB-RTR1.verizon-gni.net -    0 |   20 |   20 |   11 |   16 |   77 |   11 |
|           0.so-1-0-2.XT1.NYC4.ALTER.NET -    0 |   20 |   20 |   17 |   25 |   76 |   17 |
|                              10.10.10.1 -    0 |   20 |   20 |   15 |   16 |   17 |   15 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
 


dennismurphy
Put me on hold? I'll put YOU on hold
Premium
join:2002-11-19
Parsippany, NJ
kudos:2
Reviews:
·Verizon FiOS
reply to nycdave

said by nycdave:

What does the main GUI page show, on the left side when you get the main page loaded? Broadband connection type will say DHCP or PPPoE. Your screen captures don't say which one.

Sorry Dave, I thought the "WAN PPPoE Disabled" in the previous screenshot would show it. Here's from the front page.

One of the network architects got back to me and he's having someone look at it ... let you know what I hear from that avenue.


dennismurphy
Put me on hold? I'll put YOU on hold
Premium
join:2002-11-19
Parsippany, NJ
kudos:2
Reviews:
·Verizon FiOS
reply to guppy_fish

said by guppy_fish:

Dave, the trace I just posted is DHCP, and its a non Verizon branded Actiontech as the primary router.

Regardless, the router is doing what its suppose to, it forwards the request and the Verizon network finds the alternet device, this has nothing to do with the users and our routers are doing what they are suppose to do. You should get your NOC's to find the router responding and fix it

Indeed, but for the sake of security, I would think the Actiontecs should have a firewall rule to drop packets sourced from (192.168.0.0/16, 10.0.0.0/8, 172.16.0.0./12) inbound to the WAN port, as those "in theory" could only be spoofed packets.

What I'm seeing are three things ...

#1 - rfc1918 addresses are routable across the FiOS network;
#2 - Something on the VZ network (in alter.net) is accepting and respond to icmp at 10.10.10.1;
#3 - the Actiontecs aren't filtering those packets (though I think they should. just my $0.0002)


julesism

join:2001-12-12
Lewisville, TX
Reviews:
·Verizon FiOS
reply to dennismurphy

I'm using a Linksys E3000 with Tomato and I see it too:

C:\Users\hoss>tracert 10.10.10.1
Tracing route to 10.10.10.1 over a maximum of 30 hops
1 1 ms 1 ms 1 ms router [10.0.0.1]
2 5 ms 4 ms 4 ms L100.DLLSTX-VFTTP-59.verizon-gni.net [173.71.44.1]
3 7 ms 7 ms 7 ms G0-9-1-5.DLLSTX-LCR-22.verizon-gni.net [130.81.187.20]
4 5 ms 4 ms 4 ms ae4-0.DFW9-BB-RTR2.verizon-gni.net [130.81.199.66]
5 38 ms 39 ms 42 ms 0.so-6-0-0.XT2.NYC4.ALTER.NET [152.63.17.97]
6 41 ms 42 ms 42 ms 10.10.10.1
Trace complete.

Expand your moderator at work

buckweet1980

join:2011-12-31
Plano, TX
reply to dennismurphy

Re: [Networking] Is 10.0.0.0/8 not blackholed?

Has anyone looked at a looking glass to see if those prefixes are in the BGP tables?



Killa200
Premium
join:2005-12-02
Southeast TN
Reviews:
·Charter

1 recommendation

reply to dennismurphy

More than likely Verizon is using the 10 net for internal management space on their equipment. I know cable isn't fiber, but with our ISP we use blocks in the 10 net for the secondary internal address every modem gets in order to remote manage equipment and for the modems to communicate to the CMTS. The CMTS, or in your case the OLT would most likely be what is responding on the 10.X.X.1 pings, as it is setup to be the gateway on all of those addresses you tried, probably in class c sized blocks since all the .1 addresses are pinging.

My guess is that they are doing something similar for back end maintenance on OLT / ONT equipment. If you got the luck of the draw and guessed a 10. address that another ONT in your area is using, it would most likely also ping.


McBane

join:2008-08-22
Plano, TX

From the traceroutes I'm seeing on here it looks like it goes well past the ONT into something in NYC on Verizon's backbone.



Killa200
Premium
join:2005-12-02
Southeast TN
Reviews:
·Charter

1 edit
reply to dennismurphy

Seems while dennismurphy and McBane are both in two vastly different regions, both of you are hitting the same destination since your both on fios:

gigabitethernet7-0.gw9.nyc4.alter.net (152.63.21.194)

(Gigabit port 7-0, Gateway Router 9, NYC4 termination point, via alter.net - which is now verizon enterprise)

Verizon may be using 10 net as management a little wider than regional, and you are seeing it route further because of it. To me that would be a nightmare to group that much of your system into a universally reachable circumstance, but perhaps they are doing it because something like ONT service authorization happens on 10 net as well, and all regions of a certain area talk that that location for auth. Lots of eggs in one basket if so, unless they have some CDN built in to that in case of fail over.

guppy_fish looks like you are going to the same place as well, just ending up there on your next to last hop via a different interface or line into the same DC / POP


guppy_fish
Premium
join:2003-12-09
Lakeland, FL
kudos:1
Reviews:
·Verizon FiOS

3 recommendations

reply to dennismurphy

SO everyone understands, your router pushes EVERYTHING upstream as it should that isn't part of the LAN, that how they work. There is nothing wrong with any private range being pushed up, its very common for example to have 192* for say a department then upstream a 10* for building that then exits to an edge router for WAN

What isn't suppose to happen is the request exit Verizon to the internet, which it looks like it isn't, its still in the internal VZ network. Typically, they would be filtering at some level close to the CO ... but it really has no effect on what users are doing.



dennismurphy
Put me on hold? I'll put YOU on hold
Premium
join:2002-11-19
Parsippany, NJ
kudos:2
Reviews:
·Verizon FiOS

1 edit

said by guppy_fish:

SO everyone understands, your router pushes EVERYTHING upstream as it should that isn't part of the LAN, that how they work. There is nothing wrong with any private range being pushed up, its very common for example to have 192* for say a department then upstream a 10* for building that then exits to an edge router for WAN

... EXCEPT that I expect "my" internal LAN and Verizon's network to be different networks, and therefore, I expect to be able to use RFC1918 devices on "my" network without issue.

I stumbled on this because I was configuring an OpenVPN tunnel between two NAS devices (one at my house, one at a relative's for remote backup.)

Anywho - I was writing a script to automatically restart the VPN client on the remote NAS in case the tunnel drops for some reason (to save me from driving to my family member's house to reconnect it) and was using 10.10.10.1 as the internal OpenVPN tunnel server address.

Turns out the ping statement in my little script would return 0 (alive) no matter whether the tunnel were up or down - very bizarre, I thought. So I started digging and sure enough, there already IS a 10.10.10.1 out there. That shouldn't be, at least from my LAN's perspective.

And if it is indeed expected behavior on VZ's part, then they need to document which subnets we should/should not be using on the client side of our FiOS connections. The point of a private IP block is that, well, it's private.

If they tell us not to use 10.0.0.0/8, that's fine -- just document it for us please.

As it is, I use a subnet inside the 172.16.0.0/12 block for my home LAN. Hopefully that's not an issue either.

said by guppy_fish:

What isn't suppose to happen is the request exit Verizon to the internet, which it looks like it isn't, its still in the internal VZ network. Typically, they would be filtering at some level close to the CO ... but it really has no effect on what users are doing.

Agreed with you, except that I consider my LAN and Verizon's internal network two separate entities. If they want to use rfc1918 blocks for themselves, that's fine, but let me know I can't use them on my LAN so we can avoid address conflicts.

It sure does affect what I do - I'm sure I've been sending OpenVPN packets to some device somewhere I didn't intend, and thankfully I caught the issue with my script before I deployed the NAS elsewhere...

McBane

join:2008-08-22
Plano, TX

3 recommendations

This is essentially violating RFC1918. As stated they might be using this internally, but they should NOT be pushing this network to all customers, especially static IP customers. If it's an internal subnet route it only on internal equipment or only for internal networks. Not the whole freakin backbone.

Epic tier 1 network routing failure going on here. From a NetEng perspective this is pretty embarrassing for Verizon.