|
[HELP] NTP weirdness.Evening all. Got a potentially dumb-ass NTP issue that is driving me to distraction. I have 1 x Cisco 887 with a 15.1 IOS and 1 x 2960S with 12.2.55 On both I have a domain name (myname.cisco.com) a name-server (8.8.8.8) and ip domain-lookup configured. On the 887 if I configure an NTP sever with an IP address it works, but if I use a pool.ntp.org (» www.pool.ntp.org/en/) FQDN for it doesn't work. Conversely on the switch, the FQDN works and an IP address doesn't. On the switch I can ping IP addresses and FQDN's for anything I type in. On the router I can ping IP addresses but I cannot ping any FQDN's at all. Anyone know what's going on? |
|
markysharkey 1 edit |
Thought I had it, but no. Still can't figure out what's going on... |
|
markysharkey |
Plan C. Decided to make the router the NTP master for the LAN. Didn't want to do it this way but don't have a convincing reason why not other than my own preferences to use a public NTP server on all devices. |
|
|
to markysharkey
said by markysharkey:I have 1 x Cisco 887 with a 15.1 IOS and 1 x 2960S with 12.2.55 said by markysharkey:On the 887 if I configure an NTP sever with an IP address it works, but if I use a pool.ntp.org (»www.pool.ntp.org/en/) FQDN for it doesn't work. said by markysharkey:Conversely on the switch, the FQDN works and an IP address doesn't. Two thoughts.... do you have "ip domain lookup" enabled on either platform? Secondly, I don't think ISR platforms take the FQDN names for NTP -- there was this old thread where it was mentioned for an 877 specifically. My 00000010bits Regards |
|
|
said by HELLFIRE:Two thoughts.... do you have "ip domain lookup" enabled on either platform? quote: On both I have a domain name (myname.cisco.com) a name-server (8.8.8.8) and ip domain-lookup configured.
And on my lab 887 !
ntp master
ntp server 0.uk.pool.ntp.org
!
!
887_Router#sho ntp assoc
address ref clock st when poll reach delay offset disp
~127.127.1.1 .LOCL. 7 6 16 377 0.000 0.000 0.217
*~176.58.109.199 192.36.134.17 2 27 64 377 23.923 5.410 3.583
* sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured
!
!
887_Router#ping 0.uk.pool.ntp.org
Translating "0.uk.pool.ntp.org"...domain server (8.8.8.8) [OK]
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 176.58.109.199, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 24/24/28 ms
So FQDN's are fine on an 887 with c880data-universalk9-mz.151-4.M4.bin IOS. |
|
|
to markysharkey
quote: On both I have a domain name (myname.cisco.com) a name-server (8.8.8.8) and ip domain-lookup configured.
...I'm officially braindead... sorry, been that kind of a start of a week... add to the fact I'm going on 3weeks vacation pretty quick adds to it Went knocking on Cisco's site to see if anything had drastically changed with NTP config... go figure that in NX-OS you CAN configure a DNS name. EDIT: IOS 15.1M Command Guide -- specifically for how "ntp server" is configured. Interesting little tidbit about "ntp server hostname command quote: When you use the hostname argument, the router performs a DNS lookup on that name and stores the IPv4 or IPv6 address in the configuration. For example, if you enter the ntp server hostname command and then check the running configuration, the output shows ntp server a.b.c.d, where a.b.c.d is the IP address of the host, assuming that the router is correctly configured as a DNS client.
Only thing I can think of is doing a stare and compare on the and code rev. between the two(?) 887s. REALLY dumb question... can you post the NTP config? ...otherwise I'm drawing a blank... Regards |
|
|
|
NTP config right now is as per above, and as mentioned although I configured 0.uk.pool.ntp.org the sho ntp assoc shows an IP address. I'm on site tomorrow with the recalcitrant kit so if I get a chance I'll make some copy 'n' pastes as I go for comparison. |
|
|
to markysharkey
If you make the router the NTP master make sure it's not responding to NTP requests from the WAN side of the router. If you become part of an NTP DDoS attack it may bring the router to it's knees in a hurry. |
|
|
So how do I keep the times in sync following a re-boot of the router? The site in question suffers the occasional brown out or black out long enough to deplete the UPS and there's no back-up generator. Don't ask... |
|
|
It should sync on it's own. What I am talking about is not restricting the router from contacting the NTP server from the WAN side of the router. I'm talking about preventing external hosts from asking your router for the time. |
|
|
So deny inbound NTP requests that didn't go through the CBAC firewall first? Currently I have an entry on the inbound ACL that matches the CBAC outbound permit udp ntp. |
|
|
to markysharkey
» blog.cloudflare.com/unde ··· attacks/...here's a pretty good article about NTP amp attacks that explains what battleop is getting at. I use CBAC for my NTP and I've never had to open a "permit any any eq ntp" on my inbound ACL before. By the way, how goes about your original problem markysharkey ? Regards |
|
|
said by HELLFIRE:By the way, how goes about your original problem markysharkey? Went nowhere. Gave up and left the switches using the router as the NTP master. Just going to tune the inbound ACL and leave it as is. I'll add a note to my config cook book and make a plan to lab it some time in the next few days when I have more time to actually sit and take notes as I make changes. The problem was on a live site and I had limited time to muck around with it. |
|
TomS_Git-r-done MVM join:2002-07-19 London, UK |
to markysharkey
said by markysharkey:although I configured 0.uk.pool.ntp.org the sho ntp assoc shows an IP address Thats because it does a one off resolution of the hostname to get an IP address from it. You dont want your NTP peers changing frequently, as this will lead to jitter in your time keeping. Being a pool address, most likely every time you reboot the router it will resolve different a IP, and it will hold on to that IP until it reboots again. If that IP becomes unreachable it wont resolve a new address. Thats usually why you configure multiple peers. |
|
|
Good to know. There are several pools on ntp.org so I'll add them in,and if I can dig up the IP addresses (a few pings over a few days should do it) I'll add in the IP's and remove the FQDN's if that would be "better". |
|
TomS_Git-r-done MVM join:2002-07-19 London, UK |
TomS_
MVM
2014-Nov-23 2:43 pm
Im wondering if it might be possible to write an event script to automatically refresh NTP peers, will look in to this... |
|
TomS_ 4 edits |
TomS_
MVM
2014-Nov-23 4:16 pm
I think this will work: event manager applet redo_ntp_servers authorization bypass
event syslog pattern "NTP-4-UNSYNC|NTP Core \(NOTICE\): Clock synchronization lost"
action 0 syslog msg "NTP fell out of sync, re-configuring all servers..."
action 1.1 cli command "enable"
action 1.2 cli command "configure terminal"
action 1.3 cli command "do show running-config | include ntp server"
action 2.1 foreach _iterator "$_cli_result" "\n"
action 2.1.1 regexp "(sntp|ntp) server ([a-zA-Z0-9\.\-]*)" "$_iterator" result _match1 _match2
action 2.1.2 if $_regexp_result eq 1
action 2.1.2.1 if $_match1 eq "ntp"
action 2.1.2.1.1 if $_match2 ne ""
action 2.1.2.1.1.1 cli command "$_match1 server $_match2"
action 2.1.2.1.2 end
action 2.1.2.2 end
action 2.1.3 end
action 2.2 end
action 2.3 cli command "end"
!
Basically it responds to syslog messages related to clock sync loss, and upon seeing that does a "show run | inc ntp server" and loops through the lines of config that are returned and re-configures any NTP servers it finds. NTP servers seem to be configurable by hostname and will store the hostname in config and therefore the script will just replay any "ntp server...." lines it finds (which means hostname based NTP servers should be re-resolved.) SNTP servers seem to resolve hostnames to IPs and store IPs in the config (at least in the IOS I am testing with), so I am deliberately ignoring them. In addition is also produces its own syslog message to indicate that it has re-configured the NTP servers. Not doing a "wr mem" because if youre replaying existing lines of config theres no point burning through flash write cycles. If anyone would like to run this and see if it helps them, please feel free and send me feedback. :-) |
|
TomS_ 1 edit |
TomS_
MVM
2014-Nov-23 4:41 pm
Actually it might even be helpful to respond to two types of syslog messages:
NTP-4-UNSYNC; and NTP-4-PEERUNREACH
That way if you lose sync with an NTP server or an NTP server becomes unreachable, you'll re-configure your servers and hopefully end up with a more reliable batch.
edit
Actually I believe that responding to peer unreachable messages could cause a rather vicious loop. Removed that and went back to clock sync loss only. |
|
|
I'll need about 6 months to digest that! |
|
TomS_Git-r-done MVM join:2002-07-19 London, UK |
TomS_
MVM
2014-Nov-23 6:16 pm
Regular expressions look complicated, but once you learn how to use them you will wonder how you ever did without them. |
|
|
I'm just getting my head around them for CME. It's just about practice and repetition... For now, I need to understand if I am vulnerable to an NTP DoS attack? I am thinking yes as anyone can spoof my public IP address (if they can find it) and spoof the source IP address field for the NTP request. Is there an "easy" way to make myself less vulnerable to this sort of attack. I appreciate making the router totally impervious may not be possible but I also don't want to make it easy to attack either. Is there some middle ground here that can be accomplished with config that is more familiar? |
|
TomS_Git-r-done MVM join:2002-07-19 London, UK |
TomS_
MVM
2014-Nov-24 10:33 am
You can't stop other people sending you traffic, just like you can't stop people sending you mail in the post.
But if it does happen you need a way to identify it and then work with your upstreams to stop it getting to you. |
|
TomS_ |
to markysharkey
The kind of scenario you are suggesting would require the NTP protocol to have built in security measures to only accept incoming packets if it is expecting them. This would need to be further augmented with some kind of authentication and encryption to ensure the incoming packet is genuine.
If you're worried about NTP being messed with you'd need to run your own internal NTP servers with their own clock sources and block NTP at the edge of your network. |
|
|
Ok then. Thanks for the clarification. |
|
TomS_Git-r-done MVM join:2002-07-19 London, UK |
TomS_
MVM
2014-Nov-24 11:06 am
It's unfortunate but I don't think NTP is the super robust protocol it should perhaps be.
Most large networks generally run their own NTP servers to ensure they have as much control over it as possible. You can then use CoPP to help secure it at the packet level on the client devices and your usual firewall/ACLs at the server side. |
|
|
As this network is a domestic (albeit high end) system, I have done exactly what the ISP supplied router will do. Any more than that is gravy! But I am going to look for NTP servers that are password protected just for gits and shiggles... |
|
TomS_Git-r-done MVM join:2002-07-19 London, UK |
TomS_
MVM
2014-Nov-25 5:50 pm
Well if you want to, you could use an ACL to limit the source of NTP packets. Therefore, unless someone is spoofing the IPs of your trusted NTP servers, you should be reasonably impervious to attacks from other sources. Your greatest worry then would be a flood of packets that overwhelms the available bandwidth of the broadband circuit or the processing power of the router.
Finding a source of reliable NTP servers that you can configure is the trick. I just found out the other day that Google has a bunch, time[1234].google.com.
Also I think it was Oxford University had a couple that could be accessed from outside their network.
Level3 seemed to have a couple but they didnt appear to work.
Otherwise your best bet is ntp.org pools, but there are a lot of servers in each individual pool, and the list of available servers is likely to change with little to no notice so it would be hard to control with an ACL. A reflexive ACL could work, though... |
|
|
to markysharkey
said by markysharkey:For now, I need to understand if I am vulnerable to an NTP DoS attack? I am thinking yes as anyone can spoof my public IP address (if they can find it) and spoof the source IP address field for the NTP request. Is there an "easy" way to make myself less vulnerable to this sort of attack. I appreciate making the router totally impervious may not be possible but I also don't want to make it easy to attack either. Is there some middle ground here that can be accomplished with config that is more familiar? 2nd what TomS_ mentioned and his analogy of stopping people sending you mail in the post. I think the standard best practices of hardening the router and config, and having SNMP/xFLOW-based traffic monitoring in place is the best place to start. My 00000010bits Regards |
|
|
I'm liking the reflexive ACL approach. I'm going to lab that and see what happens. I also need to do a better search for NTP servers that don't exist as part of a pool. I like the idea of the pool, but not the fact that it is dynamic. |
|
|
to markysharkey
"uk ntp servers" and "uk ntp servers ip" should get you something. How's this for you markysharkey ? Regards |
|