Wireless Service Providers → how do you supervise your net connection

how does everyone verify the readiness of their connection?

hard to get an email if your connection is down

my phone rings soon enough but I would rather be aware of it before a sub calls.

I only monitor business subs.... resi subs i dont monitor because we have people that think when they are done using the tubes they need to shut off the power strip that has their pc and our sm powered off of it.

Can you imagine how many alerts one would get?

Business customers never power their gear down....

If I get an alert that a biz sm is down, I wait 15 minutes or so and then I give them a call to see if they are having problems or if it is something else going on.

Are you referring to each sub or the backbone itself?

I run two ipMonitor 7.5 (prior to them being purchased by Solarwinds and becoming ridiculously expensive) instances... 1 inside my network, 1 outside at a colocation center

I only monitor businesses, residential users are too flaky

Cheaper options today would be:
- »www.paessler.com/ipcheck
- batch file pinging something + »www.beyondlogic.org/solu ··· mail.htm or »caspian.dotconf.net/menu ··· endEmail on error

I have an off site server colocated at a datacenter with monitoring software running. If it can't reach the IP of my main router, it sends me an email and a text message.

It also monitors one IP from each of my towers (usually IP of the primary backhaul) and alerts me if the tower is unreachable. The main router's IP is a dependency for triggering the rest of the alerts (if my main connection is down, the rest of the towers don't really matter).

I plan to add one device at each tower that is not plugged into the battery backup (something cheap like a used routerboard or something) that I can monitor for power outages too. I've gotten a few alerts that a tower is down and drove out just to find out that the battery had drained. Had I gotten the alert a few hours sooner, I could have arranged a generator or car with an inverter to sit there until the power came back up.

@jcremin ...I think you'd be better of getting data from the UPS directly, a Smart-UPS can email you (when attached to a PC via USB or serial, or with an embedded network card) with: on battery, off battery, runtime remaining, runtime critical, batter needs replacement, over-voltage, under-voltage, overloaded, too hot, input freq. not quite 60hz (meaning you're on generator), log input/output voltage, scheduled self test pass/fail, etc.

Agreed...... apcupsd is a dream

APC : 001,037,0941
DATE : 2010-10-13 18:44:23 -0400
HOSTNAME : ##########
VERSION : 3.14.8 (16 January 2010) redhat
UPSNAME : Tower_Briland
CABLE : USB Cable
MODEL : Back-UPS XS 1300 LCD
UPSMODE : Stand Alone
STARTTIME: 2010-10-11 15:05:24 -0400
STATUS : ONLINE
LINEV : 111.0 Volts
LOADPCT : 14.0 Percent Load Capacity
BCHARGE : 100.0 Percent
TIMELEFT : 42.9 Minutes
MBATTCHG : 10 Percent
MINTIMEL : 5 Minutes
MAXTIME : 0 Seconds
SENSE : Medium
LOTRANS : 088.0 Volts
HITRANS : 139.0 Volts
ALARMDEL : Always
BATTV : 26.7 Volts
LASTXFER : Input frequency out of range
NUMXFERS : 0
TONBATT : 0 seconds
CUMONBATT: 0 seconds
XOFFBATT : N/A
SELFTEST : NO
STATFLAG : 0x07000008 Status Flag
MANDATE : 2007-11-07
SERIALNO : 8B0745R24322
BATTDATE : 2007-11-07
NOMINV : 120 Volts
NOMBATTV : 24.0 Volts
NOMPOWER : 780 Watts
FIRMWARE : 836.H7 .D USB FW:H7
APCMODEL : Back-UPS XS 1300 LC
END APC : 2010-10-13 18:44:59 -0400

let me be more precise

I am talking about your backbone

Not every customer: that I do with Solarwinds and The Dude

Are are you notified if your provider goes down?

it depends on your provider if they want to notify you or not.

I use WhatsUp for notification. It can email me, text my phone and text my pager. Obviously email and texting my phone require that the email gateway be up but text paging is via modem to our pager service so independent of the internet.

Makes more sense

Backbone (microwave relay sites) are monitored from every aspect possible. RSL, Jitter (if Canopy), etc.

Provider connectivity is monitored by having someone in the outside world with a monitoring box pinging hosts on my core, which are located in strategic places.

The way I monitor my providers is on either sides of the router...

my first static IP, the gateway IP, the next hop IP

and DNS service...

my DNS server IPs, the provider's DNS server IPs

...I compare the results from my inside monitor (on provider)
with the results from the outside monitor (geographically diverse, different provider)

A well done Dude setup is more than enough. We use the dude as well as cacti but to be honest the dude could do both just fine. we graph just about every piece of data such as temps and volts, to number of associated clients to number of active PPPoE connections. There's always a little up and down if clients disconnect etc, but we have a script where it will compare the last 2 readings and if they break a certain delta threshhold it sets off an alarm. All alarms appear onscreen at the head end as well as get sent to pagers.

We dont monitor client CPE unless it's a high end business connection or a LAN extension tunnel etc.

my business partner who has much experience in the alarm industry has suggested we use an ip module normally used to monitor an alarm panel at the other end.

The moment the monitoring station loses contact with the ip module, they notify you.

Cost of monitoring roughly $10 to $20 a month and about $100 for ip module.

I probably shouldn't have even mentioned monitoring for power outages earlier since it is off topic, but I just wanted to mention that some of my sites are fed directly from the DC off of deep cycle batteries, so there is no "intelligence" to plug into. Plus none of my tower sites have any PC's at them. The extra monitoring is nice, but I can calculate best case and worst case estimates for runtimes. As long as I know when the AC goes out, I can plan to have an alternate power source brought to a tower before I reach 50% of the worst case scenario, for example.

Sorry again for that part being off topic.

On topic, I'll mention that my off site server is a Windows 2003 server (doubles as my small web hosting machine) and runs GFI Network Server Monitor as the program that does all my monitoring and notifications. It works really well.

Depends what you're monitoring for. The alarm solution would work, but that's basically all it would do.

Jump into the Dude and SNMP and you'll soon see that there's a pile of things that can be monitored with it. It does ping and such that will accomplish a simple up/down test.

Simple example. Get a cheapo switch SNMP enabled (SW-250 by Mikrotik for example).

Now... if you want ping sensors, just ping the radios/routers/switches you have on site, that will solve you main need. However to add a kicker you create ethernet loopback sensors that you plug into these switches. And then watch the ports with SNMP, they drop or link then you know something triggered the physical sensor.

IE - Take simple magnetic door alarm sensor and rig it to make/break an ethernet loopback. Plug it into port 5... SNMP monitors port 5 for a link/break... When it triggers Dude pages you. Voila. Physical security. And again on top of it with basically any routerboard (450G is my fav) you have built in voltage and temp monitors. Then of course you use ping for the average "Is it up or down" testing, and some SNMP extras to watch associations or noise floors or signal levels.

Every one of our backhauls has the noise floor and signal level and CCQ graphed, same goes for APs with the noise, CCQ, and number of clients associated. You begin to see problems that you didn't know existed before and can fix them before they become visible.

Setup a deal with another provider to monitor your network. I use nagios to monitor a couple of key points in others networks and they do the same for me.

Additionally my provider is fairly proactive in contacting me, but only if there is a physical connection issue ie loss of light.

I have two load balanced connections and the dude pinging the dns servers of each provider. So if the connection to one goes down, it just sends everything through the working connection, and sends me an email which i pickup on my phone.

We use an open source package called "mon".

The interface isnt the most sexy thing around, but the monitoring engine side of it is as complicated or as simple as you want it to be - because you can write your own custom monitor services to do whatever you need.

Ive written so many custom monitors for our network I can barely keep track of them, but mostly they do something like grab a bunch of data from each host (if reachable, if not that causes an alert) usually via SNMP, check for various values or conditions, and raise alerts if required.

We monitor everything, including:

* reachability (i.e. ping)
* interface error counters
* microwave link parameters (RSSI, radio alarms)
* CPU, memory, and other resource utilisation
* state of BGP sessions (eBGP and iBGP)
* web requests and then validate returned content, latency, etc
* RADIUS, email, DNS, etc services
* power systems (mains input fail/out of range, circuit breakers, system components)
* dry contacts for things like door alarms etc
* ugh, so much more.....

The monitor scripts can be written in any language. I tend to write mine in perl, but you could write them in C and compile it if you really wanted to. There are also a variety of pre-fab monitor services to get you started. For everything else you can BYO and make it work exactly the way you want.

O M G - I don't know whether I should be praising your geniousness or be running in terror

Internally we do our own monitoring. We also have a server in another colo that performs monitoring of the mail server and first hop to cover the exact circumstance you're talking about.

If the colo route is too expensive there are some companies that offer external monitoring packages.

Now you have to put a PC at each tower.

Sounds complicated and expensive when these guys can do it and provide you with more information for less $$$

»www.dotcom-monitor.com/

For a little more money you could get a host linux server that would both allow you to remote monitor and act as a mail spooler if your mail server goes down.

»vr.org/

...not necessarily: »www.apc.com/products/res ··· u=AP9631

I usually sit a tub of water beside the rack, with the words "compliance bin" wrote on it, and give my equipment the stare that lets it know you mean business. So far, its worked wonders!

But seriously, we mostly run cacti internally with thresholds that trigger an email or txt to whoever needs to know whats going on. That covers our butts on the internal network. We don't have much in the ways of monitoring the outside network besides Me, the Boss, and our 2 guys on trucks having cable internet at our homes that use the network. That will probably change once we go redundant on the backbone.

Well, again, that's great if you happen to already be using APC Smart UPS's that have a slot for that card. At a quick glance, the card looks like it retails for around $350...

My point was that I can stick a $40 routerboard or a used linksys router at each site, slap an IP on it and monitor it like the rest of my network with a simple ping.

Buy for those have the right APC system and a few hundred extra bucks laying around, it looks like it would be nifty..

Ive usually had luck chasing them down for $100 - $150 on fleabay with or without environmental monitoring.

»cgi.ebay.com/APC-AP9619- ··· 604cf892 -- With Sensor

»cgi.ebay.com/APC-AP9619- ··· 5e09859e -- Without Sensor

Even includes contact inputs.

The network card costs as much as a cheap PC that would monitor the APC.

If you're going to spend the $40 for the MT and your UPS is an APC with a serial output you can do the same thing with the UPS module built into router OS.

The Dude for everything inside that matters. Smokeping for Latency/Jitter/Packet Loss inside and outside. I use SMS to a Cell phone that the on call guy carries + email to everyone in the office when there's a problem.
The network is now multi homed (x3) so it reroutes on failure of the 0.0.0.0 route and the SMS are sent via SMTP, but when I was first starting and had only one Feed I used to send an SMS to my phone every 30 minutes so I knew the Internet feed was still up because I was paranoid.