Close to 5.5M people using the network isn't bad. It's only been a year since LTE started rolling out. Many people haven't got to the point in their contracts to get a subsidized device. And until recently, all the LTE phones were the high end $200-$300 phones. Most people just don't need the top of the line phone.
I'd surely use LTE if Verizon would just expand the coverage about 1 frigging mile further! Can't quite get coverage at home but about a mile away I do.