said by blantonl;1915087 :
Around 6:30 PM CST yesterday, during a very high traffic event, our master database server experienced a hard drive failure in it's RAID array. Because we were experiencing such high traffic, it made it extremely difficult to fail over to a slave server and we had to relaunch a new master server to recover.
It took us a couple hours to re-provision the server and bring the entire site back online. During this time, the site was completely inaccessible and we were unable to record audio feeds during the outage.
Everything is now back online and operating normally - and we'll use this event as an opportunity to further harden the site from future outages.
The site is designed to bring additional servers up as traffic demands increase, but the the type of failure that occurred, the fail over didn't happen properly.--
I'm tired of killing stupid people just trying to do my job and go home!