Mountain View, CA
|reply to howardfine |
Re: FreeBSD.org "cluster server" security breach
It actually sounds like you and I interpret the scenario the same way. I'm under the impression someone with access to the package building box had their SSH private key compromised -- maybe they had it on a laptop which got stolen, maybe their own machine got compromised, who knows -- sometime on or before September 19th. Sometime between then and (or on) November 11th an illegitimate person was able to successfully log in to the package build cluster using said key. This prompted, on November 11th, FreeBSD.org cluster admins to shut down daemons and/or make boxes inaccessible while an investigation could be performed (and of course later attempting to verify integrity of all files).
I don't think there was a hidden agenda or anything like that (I'm actually quite the sceptic!).
Most of my peers want to know how that person's key got compromised. This seems to surprise a lot of people, but I don't particularly care how it got obtained -- this sort of thing happens in the real world. Hardware get stolen, machines get compromised, USB sticks get lost, you name it. What is surprising to me, however, is that for an intruder to use that SSH private key, it would had to have been passwordless (eek #1) or possibly the intruder had installed a keypress logger or something equivalent to figure out what the private key password was.
I'm not so angry about the near-2-month period of time that intruder had access to the cluster. Sometimes it takes time for admins to find stuff like that out (I speak from experience). Hell, I've heard of some Linux machines (circa late 90s) remaining compromised (or would be re-compromised) for an entire year. I'm more surprised by the fact that (from the sound of it) one or more of those build cluster boxes is on the Internet publicly, and sounds like they may have lacked firewall configuration bits limiting who could SSH to it (eek #2). I know that in the case of the server used to perform (what was then) CVS commits for committers, the box was publicly accessible via SSH -- I didn't have to provide my IP address (I imagine because a lot of the devs tend to roam / travel internationally).
Figuring out what the intruder did is a serious PITA (I sympathise there -- this is very difficult to do, especially where on FreeBSD things like acct(5) are actually busted in really depressing ways), but if the account didn't have root access then I'm surprised they'd have the ability to overwrite packages on the official package mirror server (eek #3).
My final concern (eek #4) is why the situation wasn't announced in some way publicly (like I said: website, -announce / -stable / -questions mailing lists, RSS feed, or Twitter), even if nobody knew what the root cause was. A simple "we've shut off the SVN-to-CVS gateway for an investigative matter; you won't see ports/src/etc. updates until we turn it back on. We've also shut off the portsnap master. csup/cvsup, portsnap, freebsd-update, native CVS, and GNATS [still not sure how] are all impacted. We'll provide more detail when we have it, rest assured" would have been sufficient. This didn't happen until the afternoon of the 4th day, which prompted Slashdot news submissions and so on. Users were left going "erm, am I doing something wrong?!" WRT 4 out of 5 of those services, and nobody in the know was saying anything on the mailing lists, the forum, or anywhere else. Nothing official / standardised was sent out, which is why lots of users were asking the same question across multiple places (multiple lists, the forum, etc.).
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.