dslreports logo
    All Forums Hot Topics Gallery


how-to block ads

Search Topic:
share rss forum feed

Mountain View, CA
reply to howardfine

Re: FreeBSD.org "cluster server" security breach

Announcement states November 11th was when the breach was discovered. First indication from the user community that things were unreachable: November 11th at roughly 07:00 PST. Statements from FreeBSD Project members as to what the "cause" was, prior to announcement:

»lists.freebsd.org/pipermail/free···590.html -- "working on issues"
»lists.freebsd.org/pipermail/free···581.html -- ?!
»lists.freebsd.org/pipermail/free···596.html -- "known problems"
»lists.freebsd.org/pipermail/free···501.html -- "work being done"
»lists.freebsd.org/pipermail/free···523.html -- "machines physically moved and discombobulated"
»lists.freebsd.org/pipermail/free···545.html -- ?!
»lists.freebsd.org/pipermail/free···571.html -- "problems in the cluster" which impacted third-party ports mirroring sites

freebsd-update was also impacted, as was (and still is as of this writing) GNATS (many PRs when queried aren't being listed, despite the PR numbers supposedly being valid). portsnap is still not providing updates (see last line of "What is the Impact" and last line of 4th bullet item at bottom of breach post).

Questions myself and many others FreeBSD SAs have been asking:

1. If this was truly maintenance (and instead a breach was found during that maintenance), why wasn't this maintenance pre-announced on lists, the RSS feed, forums, website, Twitter (god forbid but still), etc.?

2. More like a subset of #1, who has maintenances that last 5 days? No company or organisation I know of, including me (who ran a free hosting service for 18+ years). I refuse to believe Peter Wemm got up one morning and decided to do all of this on a whim... at least I HOPE that's not the case...

3. During the entire 5-day downtime, nothing was stated until yesterday afternoon, and it was beyond vague + placed very cleverly on the home page as to be intentionally missed. That message is gone and I do not have an archive of its content, but effectively it said there "were issues being investigated" in "the cluster". It did not say "we're performing scheduled work". So the message here was mixed; so what's the truth, and where can one read about the truth?

4. No information given about what degree of access the person had whose public key was compromised. Root or not? It matters only as a result of the degree of concern cited over the package server repositories being potentially compromised. I can't imagine someone with user-level access would have the ability to write to the central package server directly (and if they did, wow, I guess that's a separate bitch/rant item). When I was a committer the main box used by everyone was user-level access only, but that's (hopefully?) not the same box as used in the "package cluster". (Again: no technical information about "the cluster" means everyone has to keep making guesses)

5. Why did this breach take nearly 2 months to discover? The article implies that illegitimate access was performed on September 19th. I would assume servers that have the ability for someone to update packages on the official ftp package server would have an IDS or be LAN-only + be behind a firewall with IP-level restriction.

I still cease to see what portsnap/svn have to do with this security breach, which is why I'm calling shenanigans on the last page of the breach post. Some FreeBSD Project members also have the same concerns above that I do, so I'm not the only one.

I'm not surprised that this is being handled in a hush-hush manner, because the impact this could have (PR-wise) with folks like Juniper, Citrix, BlueCoat, NetApp, Netflix (they just moved some bits to FreeBSD), etc. is tremendous.
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


Saint Louis, MO
·AT&T Southwest
Well I knew the cvs thing was coming and I haven't been up to date on things for a while. As far as being hush hush goes, they put an announcement out so it doesn't sound so hush-hush unless you mean they kept quiet about it for two months but my impression was they just found out about it recently and traced it back to when it occured...two months ago.

The scenario I got out of it is this. One of the devs was doing some work on those machines. His key was exposed somewhere about two months ago and some cracker found it recently and tried to access the systems using it. This was caught by security and they removed the two machines from service.

Of course that's my guess but everyone else seems to be guessing, too, so mine's as good as anyone's...I guess. I don't see what's to be gained by keeping it quiet if you were correct in that. Especially since there was no harm done.

Mountain View, CA
It actually sounds like you and I interpret the scenario the same way. I'm under the impression someone with access to the package building box had their SSH private key compromised -- maybe they had it on a laptop which got stolen, maybe their own machine got compromised, who knows -- sometime on or before September 19th. Sometime between then and (or on) November 11th an illegitimate person was able to successfully log in to the package build cluster using said key. This prompted, on November 11th, FreeBSD.org cluster admins to shut down daemons and/or make boxes inaccessible while an investigation could be performed (and of course later attempting to verify integrity of all files).

I don't think there was a hidden agenda or anything like that (I'm actually quite the sceptic!).

Most of my peers want to know how that person's key got compromised. This seems to surprise a lot of people, but I don't particularly care how it got obtained -- this sort of thing happens in the real world. Hardware get stolen, machines get compromised, USB sticks get lost, you name it. What is surprising to me, however, is that for an intruder to use that SSH private key, it would had to have been passwordless (eek #1) or possibly the intruder had installed a keypress logger or something equivalent to figure out what the private key password was.

I'm not so angry about the near-2-month period of time that intruder had access to the cluster. Sometimes it takes time for admins to find stuff like that out (I speak from experience). Hell, I've heard of some Linux machines (circa late 90s) remaining compromised (or would be re-compromised) for an entire year. I'm more surprised by the fact that (from the sound of it) one or more of those build cluster boxes is on the Internet publicly, and sounds like they may have lacked firewall configuration bits limiting who could SSH to it (eek #2). I know that in the case of the server used to perform (what was then) CVS commits for committers, the box was publicly accessible via SSH -- I didn't have to provide my IP address (I imagine because a lot of the devs tend to roam / travel internationally).

Figuring out what the intruder did is a serious PITA (I sympathise there -- this is very difficult to do, especially where on FreeBSD things like acct(5) are actually busted in really depressing ways), but if the account didn't have root access then I'm surprised they'd have the ability to overwrite packages on the official package mirror server (eek #3).

My final concern (eek #4) is why the situation wasn't announced in some way publicly (like I said: website, -announce / -stable / -questions mailing lists, RSS feed, or Twitter), even if nobody knew what the root cause was. A simple "we've shut off the SVN-to-CVS gateway for an investigative matter; you won't see ports/src/etc. updates until we turn it back on. We've also shut off the portsnap master. csup/cvsup, portsnap, freebsd-update, native CVS, and GNATS [still not sure how] are all impacted. We'll provide more detail when we have it, rest assured" would have been sufficient. This didn't happen until the afternoon of the 4th day, which prompted Slashdot news submissions and so on. Users were left going "erm, am I doing something wrong?!" WRT 4 out of 5 of those services, and nobody in the know was saying anything on the mailing lists, the forum, or anywhere else. Nothing official / standardised was sent out, which is why lots of users were asking the same question across multiple places (multiple lists, the forum, etc.).
Making life hard for others since 1977.
I speak for myself and not my employer/affiliates of my employer.


Oakland, CA

1 recommendation


...we have to recommend you consider reinstalling any machine from scratch, using trusted sources.

In light of the vastness of potential for compromise and the widespread, severe consequences to be assured of remediation, I can imagine PTB needing to be close-mouthed until some extensive forensics had been done.

»lists.freebsd.org/pipermail/free···595.html and
»www.linux.com/news/featured-blog···ernelorg seem reasonable.