One of the displays would have to use a VGA connection since there are only 2 TMDS controllers.
This. Just to expand on it a bit, TMDS is used for HDMI and DVI, so you could have two DVI, one HDMI/one DVI, etc. If DisplayPort 1.2 MST hubs were around, you could have gone down that route with a 7770 and some DP monitors. -- KI6RIT