latency is irrelevant unless it gets high enough that the TCP windowing algorithm cannot keep up and packet loss or high latency shrinks the sliding window.
On the other hand, I've seen plenty of times where servers are "faster" (i.e. lower latency), but give erratic and/or just slow results compared to others which may be 10-20 ms higher latency.
Sorry, I'm a stickler for proper scientific method of testing --
"Women. Can't live with 'em, pass the beer nuts." -Norm