I only have a Logitech Keyboard & Mouse receiver (one USB device services them both) and a Logitech web cam. Only two USB devices are plugged into the machine at any one time. It's rare, but sometimes I connect my Android phone to the computer but usually only for short periods of time and I disconnect it when I'm finished.
LatencyMon says that some nVidia driver along with the NDIS, TCP/IP, and USBPORT driver are also part of the problem but it changes depending upon the state of the machine. As you can guess, tracking down which driver is causing this issue has been rather difficult.--
Boycott AT&T uVerse! | Tom's Android Blog | AOKP (The Android Open Kang Project)