Downloading at full speed gives 'lag'.

So aye, I've been having this struggle now for a few days.
It got worse now when I upgraded my computer to 6GB RAM, AMD A2 6400+,
GeForce 9600 GT 1GB + a new mainboard to handle it all.
Everything seems to be working, except one thing.

When I download, I'm usually at ~1.5mbps, and if the server allows me to
download at max speed, my mouse gets really slow, and everything is 'laggy'.
If I turn down the downloadspeed to, lets say 400kbps, its fine.
I cant understand why, anyone got any solution to my problem?

I'm at Windows Vista x64 SP1 btw.
All updates installed.
And ofc, all drivers installed.
Newest NVIDIA Drivers etc.