Slow FTP Uploads and POP3 downloads

I am running ISA server 2000 on a Windows 2003 server. A few weeks ago my
remote users reported slow downloads from the POP3 server, I also noticed
slow upload times to my FTP server. Both my Exchange and FTP servers are
behind the ISA server. When I restart the Microsoft Firewall service on the
ISA server the uploads to the FTP seed up as well as the POP3 downloads,
whithin a few minutes of the service running the speed drops back down again.
This is really becoming a problem any suggestions?