When copying files between two Windows 2003 SP2 Servers across our WAN, I can't get more than about 3Mbps for a single copy operation. When running multiple copies simultaneously, each one gets about 3Mbps up to the maximum bandwidth of the connection (on both a 1GB and a 100MB connection). My servers are in a data center on each coast of the US (one on the West Coast, the other on the East Coast), with about 100ms of latency. I have observed the performance limits with both a drag-and-drop from Windows Explorer and from Robocopy. When using iPerf with the default settings I see the same performance, but when running multiple simultaneous tests or increasing the TCP Window size I can get nearly my full speed. When I attempt to make any changes to the registry (Tcp1323Opts=3 and GlobalMaxTcpWindowSize={various values tried} in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters) I get no performance increase. I've also tried adding DefaultSendWindow to [HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\AFD\Parameters] with no luck.
Dear Customer, Thank you for your post. This is Neo and I will be assisting you in this post. From your description, I understand that: When files are copied through WAN connection between two Windows Server 2003 SP2 Servers, the speed of one single session is limited to 3Mbps. When multiple sessions are running, the total speed can reach the maximum bandwidth of the WAN connection, however, the speed of each single session is still limited to 3Mbps. Any changes made to the registry don't help. If there is any misunderstanding, please let me know. This issue might be caused by WAN connection bandwidth limitation. In order to narrow down this issue, please help confirm the following questions with me. (Let's call the West Coast location as local site, the other as remote site.) 1. Please describe the detailed Network Topology. 2. Is there any routing/NAT devices between your two Windows Servers? If yes, please check whether there are any bandwidth limitation policies set on them. 3. Please tell me the way how these two Servers communicate with each other. (Though VPN or WAN connection directly.) 4. Please test the copy speed between two Windows Servers in the same local LAN and check whether the issue still occur. 5. Please choose a different Windows Server located in remote side and test the copy speed between local site's Windows Server and remote site's Windows Server via WAN connection. 6. Please try to download some big document from the internet on local site's Windows Server and test the download speed. After you finish the above tests, please tell me all the detailed result. Thank you for your time and I look forward to hearing from you. Sincerely, Neo Zhu, Microsoft Online Support Microsoft Global Technical Support Center Get Secure! - www.microsoft.com/security ===================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ===================================================== This posting is provided "AS IS" with no warranties, and confers no rights.
1. West Coast Site and Remote Site are on Gigabit Ethernet LAN with Gigabit to the public internet and are interconnected to each other by private IP transport (Routed Gigabit). Both are connected directly to the external router, bypassing the Firewall (Public IP addresses entered directly on the Windows NIC). They each also have a second interface/NIC on our private LAN (10.x.x.x IP range). 2. Each site has a Fibre-connected gigabit router, no rate limit set. 3. The servers can communicate with each other by either their public interface or their private interface. The private interface uses the private IP transport (effectively a site-to-site VPN), the Public interface uses the internet. Both have the same performance. 4. Copies from the West Coast server to another in the same site happen at the expected speed (20%-40% utilization of the max interface speed), the same is seen at the remote site. 5. Any server in the West Coast site has the same performance limitations when communicating with any server in the remote site. 6. When using some of the available network speed performance test websites, I’ve gotten speeds of: Download Speed: 201673 kbps (25209.1 KB/sec transfer rate) Upload Speed: 18124 kbps (2265.5 KB/sec transfer rate) Those speeds are representative of either site (when testing against a test site geographically close). Additional information: When I test using iPerf, I can see 30%-40% NIC utilization when I adjust up the TCP window size. I can also see this performance when using multiple simultaneous copy operations (either FTP or Robocopy).
Dear Customer, Thank you for your reply. I performed further analysis and I noticed: 1. If copy between two servers in the same site, the issue does not occur. 2. The issue occurs with several different computers as long as the two related computers are in different sites. Therefore, I think the issue is not due to a Windows based setting or limitation. Instead, it should be very likely a network related limitation. To efficiently resolve the issue, I strongly recommend you contact the ISP and the router manufacturer for more investigation, to see if there are some known issues or settings on the router or on the ISP side. Thanks. Sincerely, Neo Zhu, Microsoft Online Support Microsoft Global Technical Support Center Get Secure! - www.microsoft.com/security ===================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ===================================================== This posting is provided "AS IS" with no warranties, and confers no rights.
Thanks for your insight. I performed further analysis and I noticed: 1. When running iPerf (on the two Microsoft Windows servers), after modifying the TCP Window Size option, performance tests show over 100 times faster than standard Windows copies. 2. These tests are being performed across the WAN, so they appear to dispute your theory that it is a router/firewall/ISP limitation. Can you explain why the WAN is able to perform as expected, but native Windows copies cannot? Thanks. Sincerely, David.
Hi David, Have a look to make sure you are not fragmenting encrypted packets on either end of the tunnel. If the packets are put into the VPN or otherwise encrypted, they'll grow, potentially pushing them past your MTU. This means that they'll fragment as they are shipped into the WAN. This is further complicated by Windowing as it allows more packets through before it validates making the reassembly processor intensive. You can lose half of your bandwidth ore more due to retransmits and the routing of nearly empty fragments. This theorem is supported by the fact that wrenching up the window size makes the problem less visible. You could attack this by dropping the MTU on the servers or by forcing the encryption after fragmentation. The latter option is done on your router (or switch if you're running one of the big ones for IP distribution -- sup 720 or the like). Finally, you might also look at your QoS -- it is possible that you are being bumped in the transfer or aren't maintaining priority. Check this both locally and as you enter the IP cloud. Oh, and looking at SMB signing might also be worth while -- you can see a 30% overhead there. -- Ryan Hanisco MCSE, MCTS: SQL 2005, Project+ http://www.techsterity.com Chicago, IL Remember: Marking helpful answers helps everyone find the info they need quickly.
Dear David, Thank you for your reply. I didn't know IPERF tool before, and after reading your e-mail I had a look at it. As far as I understand it has a server side service and a client side component. It looks like it measures network performance by sending raw data over TCP connection with several options such as changing TCP window size, changing amount of data to be sent etc. IPERF uses streaming protocol and it does not rely on the SMB (Server Message Block) application-layer protocol which is used when performing native Windows copies. I'd like to confirm whether there are any bandwidth limitation policies of specific sessions or QoS policies enabled on your Router or ISP side. Therefore, I still recommend you contact to the ISP and the Router manufacturer to confirm this first. This will ensure that we troubleshoot the issue in the most efficiently way and I appreciate your time. However, at the same time, I will also keep researching this issue for you. If I have made any progress, I'll inform you as soon as possible. Thanks. Sincerely, Neo Zhu, Microsoft Online Support Microsoft Global Technical Support Center Get Secure! - www.microsoft.com/security ===================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ===================================================== This posting is provided "AS IS" with no warranties, and confers no rights.
Thanks. I've forwarded off your response (and this thread) to my ISP. They have said there is no rate limit on my interface, and they are checking further for anything on their end..
Ryan - thanks for your time on this. I have checked for MTU, and it remains unfragmented to 1472 (I'm testing the network across the unencrypted WAN, not through a VPN tunnel). I haven't looked at SMB yet because of the huge disparity on performance. At less than 1% network utilization I don't think that the 30% SMB overhead is causing the problem. My ISP is saying there is no limitation or QOS, and so I'm further at a loss.
Dear David, I spent several hours investigating this issue and I'd like to share you some progress I have made up till now. According to your description, you have two large-bandwidth and long-latency WAN links on two sites. As we know, long latency is a challenging circumstance for the efficient operation of most network protocols, in particular those connection-oriented transport protocols such as TCP which rely on periodic acknowledgement of receipt. Maximum throughput can be calculated as the size of the payload that the sender can place on the wire before having to wait for confirmation multiplied by the number of times per second that acknowledgement is received and the next lot of payload can be sent. While decreasing link latency tends to be impractical or not cost-effective in most situations, the payload size is a function of protocol configuration. To improve throughput, enabling the TCP receive window scaling options is recommended, so that the effective maximum receive window size becomes a multiple of the advertised window. Also, by enabling TCP time stamps through the same registry bitmask value ('Tcp1323Opts'), will help improve effective use of the available bandwidth on this link. This could explain why after you modify the TCP Window Size option in IPERF tool, performance tests show much faster than before. I also notice that you have already tried to change registry to change windows scaling options and timestamp settings but with no help. After further research, I find out that there is an inherent limitation of the SMB protocol used by Windows Explorer and the copy command. SMB has an architectural limitation in the form of a 64KB buffer size limit which cannot be overcome through the use of TCP window scaling. Therefore, SMB tends to perform poorly over high latency WAN links. Fortunately, this limitation no longer exists in SMB 2.0 which is used on Windows Vista Operating System and Windows Server 2008. Having said above, I recommend you not use applications which rely on the SMB protocol to send large amounts of data across the long latency WAN links. You might use FTP as the protocol is faster and has the ability to continue a download were interrupted without having to start the entire file over again. Upgrading client OS to Windows Vista and servers OS to Longhorn will address the problem in the long term. I hope this helps. Thanks. Sincerely, Neo Zhu, Microsoft Online Support Microsoft Global Technical Support Center Get Secure! - www.microsoft.com/security ===================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ===================================================== This posting is provided "AS IS" with no warranties, and confers no rights.
Thanks for your research. I have had better luck with FTP, but only getting about 7-8mbs. Can you let me know the proper registry keys/values to change
Dear David, Thank you for your post. Firstly, I'd like to share some more information about TCP Windows Size. The TCP receive window size is the amount of receive data (in bytes) that can be buffered during a connection. The sending host can send only that amount of data before it must wait for an acknowledgment and window update from the receiving host. For more efficient use of high bandwidth networks, a larger TCP window size may be used. The TCP window size field controls the flow of data and is limited to 2 bytes, or a window size of 65,535 bytes (64K). Since the size field cannot be expanded, a scaling factor is used. TCP window scale is an option used to increase the maximum window size from 65,535 bytes to 1 Gigabyte. When the value for window size is added to the registry and its size is larger than the default value (64K), Windows attempts to use a scale value that accommodates the new window size. The window scale option is used only during the TCP 3-way handshake. The window scale value can be set from 0 (no shift) to 14. To calculate the true window size, multiply the window size by 2^S where S is the scale value. For example, If the window size is 65,535 bytes with a window scale factor of 4. True window size = 65535*2^4 True window size = 1048560 (1MB) Having said above, the following are steps you might try to change your registry key. 1. Back up the registry before you modify it. For more information about how to back up, restore, and modify the registry, please refer to the following KB article. http://support.microsoft.com/kb/256986/ 2. Click Start, click Run, type Regedit, and then click OK. 3. Expand the registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters 4. On the Edit menu, point to New, and then click DWORD Value. 5. In the New Value box, type Tcp1323Opts, press ENTER, and then on the Edit menu, click Modify and Type 3. Note The valid range is 0,1,2 or 3 where: 0 (disable RFC 1323 options) 1 (window scale enabled only) 2 (timestamps enabled only) 3 (both options enabled) 6. Expand the registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters 7. On the Edit menu, point to New, and then click DWORD Value. 8. Type TcpWindowSize in the New Value box, and then press Enter 9. Click Modify on the Edit menu. 10. Type the desired window size in the Value data box. Note. The valid range for window size is 0-0x3FFFC000 Hexadecimal, I recommend you try 100000 Hexadecim (1MB) first. If we set this value too large, some unexpected problems might occur. Please finish the above steps on both of your two Windows Server 2003 machines and then test the performance again. For more information about Windows TCP feature, you may refer to the following article: Description of Windows 2000 and Windows Server 2003 TCP Features http://support.microsoft.com/kb/q224829/ I hope this helps. Sincerely, Neo Zhu, Microsoft Online Support Microsoft Global Technical Support Center Get Secure! - www.microsoft.com/security ===================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ===================================================== This posting is provided "AS IS" with no warranties, and confers no rights.
Hello David, How's everything going? I'm wondering if the suggestion has helped or if you have any further questions. Please feel free to respond to the newsgroups if I can assist further. Sincerely, Neo Zhu, Microsoft Online Support Microsoft Global Technical Support Center Get Secure! - www.microsoft.com/security ===================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ===================================================== This posting is provided "AS IS" with no warranties, and confers no rights.
I am trying to copy a file from one server to other server with the Windows Explorer but this devices are connected over Internet through a VPN site-to-Site network. Well, I can copy files over the link until 1024 MB, bu when I was trying to copy a file over 1024 MB the process was interrupted and I need to start again the process without resume options. This problem is not permanent, sometimes I can copy files over 1024 MB without problems. I call my ISP but they told me that is not a problem with their networks, then I want to know if I can change some parameters on the windows registry to rise the amount of wait time for the ACK and let more time to kill the copy process. Regards, Jose Bonilla
I am trying to copy a file from one server to other server with the Windows I would suggest using robocopy, from the resource kit. hth DDS
Yes, try robocopy first. If robocopy doesn't work, check the MTU size. This search result may help too. VPN connection is disconnected after several minutes VPN drops the connection. VPN is very slow. To change the MTU Settings, ... Resolution: Set my VPN client MTU to 1400. To modify MTU, please refer to this ... www.chicagotech.net/VPN/vpn3minutes.htm -- Bob Lin, MS-MVP, MCSE & CNE Networking, Internet, Routing, VPN Troubleshooting on http://www.ChicagoTech.net How to Setup Windows, Network, VPN & Remote Access on http://www.HowToNetworking.com
