Slow Network Speed over WAN

Discussion in 'Server Networking' started by David, Jan 10, 2008.

  1. David

    David Guest

    When copying files between two Windows 2003 SP2 Servers across our WAN, I
    can't get more than about 3Mbps for a single copy operation. When running
    multiple copies simultaneously, each one gets about 3Mbps up to the maximum
    bandwidth of the connection (on both a 1GB and a 100MB connection).

    My servers are in a data center on each coast of the US (one on the West
    Coast, the other on the East Coast), with about 100ms of latency. I have
    observed the performance limits with both a drag-and-drop from Windows
    Explorer and from Robocopy. When using iPerf with the default settings I see
    the same performance, but when running multiple simultaneous tests or
    increasing the TCP Window size I can get nearly my full speed.

    When I attempt to make any changes to the registry (Tcp1323Opts=3 and
    GlobalMaxTcpWindowSize={various values tried} in
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters) I get
    no performance increase.

    I've also tried adding DefaultSendWindow to
    [HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\AFD\Parameters] with no
    luck.
     
    David, Jan 10, 2008
    #1
    1. Advertisements

  2. Dear Customer,

    Thank you for your post. This is Neo and I will be assisting you in this
    post.

    From your description, I understand that:

    When files are copied through WAN connection between two Windows Server
    2003 SP2 Servers, the speed of one single session is limited to 3Mbps. When
    multiple sessions are running, the total speed can reach the maximum
    bandwidth of the WAN connection, however, the speed of each single session
    is still limited to 3Mbps. Any changes made to the registry don't help.

    If there is any misunderstanding, please let me know.

    This issue might be caused by WAN connection bandwidth limitation.

    In order to narrow down this issue, please help confirm the following
    questions with me. (Let's call the West Coast location as local site, the
    other as remote site.)

    1. Please describe the detailed Network Topology.

    2. Is there any routing/NAT devices between your two Windows Servers? If
    yes, please check whether there are any bandwidth limitation policies set
    on them.

    3. Please tell me the way how these two Servers communicate with each
    other. (Though VPN or WAN connection directly.)

    4. Please test the copy speed between two Windows Servers in the same local
    LAN and check whether the issue still occur.

    5. Please choose a different Windows Server located in remote side and test
    the copy speed between local site's Windows Server and remote site's
    Windows Server via WAN connection.

    6. Please try to download some big document from the internet on local
    site's Windows Server and test the download speed.

    After you finish the above tests, please tell me all the detailed result.

    Thank you for your time and I look forward to hearing from you.

    Sincerely,
    Neo Zhu,
    Microsoft Online Support
    Microsoft Global Technical Support Center

    Get Secure! - www.microsoft.com/security
    =====================================================
    When responding to posts, please "Reply to Group" via your newsreader so
    that others may learn and benefit from your issue.
    =====================================================
    This posting is provided "AS IS" with no warranties, and confers no rights.
     
    Jian-Ping Zhu [MSFT], Jan 11, 2008
    #2
    1. Advertisements

  3. David

    David Guest

    1. West Coast Site and Remote Site are on Gigabit Ethernet LAN with Gigabit
    to the public internet and are interconnected to each other by private IP
    transport (Routed Gigabit). Both are connected directly to the external
    router, bypassing the Firewall (Public IP addresses entered directly on the
    Windows NIC). They each also have a second interface/NIC on our private LAN
    (10.x.x.x IP range).
    2. Each site has a Fibre-connected gigabit router, no rate limit set.
    3. The servers can communicate with each other by either their public
    interface or their private interface. The private interface uses the private
    IP transport (effectively a site-to-site VPN), the Public interface uses the
    internet. Both have the same performance.
    4. Copies from the West Coast server to another in the same site happen at
    the expected speed (20%-40% utilization of the max interface speed), the same
    is seen at the remote site.
    5. Any server in the West Coast site has the same performance limitations
    when communicating with any server in the remote site.
    6. When using some of the available network speed performance test
    websites, I’ve gotten speeds of:
    Download Speed: 201673 kbps (25209.1 KB/sec transfer rate)
    Upload Speed: 18124 kbps (2265.5 KB/sec transfer rate)
    Those speeds are representative of either site (when testing against a test
    site geographically close).

    Additional information: When I test using iPerf, I can see 30%-40% NIC
    utilization when I adjust up the TCP window size. I can also see this
    performance when using multiple simultaneous copy operations (either FTP or
    Robocopy).
     
    David, Jan 11, 2008
    #3
  4. Dear Customer,

    Thank you for your reply.

    I performed further analysis and I noticed:

    1. If copy between two servers in the same site, the issue does not occur.

    2. The issue occurs with several different computers as long as the two
    related computers are in different sites.

    Therefore, I think the issue is not due to a Windows based setting or
    limitation. Instead, it should be very likely a network related limitation.
    To efficiently resolve the issue, I strongly recommend you contact the ISP
    and the router manufacturer for more investigation, to see if there are
    some known issues or settings on the router or on the ISP side.

    Thanks.

    Sincerely,
    Neo Zhu,
    Microsoft Online Support
    Microsoft Global Technical Support Center

    Get Secure! - www.microsoft.com/security
    =====================================================
    When responding to posts, please "Reply to Group" via your newsreader so
    that others may learn and benefit from your issue.
    =====================================================
    This posting is provided "AS IS" with no warranties, and confers no rights.
     
    Jian-Ping Zhu [MSFT], Jan 14, 2008
    #4
  5. David

    David Guest

    Thanks for your insight.

    I performed further analysis and I noticed:
    1. When running iPerf (on the two Microsoft Windows servers), after
    modifying the TCP Window Size option, performance tests show over 100 times
    faster than standard Windows copies.

    2. These tests are being performed across the WAN, so they appear to
    dispute your theory that it is a router/firewall/ISP limitation.

    Can you explain why the WAN is able to perform as expected, but native
    Windows copies cannot?

    Thanks.

    Sincerely,
    David.
     
    David, Jan 14, 2008
    #5
  6. David

    Ryan Hanisco Guest

    Hi David,

    Have a look to make sure you are not fragmenting encrypted packets on either
    end of the tunnel. If the packets are put into the VPN or otherwise
    encrypted, they'll grow, potentially pushing them past your MTU. This means
    that they'll fragment as they are shipped into the WAN. This is further
    complicated by Windowing as it allows more packets through before it
    validates making the reassembly processor intensive. You can lose half of
    your bandwidth ore more due to retransmits and the routing of nearly empty
    fragments. This theorem is supported by the fact that wrenching up the
    window size makes the problem less visible.

    You could attack this by dropping the MTU on the servers or by forcing the
    encryption after fragmentation. The latter option is done on your router (or
    switch if you're running one of the big ones for IP distribution -- sup 720
    or the like).

    Finally, you might also look at your QoS -- it is possible that you are
    being bumped in the transfer or aren't maintaining priority. Check this
    both locally and as you enter the IP cloud.

    Oh, and looking at SMB signing might also be worth while -- you can see a
    30% overhead there.
    --
    Ryan Hanisco
    MCSE, MCTS: SQL 2005, Project+
    http://www.techsterity.com
    Chicago, IL

    Remember: Marking helpful answers helps everyone find the info they need
    quickly.
     
    Ryan Hanisco, Jan 15, 2008
    #6
  7. Dear David,

    Thank you for your reply.

    I didn't know IPERF tool before, and after reading your e-mail I had a look
    at it. As far as I understand it has a server side service and a client
    side component. It looks like it measures network performance by sending
    raw data over TCP connection with several options such as changing TCP
    window size, changing amount of data to be sent etc. IPERF uses streaming
    protocol and it does not rely on the SMB (Server Message Block)
    application-layer protocol which is used when performing native Windows
    copies.

    I'd like to confirm whether there are any bandwidth limitation policies of
    specific sessions or QoS policies enabled on your Router or ISP side.
    Therefore, I still recommend you contact to the ISP and the Router
    manufacturer to confirm this first. This will ensure that we troubleshoot
    the issue in the most efficiently way and I appreciate your time.

    However, at the same time, I will also keep researching this issue for you.

    If I have made any progress, I'll inform you as soon as possible.

    Thanks.

    Sincerely,
    Neo Zhu,
    Microsoft Online Support
    Microsoft Global Technical Support Center

    Get Secure! - www.microsoft.com/security
    =====================================================
    When responding to posts, please "Reply to Group" via your newsreader so
    that others may learn and benefit from your issue.
    =====================================================
    This posting is provided "AS IS" with no warranties, and confers no rights.
     
    Jian-Ping Zhu [MSFT], Jan 15, 2008
    #7
  8. David

    David Guest

    Thanks. I've forwarded off your response (and this thread) to my ISP. They
    have said there is no rate limit on my interface, and they are checking
    further for anything on their end..
     
    David, Jan 15, 2008
    #8
  9. David

    David Guest

    Ryan - thanks for your time on this. I have checked for MTU, and it remains
    unfragmented to 1472 (I'm testing the network across the unencrypted WAN, not
    through a VPN tunnel).

    I haven't looked at SMB yet because of the huge disparity on performance.
    At less than 1% network utilization I don't think that the 30% SMB overhead
    is causing the problem.

    My ISP is saying there is no limitation or QOS, and so I'm further at a loss.
     
    David, Jan 15, 2008
    #9
  10. Dear David,

    I spent several hours investigating this issue and I'd like to share you
    some progress I have made up till now.

    According to your description, you have two large-bandwidth and
    long-latency WAN links on two sites.

    As we know, long latency is a challenging circumstance for the efficient
    operation of most network protocols, in particular those
    connection-oriented transport protocols such as TCP which rely on periodic
    acknowledgement of receipt. Maximum throughput can be calculated as the
    size of the payload that the sender can place on the wire before having to
    wait for confirmation multiplied by the number of times per second that
    acknowledgement is received and the next lot of payload can be sent. While
    decreasing link latency tends to be impractical or not cost-effective in
    most situations, the payload size is a function of protocol configuration.

    To improve throughput, enabling the TCP receive window scaling options is
    recommended, so that the effective maximum receive window size becomes a
    multiple of the advertised window. Also, by enabling TCP time stamps
    through the same registry bitmask value ('Tcp1323Opts'), will help improve
    effective use of the available bandwidth on this link. This could explain
    why after you modify the TCP Window Size option in IPERF tool, performance
    tests show much faster than before.

    I also notice that you have already tried to change registry to change
    windows scaling options and timestamp settings but with no help.

    After further research, I find out that there is an inherent limitation of
    the SMB protocol used by Windows Explorer and the copy command. SMB has an
    architectural limitation in the form of a 64KB buffer size limit which
    cannot be overcome through the use of TCP window scaling. Therefore, SMB
    tends to perform poorly over high latency WAN links.
    Fortunately, this limitation no longer exists in SMB 2.0 which is used on
    Windows Vista Operating System and Windows Server 2008.

    Having said above, I recommend you not use applications which rely on the
    SMB protocol to send large amounts of data across the long latency WAN
    links.
    You might use FTP as the protocol is faster and has the ability to continue
    a download were interrupted without having to start the entire file over
    again.
    Upgrading client OS to Windows Vista and servers OS to Longhorn will
    address the problem in the long term.

    I hope this helps.

    Thanks.

    Sincerely,
    Neo Zhu,
    Microsoft Online Support
    Microsoft Global Technical Support Center

    Get Secure! - www.microsoft.com/security
    =====================================================
    When responding to posts, please "Reply to Group" via your newsreader so
    that others may learn and benefit from your issue.
    =====================================================
    This posting is provided "AS IS" with no warranties, and confers no rights.
     
    Jian-Ping Zhu [MSFT], Jan 17, 2008
    #10
  11. David

    David Guest

    Thanks for your research. I have had better luck with FTP, but only getting
    about 7-8mbs. Can you let me know the proper registry keys/values to change
     
    David, Jan 17, 2008
    #11
  12. Dear David,

    Thank you for your post.

    Firstly, I'd like to share some more information about TCP Windows Size.

    The TCP receive window size is the amount of receive data (in bytes) that
    can be buffered during a connection. The sending host can send only that
    amount of data before it must wait for an acknowledgment and window update
    from the receiving host.

    For more efficient use of high bandwidth networks, a larger TCP window size
    may be used. The TCP window size field controls the flow of data and is
    limited to 2 bytes, or a window size of 65,535 bytes (64K). Since the size
    field cannot be expanded, a scaling factor is used. TCP window scale is an
    option used to increase the maximum window size from 65,535 bytes to 1
    Gigabyte.

    When the value for window size is added to the registry and its size is
    larger than the default value (64K), Windows attempts to use a scale value
    that accommodates the new window size. The window scale option is used only
    during the TCP 3-way handshake. The window scale value can be set from 0
    (no shift) to 14. To calculate the true window size, multiply the window
    size by 2^S where S is the scale value. For example, If the window size is
    65,535 bytes with a window scale factor of 4.
    True window size = 65535*2^4
    True window size = 1048560 (1MB)

    Having said above, the following are steps you might try to change your
    registry key.

    1. Back up the registry before you modify it.

    For more information about how to back up, restore, and modify the
    registry, please refer to the following KB article.
    http://support.microsoft.com/kb/256986/

    2. Click Start, click Run, type Regedit, and then click OK.

    3. Expand the registry subkey:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

    4. On the Edit menu, point to New, and then click DWORD Value.

    5. In the New Value box, type Tcp1323Opts, press ENTER, and then on the
    Edit menu, click Modify and Type 3.
    Note The valid range is 0,1,2 or 3 where:
    0 (disable RFC 1323 options)
    1 (window scale enabled only)
    2 (timestamps enabled only)
    3 (both options enabled)

    6. Expand the registry subkey:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

    7. On the Edit menu, point to New, and then click DWORD Value.

    8. Type TcpWindowSize in the New Value box, and then press Enter

    9. Click Modify on the Edit menu.

    10. Type the desired window size in the Value data box.
    Note. The valid range for window size is 0-0x3FFFC000 Hexadecimal, I
    recommend you try 100000 Hexadecim (1MB) first. If we set this value too
    large, some unexpected problems might occur.

    Please finish the above steps on both of your two Windows Server 2003
    machines and then test the performance again.

    For more information about Windows TCP feature, you may refer to the
    following article:

    Description of Windows 2000 and Windows Server 2003 TCP Features
    http://support.microsoft.com/kb/q224829/

    I hope this helps.

    Sincerely,
    Neo Zhu,
    Microsoft Online Support
    Microsoft Global Technical Support Center

    Get Secure! - www.microsoft.com/security
    =====================================================
    When responding to posts, please "Reply to Group" via your newsreader so
    that others may learn and benefit from your issue.
    =====================================================
    This posting is provided "AS IS" with no warranties, and confers no rights.
     
    Jian-Ping Zhu [MSFT], Jan 18, 2008
    #12
  13. Hello David,

    How's everything going?

    I'm wondering if the suggestion has helped or if you have any further
    questions.

    Please feel free to respond to the newsgroups if I can assist further.

    Sincerely,
    Neo Zhu,
    Microsoft Online Support
    Microsoft Global Technical Support Center

    Get Secure! - www.microsoft.com/security
    =====================================================
    When responding to posts, please "Reply to Group" via your newsreader so
    that others may learn and benefit from your issue.
    =====================================================
    This posting is provided "AS IS" with no warranties, and confers no rights.
     
    Jian-Ping Zhu [MSFT], Jan 23, 2008
    #13
  14. David

    Jose Bonilla Guest

    I am trying to copy a file from one server to other server with the Windows Explorer but this devices are connected over Internet through a VPN site-to-Site network.

    Well, I can copy files over the link until 1024 MB, bu when I was trying to copy a file over 1024 MB the process was interrupted and I need to start again the process without resume options.

    This problem is not permanent, sometimes I can copy files over 1024 MB without problems.

    I call my ISP but they told me that is not a problem with their networks, then I want to know if I can change some parameters on the windows registry to rise the amount of wait time for the ACK and let more time to kill the copy process.

    Regards,
    Jose Bonilla
     
    Jose Bonilla, Jun 24, 2008
    #14
  15. I am trying to copy a file from one server to other server with the Windows
    I would suggest using robocopy, from the resource kit.


    hth
    DDS
     
    Danny Sanders, Jun 24, 2008
    #15
  16. Yes, try robocopy first. If robocopy doesn't work, check the MTU size. This
    search result may help too.
    VPN connection is disconnected after several minutes
    VPN drops the connection. VPN is very slow. To change the MTU
    Settings, ... Resolution: Set my VPN client MTU to 1400. To modify MTU,
    please refer to this ...
    www.chicagotech.net/VPN/vpn3minutes.htm


    --
    Bob Lin, MS-MVP, MCSE & CNE
    Networking, Internet, Routing, VPN Troubleshooting on
    http://www.ChicagoTech.net
    How to Setup Windows, Network, VPN & Remote Access on
    http://www.HowToNetworking.com
     
    Robert L. \(MS-MVP\), Jun 25, 2008
    #16
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.