Finding the bandwidth bandit

Discussion in 'Server Networking' started by Jim in Arizona, Aug 6, 2009.

  1. Out network consists of about 30 users (PCs) on an internal network which
    goes out via a linksys router to the internet through a dual T1 (running at
    3mb up and down).
    For a few months now we've been experiencing severe bandwidth loss during
    the course of the day and I have been unable to find the cause. A few days
    ago I walked from comptuer to computer to see if I could find the user
    responsible but was unable to. I did find one user running FrostWire, a form
    of P2P file sharing software but after uninstalling the software and
    removing the installation rights from that user, the problem did not go away
    so it's still somewhere else on the network. I have, through the process of
    elimination, determined that it is not either one of my two servers doing
    it.

    Our bandwidth speeds will drop pretty low, looking like this (I took speed
    tests yesterday via qwest's speed test site from my workstation)

    Time Download/Upload
    8:39AM 2.745/2.597
    9:42AM 2.207/1.204
    10:44AM 2.003/1.090
    12:13PM 0.915/0.845
    2:08PM 1.826/0.584

    These speeds from yesterday aren't too bad except for the upload speeds.
    They're usually much worse. Often, the download speeds are down to 0.300 or
    so with the upload speeds a bit worse .. around 200K. So far, right now (as
    I type this), no one seems to be sucking up the bandwidth. The speed test
    right now check in at 2.707/1.593 DOWN/UP.

    My boss seems to think that my efforts aren't working in getting results
    (which they aren't) and although I have offered the solution of temporarily
    installing an ISA server over the weekend to see if we can catch the culprit
    that way, my boss asked me if we should bring in some 'experts'. Maybe he
    didn't read my entire email.

    In any case, if I were to bring in some 'experts', what would they be able
    to do?

    My network looks like this:


    LinkSys Router --- Switches ---
    Workstations and Domain Controller
    -
    -
    Internet ----- ADTRAN ------ Switch ----- Web/FTP Server
    -
    -
    ----------
    Teleconference Equipment


    Its my plan to place an ISA server in place of the Linksys router on a
    temporary basis until I find the person responsible.
    The Web/FTP Server and Teleconference equipment have public IPs so that's
    why they're not behind the linksys router. I know for sure that the
    teleconference equipment is not the culprit as it takes a specific amount of
    bandwidth up when running and I know what that is. Also, I know its behind
    the linksys because when the bandwidth is really bad, I can power cycle the
    linksys router or just unplug the LAN cable from it for a few moments and
    then plug it back in and our bandwidth is back up and running normally for a
    short while until the bad PC reconnects to whatever systems it's
    communicating with on the net.

    So? Any ideas on how to do this other than ISA? I'm not good at using packet
    sniffing software like wireshark and even if I could, where would I plug a
    laptop running such software into? Swiches only route to predefined ports so
    I would need an old style hub in between the switch and linksys router,
    which I don't have.

    Back to my last question, what would an 'expert' even do if we could call
    one in?
     
    Jim in Arizona, Aug 6, 2009
    #1
    1. Advertisements

  2. ISA's Reports will give "hints" to who it might be,...it is not going to
    announce them.

    Home user Linksys NAT boxes are not sufficient for businesses. You should
    have already replaced the Linsys box with the ISA a log time ago.
     
    Phillip Windell, Aug 6, 2009
    #2
    1. Advertisements


  3. I agree that using a linksys nat box is not a good idea. However, I get a
    lot of resistance when asking for anything where I'm working. The network
    was set up by a contractor and they have a VPN tunnel between our plant, the
    corporate office and rackspace where the FSMO DC Sits (yea, I know) and use
    these Linksys VPN boxes to keep up the VPN tunnels. Want a even better laugh
    (as an unrelated note)? They even use this linksys router as the DHCP
    server. ;) I'm planning on changing that this coming weekend when I set up
    the ISA server. The only downside to doing that, which I have done before,
    is the Konica Minoltas that people use to scan documents to their computers
    is very sensitive when it comes to network changes and after two attempts at
    making the change on a weekday, I gave up that idea and have to plan to
    spend a good part of my weekend to get it done and test test test.

    The ISA server will be able to tell me which user is using up the most
    bandwidth and I have a strong feeling that whomever ISA says that person is,
    is most likely the one sucking up the bandwidth all day long. Am I right to
    make that assumption?

    Also, it will be able to tell me where everyone is going and if it's a file
    sharing network, that may also be a clue. But I've never done an
    investigation into ISA logs where I found any file sharing network IPs or
    DNS entries so I won't know what one looks like right off but once I
    invetigate the highest bandwidth user's log entries, I should be able to
    piece it all together.
     
    Jim in Arizona, Aug 6, 2009
    #3
  4. Hi Jim,

    Natively, this is not ISA's bag. However the following may help:

    Bandwidth ControlBandwidth Splitter is a program extension for Microsoft ISA
    Server that ... real time with the built in traffic monitor; Advanced
    bandwidth management: Use ...
    www.isaserver.org/software/ISA/Bandwidth-Control/

    free isa bandwidth monitor downloadAccess Monitor is a comprehensive
    Internet use monitoring and reporting utility for corporate networks. The
    program takes advantage of the fact that most ...
    http://3d2f.com/tags/isa/bandwidth/monitor/

    The real deal and the real McCoy, is to use something like Blue Coat and
    Packeteer. The thing is awesome. Not only can you find the culpript(s), but
    you can throttle various types of traffic by using policies. Kind of pricey.
    Case in point, one of my old customers had a similar problem. Tried various
    tools, but they only hinted at the additional bandwidth being consumed. They
    even bumped up the speed to two T1s, such as what you have, but it didn;t
    help. Users constantly complaining 'everything is slow.' After some research
    into various products, we got the Ok to get Blue Coat and Packeteer with a
    30 day trial (if I remember correctly), and immediately we found out who
    THEY were. They were watching numerous YouTube and other videos. One guy was
    using a P2P on a Mac as well. For the first few weeks, we would just call
    them up telling them we see them what they're up to using up the bandwidth
    and to cut it out, as well as told the one guy to uninstall that P2P . Sure,
    they said, but an hour or two later, they're at it again. Finally we
    instituted policies to throttle YouTube and other vid sites (they have built
    in policies and you make your own) to throttle them to 10%, as well as block
    P2P. They got upset (I had another word in mind...), but they had no choice
    but to live wtih it. The boss and the rest of the user base were happy that
    the 'slowness' was no longer there.

    Like I said, it's not cheap, but well worth the investment.

    Blue Coat has acquired Packeteer...
    http://www.bluecoat.com/packeteer

    However, you also stated there is a 24/7 VPN to a rack hosting company. What
    type of applications are installed and running, as well as being accessed
    across the WAN link? There's a possibility that this is eating up bandwidth,
    too, which would make it legit traffic.


    --
    Ace

    This posting is provided "AS-IS" with no warranties or guarantees and
    confers no rights.

    Please reply back to the newsgroup or forum to benefit from collaboration
    among responding engineers, and to help others benefit from your resolution.

    Ace Fekay, MCT, MCTS Exchange, MCSE, MCSA 2003 & 2000, MCSA Messaging
    Microsoft Certified Trainer

    For urgent issues, please contact Microsoft PSS directly. Please check
    http://support.microsoft.com for regional support phone numbers.
     
    Ace Fekay [MCT], Aug 7, 2009
    #4
  5. No. ISA is not going to be the "sledgehammer" to solve that. It will only
    give hints to who or what is doing this in the Reports (not logs). It will
    be in the form of "Top 10 Users" in various catagories. Just because a
    user may be the top "1 out of 10" does not mean they are clogging up the
    network,...it only means they are doing more than the other 9, so you have
    to approach it in the right context with the right perspective.

    There are third party plugins for ISA as Ace has mentioned. They cost $$$
    for most anything worth having. This kind of stuff is difficult for any
    product to do accurately,...it is not a simple job,...so any product that
    does it very well at all is going to be $$$,...hence why cheap Linksys boxes
    don't do it or can't do it well.

    --
    Phillip Windell

    The views expressed, are my own and not those of my employer, or Microsoft,
    or anyone else associated with me, including my cats.
    -----------------------------------------------------
     
    Phillip Windell, Aug 7, 2009
    #5

  6. Thanks Ace and Phillip.

    I've got the ISA box set up and will make the appropriate network changes
    this weekend, making everyone secure nat clients. I haven't decided if I'm
    going to install the firewall client on all machines yet but I may need to
    in order to get better reports.

    I'm hoping that finding the highest bandwidth users will be enough to narrow
    my search and find the person I'm looking for. I'll carefully inspect the
    top five users's computers for anything out of the ordinary and maybe
    perform an interview or two and I'm hoping that will be enough to put a stop
    to this mess.

    The place I work is still under construction and we have yet to implement
    our final network configuration (it's a very large cement manufacturing
    plant with its own mining operations so we have yet to lay the fiber
    throughout the plan and implent all the variou systems; our network in place
    right now is more or less a temporary domain which we may or may not
    continue to use when the plant goes live). As the final days of plant
    construction come up, I'll be looking into other more permanent options for
    our network configuration and I will be looking into these other products
    you both have mentioned.

    I'll also post back here with my ISA results, assuming it works for what I'm
    using it for at this moment.
     
    Jim in Arizona, Aug 7, 2009
    #6
  7. You don't want SecureNAT Clients.
    SecureNAT Clients will not authenticate, not identify the user, will not
    show the URL or the Domain Name that was targeted.

    You want the Clients to be both Web Proxy (browser proxy settings) and a
    Firewall Client at the same time. That will give the most details in both
    the logs and the Reports. It is even better if you configure the LAN for
    Proxy autodetection via WPAD through DNS and DHCP.
    I would not try to create a new domain later. Keep the one your have. No
    reason to create all that extra work.

    --
    Phillip Windell

    The views expressed, are my own and not those of my employer, or Microsoft,
    or anyone else associated with me, including my cats.
    -----------------------------------------------------
     
    Phillip Windell, Aug 7, 2009
    #7
  8. Bill,

    I've used this method using a utility called NetBoy, as well as NetMON
    sniffing traffic at that location I previously mentioned, as well as others.
    That was before I got the approval for Packeteer, because it just wasn't
    enough. I mean I would see one user spike (Netboy shows it in realtime),
    then another would spike, but I couldn't exactly tell if it was legit or
    not. It was alot of work, and I was charging them by the hour. That was one
    of the reasons they approved the appliance, they didn't want to pay for the
    time I was spending on this issue. Oh well... I didn't mind. :)

    Ace
     
    Ace Fekay [MCT], Aug 8, 2009
    #8
  9. You are welcome. However, I must agree with Phillip. You are not going to
    get the satisfaction that you're looking for.

    Ace
     
    Ace Fekay [MCT], Aug 8, 2009
    #9
  10. That was my concern also; looking at packets and not knowing for sure what
    was bad and good, even if one specific host was generating more traffic than
    another. This would also require my constant attention to get the job done
    which I just don't thave the time (or the immediately avail resources) to
    work with such a method.
     
    Jim in Arizona, Aug 10, 2009
    #10
  11. I realize that this method won't work as well as some of the aforementioned
    software packages available but I have to work with what I have readily
    available. Setting up ISA will at least give me a lot more options than what
    I had before, which were none. I can check the logs the next day and get a
    real good idea of who may be the person I'm looking for and the next day, if
    we're facing a bandwidth issue, I'll just cut that one specific person off
    for a few minutes via ISA and see what happens. I may even be lucky enough
    to see in the reports that a specific protocol was using up most of the
    bandwidth and just prevent that application from going outbound. We'll see
    if it's that easy and I will know here in the next few days.
     
    Jim in Arizona, Aug 10, 2009
    #11
  12. You keep saying logs. It is not the logs that are going to help. Even with
    normal traffic on a good sunny day the logs are going to scrow so fast you
    eyes can barely follow it. It will be useless. It is the Reports gernerated
    in the reports section that is going to help at all. The reports are
    generated from the logs of course,..but the process handles all the "math"
    to make the reports mean anything. The Clients will need to be both Web
    Proxy and Firewall Clients at the same time for the reports to have any kind
    of detailed information. SecureNAT Clients will be a waste of time.

    --
    Phillip Windell

    The views expressed, are my own and not those of my employer, or Microsoft,
    or anyone else associated with me, including my cats.
    -----------------------------------------------------
     
    Phillip Windell, Aug 10, 2009
    #12

  13. Looking forward to your results!

    Ace
     
    Ace Fekay [MCT], Aug 10, 2009
    #13
  14. It's justifiable if you're hourly. :)

    Ace
     
    Ace Fekay [MCT], Aug 10, 2009
    #14
  15. Right right. I keep calling them logs when I mean reports. In the past I
    would look at the reports for an overall view then go through the logs as
    ncessary to see exactly what people were viewing (I would often export into
    excel, do some column sorting and search for specific things). Honestly, I
    never cared much of what people were viewing, it was the other IT people I
    once worked with that were so anal. In any case, I'm hoping ISA will help me
    solve my issue without having to buy more software. I consider this issue
    tempory and hoping that ISA will be enough for all future problems where I
    currently work. I started with ISA 2K about 5 years ago and really liked it
    then moved on to 2004 then 2006. I'm currently using 04 because I remember
    having issues with 06 but can't remember what they were. I know there were
    some site to site vpn issues with ISA that made a previous employer go to
    cisco hardware for the site to site VPNs instead of using ISA. Anyway.
    Enough rambling. I'll let you know of my results. :)
     
    Jim in Arizona, Aug 10, 2009
    #15
  16. We have Dell Powerconnect 2700 series switches on the network. I don't know
    if they're the managed type or not. All my managed switch experience is with
    Cicsco switches, which doesn't help me much with these Dell's. I just
    assumed they're not managed switches.

    I've set up an ISA server and so far, our bandwidth issue has disappeared.
    The problem just seemed to have mysteriously stopped last Friday although I
    did not set up the ISA server until Saturday. It's possilbe that someone got
    wind of what was happening and ceased their activity. Since I've installed
    the ISA server, I've seen a lot of streaming radio activity adding up to a
    few gigs of data transfer a day but it has not adversely affected our
    internet use. So, either that was the problem before and the Linksys VPN
    router that was acting as our internet gateway was having problems working
    with this traffic (and the ISA server works great at handling it) (It has
    had issues with other types of data, like video streaming in which it messed
    up the video signal and we had to reroute the video strait out to our
    internet router (adtran router) instead of through the DMZ port on the
    linksys router), or the person who was causing issues stopped what they were
    doing. Whatever the case, the problem has stopped and if it ever comes up
    again, I feel confident in being able to isolate the host wtihin a day's
    time when before I had no options at all.
     
    Jim in Arizona, Aug 13, 2009
    #16

  17. Sometimes just telling your users you will start monitoring usage will make
    the issue go away. Isn't it amazing?!!

    Glad you got it nailed down.
    :)

    Ace
     
    Ace Fekay [MCT], Aug 14, 2009
    #17
  18. I don't see a switch helping that much. Switches are Layer2,...all they
    know are MAC addresses. They are not aware of anything above Layer3 and are
    only aware of Layer3 with respct to the ARP Tables tying MACs to IP#s.
    Protocols like HTTP, MMS, RTSM, FTP, etc, are all Application Layer
    Protocols. Yes they are associated with port addrersses (Layer4) but ports
    and protocols are not the same thing.

    --
    Phillip Windell

    The views expressed, are my own and not those of my employer, or Microsoft,
    or anyone else associated with me, including my cats.
    -----------------------------------------------------
     
    Phillip Windell, Aug 14, 2009
    #18
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.