Finding the bandwidth bandit

Out network consists of about 30 users (PCs) on an internal network which
goes out via a linksys router to the internet through a dual T1 (running at
3mb up and down).
For a few months now we've been experiencing severe bandwidth loss during
the course of the day and I have been unable to find the cause. A few days
ago I walked from comptuer to computer to see if I could find the user
responsible but was unable to. I did find one user running FrostWire, a form
of P2P file sharing software but after uninstalling the software and
removing the installation rights from that user, the problem did not go away
so it's still somewhere else on the network. I have, through the process of
elimination, determined that it is not either one of my two servers doing
it.

Our bandwidth speeds will drop pretty low, looking like this (I took speed
tests yesterday via qwest's speed test site from my workstation)

Time Download/Upload
8:39AM 2.745/2.597
9:42AM 2.207/1.204
10:44AM 2.003/1.090
12:13PM 0.915/0.845
2:08PM 1.826/0.584

These speeds from yesterday aren't too bad except for the upload speeds.
They're usually much worse. Often, the download speeds are down to 0.300 or
so with the upload speeds a bit worse .. around 200K. So far, right now (as
I type this), no one seems to be sucking up the bandwidth. The speed test
right now check in at 2.707/1.593 DOWN/UP.

My boss seems to think that my efforts aren't working in getting results
(which they aren't) and although I have offered the solution of temporarily
installing an ISA server over the weekend to see if we can catch the culprit
that way, my boss asked me if we should bring in some 'experts'. Maybe he
didn't read my entire email.

In any case, if I were to bring in some 'experts', what would they be able
to do?

My network looks like this:


LinkSys Router --- Switches ---
Workstations and Domain Controller
-
-
Internet ----- ADTRAN ------ Switch ----- Web/FTP Server
-
-
----------
Teleconference Equipment


Its my plan to place an ISA server in place of the Linksys router on a
temporary basis until I find the person responsible.
The Web/FTP Server and Teleconference equipment have public IPs so that's
why they're not behind the linksys router. I know for sure that the
teleconference equipment is not the culprit as it takes a specific amount of
bandwidth up when running and I know what that is. Also, I know its behind
the linksys because when the bandwidth is really bad, I can power cycle the
linksys router or just unplug the LAN cable from it for a few moments and
then plug it back in and our bandwidth is back up and running normally for a
short while until the bad PC reconnects to whatever systems it's
communicating with on the net.

So? Any ideas on how to do this other than ISA? I'm not good at using packet
sniffing software like wireshark and even if I could, where would I plug a
laptop running such software into? Swiches only route to predefined ports so
I would need an old style hub in between the switch and linksys router,
which I don't have.

Back to my last question, what would an 'expert' even do if we could call
one in?

ISA's Reports will give "hints" to who it might be,...it is not going to
announce them.

Home user Linksys NAT boxes are not sufficient for businesses. You should
have already replaced the Linsys box with the ISA a log time ago.

--
Phillip Windell

The views expressed, are my own and not those of my employer, or Microsoft,
or anyone else associated with me, including my cats.
-----------------------------------------------------



"Jim in Arizona" <> wrote in message
news:%...
> Out network consists of about 30 users (PCs) on an internal network which
> goes out via a linksys router to the internet through a dual T1 (running
> at 3mb up and down).
> For a few months now we've been experiencing severe bandwidth loss during
> the course of the day and I have been unable to find the cause. A few days
> ago I walked from comptuer to computer to see if I could find the user
> responsible but was unable to. I did find one user running FrostWire, a
> form of P2P file sharing software but after uninstalling the software and
> removing the installation rights from that user, the problem did not go
> away so it's still somewhere else on the network. I have, through the
> process of elimination, determined that it is not either one of my two
> servers doing it.
>
> Our bandwidth speeds will drop pretty low, looking like this (I took speed
> tests yesterday via qwest's speed test site from my workstation)
>
> Time Download/Upload
> 8:39AM 2.745/2.597
> 9:42AM 2.207/1.204
> 10:44AM 2.003/1.090
> 12:13PM 0.915/0.845
> 2:08PM 1.826/0.584
>
> These speeds from yesterday aren't too bad except for the upload speeds.
> They're usually much worse. Often, the download speeds are down to 0.300
> or so with the upload speeds a bit worse .. around 200K. So far, right now
> (as I type this), no one seems to be sucking up the bandwidth. The speed
> test right now check in at 2.707/1.593 DOWN/UP.
>
> My boss seems to think that my efforts aren't working in getting results
> (which they aren't) and although I have offered the solution of
> temporarily installing an ISA server over the weekend to see if we can
> catch the culprit that way, my boss asked me if we should bring in some
> 'experts'. Maybe he didn't read my entire email.
>
> In any case, if I were to bring in some 'experts', what would they be able
> to do?
>
> My network looks like this:
>
>
> LinkSys Router ---
> Switches --- Workstations and Domain Controller
> -
> -
> Internet ----- ADTRAN ------ Switch ----- Web/FTP Server
> -
> -
> ----------
> Teleconference Equipment
>
>
> Its my plan to place an ISA server in place of the Linksys router on a
> temporary basis until I find the person responsible.
> The Web/FTP Server and Teleconference equipment have public IPs so that's
> why they're not behind the linksys router. I know for sure that the
> teleconference equipment is not the culprit as it takes a specific amount
> of bandwidth up when running and I know what that is. Also, I know its
> behind the linksys because when the bandwidth is really bad, I can power
> cycle the linksys router or just unplug the LAN cable from it for a few
> moments and then plug it back in and our bandwidth is back up and running
> normally for a short while until the bad PC reconnects to whatever systems
> it's communicating with on the net.
>
> So? Any ideas on how to do this other than ISA? I'm not good at using
> packet sniffing software like wireshark and even if I could, where would I
> plug a laptop running such software into? Swiches only route to predefined
> ports so I would need an old style hub in between the switch and linksys
> router, which I don't have.
>
> Back to my last question, what would an 'expert' even do if we could call
> one in?
>
>
>
>
>
>

"Phillip Windell" <> wrote in message
news:%...
>
> ISA's Reports will give "hints" to who it might be,...it is not going to
> announce them.
>
> Home user Linksys NAT boxes are not sufficient for businesses. You should
> have already replaced the Linsys box with the ISA a log time ago.
>
> --
> Phillip Windell
>
> The views expressed, are my own and not those of my employer, or
> Microsoft,
> or anyone else associated with me, including my cats.
> -----------------------------------------------------
>
> <SNIP>
>


I agree that using a linksys nat box is not a good idea. However, I get a
lot of resistance when asking for anything where I'm working. The network
was set up by a contractor and they have a VPN tunnel between our plant, the
corporate office and rackspace where the FSMO DC Sits (yea, I know) and use
these Linksys VPN boxes to keep up the VPN tunnels. Want a even better laugh
(as an unrelated note)? They even use this linksys router as the DHCP
server. I'm planning on changing that this coming weekend when I set up
the ISA server. The only downside to doing that, which I have done before,
is the Konica Minoltas that people use to scan documents to their computers
is very sensitive when it comes to network changes and after two attempts at
making the change on a weekday, I gave up that idea and have to plan to
spend a good part of my weekend to get it done and test test test.

The ISA server will be able to tell me which user is using up the most
bandwidth and I have a strong feeling that whomever ISA says that person is,
is most likely the one sucking up the bandwidth all day long. Am I right to
make that assumption?

Also, it will be able to tell me where everyone is going and if it's a file
sharing network, that may also be a clue. But I've never done an
investigation into ISA logs where I found any file sharing network IPs or
DNS entries so I won't know what one looks like right off but once I
invetigate the highest bandwidth user's log entries, I should be able to
piece it all together.

"Jim in Arizona" <> wrote in message
news:%...
>
> I agree that using a linksys nat box is not a good idea. However, I get a
> lot of resistance when asking for anything where I'm working. The network
> was set up by a contractor and they have a VPN tunnel between our plant,
> the corporate office and rackspace where the FSMO DC Sits (yea, I know)
> and use these Linksys VPN boxes to keep up the VPN tunnels. Want a even
> better laugh (as an unrelated note)? They even use this linksys router as
> the DHCP server. I'm planning on changing that this coming weekend when
> I set up the ISA server. The only downside to doing that, which I have
> done before, is the Konica Minoltas that people use to scan documents to
> their computers is very sensitive when it comes to network changes and
> after two attempts at making the change on a weekday, I gave up that idea
> and have to plan to spend a good part of my weekend to get it done and
> test test test.
>
> The ISA server will be able to tell me which user is using up the most
> bandwidth and I have a strong feeling that whomever ISA says that person
> is, is most likely the one sucking up the bandwidth all day long. Am I
> right to make that assumption?
>
> Also, it will be able to tell me where everyone is going and if it's a
> file sharing network, that may also be a clue. But I've never done an
> investigation into ISA logs where I found any file sharing network IPs or
> DNS entries so I won't know what one looks like right off but once I
> invetigate the highest bandwidth user's log entries, I should be able to
> piece it all together.

Hi Jim,

Natively, this is not ISA's bag. However the following may help:

Bandwidth ControlBandwidth Splitter is a program extension for Microsoft ISA
Server that ... real time with the built in traffic monitor; Advanced
bandwidth management: Use ...
http://www.isaserver.org/software/IS...width-Control/

free isa bandwidth monitor downloadAccess Monitor is a comprehensive
Internet use monitoring and reporting utility for corporate networks. The
program takes advantage of the fact that most ...
http://3d2f.com/tags/isa/bandwidth/monitor/

The real deal and the real McCoy, is to use something like Blue Coat and
Packeteer. The thing is awesome. Not only can you find the culpript(s), but
you can throttle various types of traffic by using policies. Kind of pricey.
Case in point, one of my old customers had a similar problem. Tried various
tools, but they only hinted at the additional bandwidth being consumed. They
even bumped up the speed to two T1s, such as what you have, but it didn;t
help. Users constantly complaining 'everything is slow.' After some research
into various products, we got the Ok to get Blue Coat and Packeteer with a
30 day trial (if I remember correctly), and immediately we found out who
THEY were. They were watching numerous YouTube and other videos. One guy was
using a P2P on a Mac as well. For the first few weeks, we would just call
them up telling them we see them what they're up to using up the bandwidth
and to cut it out, as well as told the one guy to uninstall that P2P . Sure,
they said, but an hour or two later, they're at it again. Finally we
instituted policies to throttle YouTube and other vid sites (they have built
in policies and you make your own) to throttle them to 10%, as well as block
P2P. They got upset (I had another word in mind...), but they had no choice
but to live wtih it. The boss and the rest of the user base were happy that
the 'slowness' was no longer there.

Like I said, it's not cheap, but well worth the investment.

Blue Coat has acquired Packeteer...
http://www.bluecoat.com/packeteer

However, you also stated there is a 24/7 VPN to a rack hosting company. What
type of applications are installed and running, as well as being accessed
across the WAN link? There's a possibility that this is eating up bandwidth,
too, which would make it legit traffic.


--
Ace

This posting is provided "AS-IS" with no warranties or guarantees and
confers no rights.

Please reply back to the newsgroup or forum to benefit from collaboration
among responding engineers, and to help others benefit from your resolution.

Ace Fekay, MCT, MCTS Exchange, MCSE, MCSA 2003 & 2000, MCSA Messaging
Microsoft Certified Trainer

For urgent issues, please contact Microsoft PSS directly. Please check
http://support.microsoft.com for regional support phone numbers.

"Jim in Arizona" <> wrote in message
news:%...
> The ISA server will be able to tell me which user is using up the most
> bandwidth and I have a strong feeling that whomever ISA says that person
> is, is most likely the one sucking up the bandwidth all day long. Am I
> right to make that assumption?

No. ISA is not going to be the "sledgehammer" to solve that. It will only
give hints to who or what is doing this in the Reports (not logs). It will
be in the form of "Top 10 Users" in various catagories. Just because a
user may be the top "1 out of 10" does not mean they are clogging up the
network,...it only means they are doing more than the other 9, so you have
to approach it in the right context with the right perspective.

There are third party plugins for ISA as Ace has mentioned. They cost $$$
for most anything worth having. This kind of stuff is difficult for any
product to do accurately,...it is not a simple job,...so any product that
does it very well at all is going to be $$$,...hence why cheap Linksys boxes
don't do it or can't do it well.

--
Phillip Windell

The views expressed, are my own and not those of my employer, or Microsoft,
or anyone else associated with me, including my cats.
-----------------------------------------------------

"Phillip Windell" <> wrote in message
news:%...
> "Jim in Arizona" <> wrote in message
> news:%...
>> The ISA server will be able to tell me which user is using up the most
>> bandwidth and I have a strong feeling that whomever ISA says that person
>> is, is most likely the one sucking up the bandwidth all day long. Am I
>> right to make that assumption?
>
> No. ISA is not going to be the "sledgehammer" to solve that. It will only
> give hints to who or what is doing this in the Reports (not logs). It
> will be in the form of "Top 10 Users" in various catagories. Just
> because a user may be the top "1 out of 10" does not mean they are
> clogging up the network,...it only means they are doing more than the
> other 9, so you have to approach it in the right context with the right
> perspective.
>
> There are third party plugins for ISA as Ace has mentioned. They cost $$$
> for most anything worth having. This kind of stuff is difficult for any
> product to do accurately,...it is not a simple job,...so any product that
> does it very well at all is going to be $$$,...hence why cheap Linksys
> boxes don't do it or can't do it well.
>
> --
> Phillip Windell
>
> The views expressed, are my own and not those of my employer, or
> Microsoft,
> or anyone else associated with me, including my cats.
> -----------------------------------------------------


Thanks Ace and Phillip.

I've got the ISA box set up and will make the appropriate network changes
this weekend, making everyone secure nat clients. I haven't decided if I'm
going to install the firewall client on all machines yet but I may need to
in order to get better reports.

I'm hoping that finding the highest bandwidth users will be enough to narrow
my search and find the person I'm looking for. I'll carefully inspect the
top five users's computers for anything out of the ordinary and maybe
perform an interview or two and I'm hoping that will be enough to put a stop
to this mess.

The place I work is still under construction and we have yet to implement
our final network configuration (it's a very large cement manufacturing
plant with its own mining operations so we have yet to lay the fiber
throughout the plan and implent all the variou systems; our network in place
right now is more or less a temporary domain which we may or may not
continue to use when the plant goes live). As the final days of plant
construction come up, I'll be looking into other more permanent options for
our network configuration and I will be looking into these other products
you both have mentioned.

I'll also post back here with my ISA results, assuming it works for what I'm
using it for at this moment.

"Jim in Arizona" <> wrote in message
news:%...
> I've got the ISA box set up and will make the appropriate network changes
> this weekend, making everyone secure nat clients. I haven't decided if I'm
> going to install the firewall client on all machines yet but I may need to
> in order to get better reports.

You don't want SecureNAT Clients.
SecureNAT Clients will not authenticate, not identify the user, will not
show the URL or the Domain Name that was targeted.

You want the Clients to be both Web Proxy (browser proxy settings) and a
Firewall Client at the same time. That will give the most details in both
the logs and the Reports. It is even better if you configure the LAN for
Proxy autodetection via WPAD through DNS and DHCP.

> The place I work is still under construction and we have yet to implement
> our final network configuration (it's a very large cement manufacturing
> plant with its own mining operations so we have yet to lay the fiber
> throughout the plan and implent all the variou systems; our network in
> place right now is more or less a temporary domain which we may or may not
> continue to use when the plant goes live).

I would not try to create a new domain later. Keep the one your have. No
reason to create all that extra work.

--
Phillip Windell

The views expressed, are my own and not those of my employer, or Microsoft,
or anyone else associated with me, including my cats.
-----------------------------------------------------

"Bill Kearney" <> wrote in message
news:ZbCdnd0r_8WwhuHXnZ2dnUVZ_s-...
>> Back to my last question, what would an 'expert' even do if we could call
>> one in?
>
> Having done this sort of thing myself, here's a low-cost suggestion.
>
> Install an ethernet hub between the main office network and the VPN
> router. Then also plug a computer running Wireshark into the hub.
>
> http://www.wireshark.org/download.html
>
> Wireshark being a packet sniffer. It'll collect the packets being sent
> across the link to the hub. You'd have to use a hub because that's the
> only way to listen to ALL packets on the wire. A switch (generally) will
> only pass packets destined for that port. You generally don't use hubs in
> regular use because of how they share all traffic. But they're darned
> handy for this sort of monitoring.
>
> For this monitoring to work without changing anything on your network you
> HAVE TO USE A HUB. You CANNOT use a switch (at least not unless it has
> more advanced built-in controls for allowing this sort of thing). To do
> it with a PC would require two network interfaces on it and some changes
> to your network. Neither of which is anywhere as easy to setup as a hub.

Bill,

I've used this method using a utility called NetBoy, as well as NetMON
sniffing traffic at that location I previously mentioned, as well as others.
That was before I got the approval for Packeteer, because it just wasn't
enough. I mean I would see one user spike (Netboy shows it in realtime),
then another would spike, but I couldn't exactly tell if it was legit or
not. It was alot of work, and I was charging them by the hour. That was one
of the reasons they approved the appliance, they didn't want to pay for the
time I was spending on this issue. Oh well... I didn't mind. :-)

Ace

"Jim in Arizona" <> wrote in message
news:%...
>
> Thanks Ace and Phillip.
>
> I've got the ISA box set up and will make the appropriate network changes
> this weekend, making everyone secure nat clients. I haven't decided if I'm
> going to install the firewall client on all machines yet but I may need to
> in order to get better reports.
>
> I'm hoping that finding the highest bandwidth users will be enough to
> narrow my search and find the person I'm looking for. I'll carefully
> inspect the top five users's computers for anything out of the ordinary
> and maybe perform an interview or two and I'm hoping that will be enough
> to put a stop to this mess.
>
> The place I work is still under construction and we have yet to implement
> our final network configuration (it's a very large cement manufacturing
> plant with its own mining operations so we have yet to lay the fiber
> throughout the plan and implent all the variou systems; our network in
> place right now is more or less a temporary domain which we may or may not
> continue to use when the plant goes live). As the final days of plant
> construction come up, I'll be looking into other more permanent options
> for our network configuration and I will be looking into these other
> products you both have mentioned.
>
> I'll also post back here with my ISA results, assuming it works for what
> I'm using it for at this moment.

You are welcome. However, I must agree with Phillip. You are not going to
get the satisfaction that you're looking for.

Ace

"Ace Fekay [MCT]" <> wrote in message
news:%...
> "Bill Kearney" <> wrote in message
> news:ZbCdnd0r_8WwhuHXnZ2dnUVZ_s-...
>>> Back to my last question, what would an 'expert' even do if we could
>>> call one in?
>>
>> Having done this sort of thing myself, here's a low-cost suggestion.
>>
>> Install an ethernet hub between the main office network and the VPN
>> router. Then also plug a computer running Wireshark into the hub.
>>
>> http://www.wireshark.org/download.html
>>
>> Wireshark being a packet sniffer. It'll collect the packets being sent
>> across the link to the hub. You'd have to use a hub because that's the
>> only way to listen to ALL packets on the wire. A switch (generally) will
>> only pass packets destined for that port. You generally don't use hubs
>> in regular use because of how they share all traffic. But they're darned
>> handy for this sort of monitoring.
>>
>> For this monitoring to work without changing anything on your network you
>> HAVE TO USE A HUB. You CANNOT use a switch (at least not unless it has
>> more advanced built-in controls for allowing this sort of thing). To do
>> it with a PC would require two network interfaces on it and some changes
>> to your network. Neither of which is anywhere as easy to setup as a hub.
>
> Bill,
>
> I've used this method using a utility called NetBoy, as well as NetMON
> sniffing traffic at that location I previously mentioned, as well as
> others. That was before I got the approval for Packeteer, because it just
> wasn't enough. I mean I would see one user spike (Netboy shows it in
> realtime), then another would spike, but I couldn't exactly tell if it was
> legit or not. It was alot of work, and I was charging them by the hour.
> That was one of the reasons they approved the appliance, they didn't want
> to pay for the time I was spending on this issue. Oh well... I didn't
> mind. :-)
>
> Ace
>

That was my concern also; looking at packets and not knowing for sure what
was bad and good, even if one specific host was generating more traffic than
another. This would also require my constant attention to get the job done
which I just don't thave the time (or the immediately avail resources) to
work with such a method.