Ventrilo Tech Support

Ventrilo Tech Support (http://forum.ventrilo.com/index.php)
-   Windows Client (http://forum.ventrilo.com/forumdisplay.php?f=4)
-   -   Ventrilo Severely Degrading Network Performance (http://forum.ventrilo.com/showthread.php?t=38194)

NeoLojik 07-23-2009 04:15 PM

Ventrilo Severely Degrading Network Performance
 
Update: Solved. See http://www.ventrilo.com/forums/showp...7&postcount=17

Good evening,

I've discovered that when the Ventrilo client is running and connected to a server, the machine it's running on has it's ethernet receive throughput cut by up to 85%.

I have 2 PC's on my desk here, let's call them PC1 and PC2.

PC1 runs Windows Vista 64bit, has all the latest drivers and updates from Microsoft. PC1 also runs the Ventrilo client (v3.0.5).

PC2 runs Windows XP 32bit. Again all drivers are the latest and it has all updates available.

Using iperf, a command-line network performance utility, I'm able to measure throughput both before and after launching Ventrilo. Showing the issue.

PC1 is acting as the iperf server, PC2 is the client. They are connected via a gigabit switch.

Without Ventrilo running:
Code:

C:\Users\Nexx\Desktop>iperf -s -w 128k
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte
------------------------------------------------------------
[160] local 192.168.0.20 port 5001 connected with 192.168.0.21 port 3134
[188] local 192.168.0.20 port 5001 connected with 192.168.0.21 port 3135
[ ID] Interval      Transfer    Bandwidth
[188]  0.0-30.0 sec  1.65 GBytes  472 Mbits/sec
[160]  0.0-30.0 sec  1.65 GBytes  472 Mbits/sec
[SUM]  0.0-30.0 sec  3.29 GBytes  943 Mbits/sec

With Ventrilo running:
Code:

[164] local 192.168.0.20 port 5001 connected with 192.168.0.21 port 3136
[204] local 192.168.0.20 port 5001 connected with 192.168.0.21 port 3137
[ ID] Interval      Transfer    Bandwidth
[204]  0.0-30.0 sec  209 MBytes  58.4 Mbits/sec
[164]  0.0-30.0 sec  209 MBytes  58.4 Mbits/sec
[SUM]  0.0-30.0 sec  418 MBytes  117 Mbits/sec

Without Ventrilo running, or running but not connected, the actual throughput fluctuates by a couple of megabits each way. When Ventrilo is running the result is always -exactly- the same.

The issue is system-wide, it's not limited to iperf. When copying a file to PC1 the throughput drops from 65-70MB/sec to barely 12-13MB/sec when Ventrilo is running and connected.

I've also confirmed the issue with a friend of mine, who has spent weeks thinking there was something wrong with his gigabit network. He sees the exact same issue when Ventrilo is running and connected.

I have to wonder what on earth Ventrilo is doing to the networking stack to cause an 85-90% reduction in Rx traffic.

Hopefully someone from flagship will see this and look into the matter.

Prog-Rocker 07-23-2009 04:49 PM

where did you get iperf?

NeoLojik 07-23-2009 05:16 PM

http://code.google.com/p/xjperf/

Download the zip, iperf.exe is in the bin directory.

Invoke the server like so:
iperf -s -w 128k

Invoke the client like so:
iperf -c SERVER_IP -w 128k -P 2 -t 30

It'll take 30 seconds then spit out the results. You're interested in [SUM] which is the combined total of both threads.

Prog-Rocker 07-23-2009 06:39 PM

here's my results and network config:
note - I do adjust some lan card properties. disable QoS and change some of the advanced settings for the card.

pc1 - vista 64bit SP2 all updates installed

pc2 - windows XP SP3 all updates installed

linksys 100Mb switch

first run is without vent running on either system

second run is with vent running on both, ver 305 and connected to an external server.

C:\Xfer>iperf -c 192.168.0.80 -w 128k -P 2 -t 30
------------------------------------------------------------
Client connecting to 192.168.0.80, TCP port 5001
TCP window size: 128 KByte
------------------------------------------------------------
[1812] local 192.168.0.50 port 1455 connected with 192.168.0.80 port 5001
[1800] local 192.168.0.50 port 1456 connected with 192.168.0.80 port 5001
[ ID] Interval Transfer Bandwidth
[1812] 0.0-30.0 sec 169 MBytes 47.2 Mbits/sec
[1800] 0.0-30.0 sec 169 MBytes 47.2 Mbits/sec
[SUM] 0.0-30.0 sec 338 MBytes 94.4 Mbits/sec


C:\Xfer>iperf -c 192.168.0.80 -w 128k -P 2 -t 30
------------------------------------------------------------
Client connecting to 192.168.0.80, TCP port 5001
TCP window size: 128 KByte
------------------------------------------------------------
[1800] local 192.168.0.50 port 1463 connected with 192.168.0.80 port 5001
[1812] local 192.168.0.50 port 1462 connected with 192.168.0.80 port 5001
[ ID] Interval Transfer Bandwidth
[1812] 0.0-30.0 sec 169 MBytes 47.3 Mbits/sec
[1800] 0.0-30.0 sec 169 MBytes 47.2 Mbits/sec
[SUM] 0.0-30.0 sec 338 MBytes 94.4 Mbits/sec

NeoLojik 07-23-2009 07:56 PM

I had wondered if QoS was causing the issue, but I just disabled it and it made no difference. I'm also wondering if this issue only occurs on gigabit ethernet, as the reduced speed is still above 100Mbit.

Like I said, I've encountered this here, and also with a friend, we both have gigabit LANs and both have the problem of massively reduced speed (85%+ reduction) when Ventrilo is running and connected.

Closing Ventrilo resolves the problem.

Prog-Rocker 07-23-2009 11:13 PM

i'll see if i can get hold of a Gb switch and rerun the test.

mjgraf 07-25-2009 05:23 AM

I have GigE and with the ventrilo client running I get the following using your commandline options.
I have a similar setup, server is running XP32bit, client I used is running Vista 32bit.

Connecting locally on the server, same with and without ventrilo running/connected
Code:

[1868] local 192.168.2.105 port 5001 connected with 192.168.2.105 port 3307
[1832] local 192.168.2.105 port 5001 connected with 192.168.2.105 port 3308
[ ID] Interval      Transfer    Bandwidth
[1868]  0.0-30.0 sec  1.45 GBytes  416 Mbits/sec
[1832]  0.0-30.0 sec  1.39 GBytes  398 Mbits/sec
[SUM]  0.0-30.0 sec  2.85 GBytes  814 Mbits/sec

Connecting on GigE internal lan
Without Ventrilo running
Code:

[1812] local 192.168.2.105 port 5001 connected with 192.168.2.108 port 49227
[1820] local 192.168.2.105 port 5001 connected with 192.168.2.108 port 49228
[ ID] Interval      Transfer    Bandwidth
[1820]  0.0-30.0 sec  505 MBytes  141 Mbits/sec
[1812]  0.0-30.3 sec  490 MBytes  136 Mbits/sec
[SUM]  0.0-30.3 sec  994 MBytes  275 Mbits/sec

With Ventrilo running on the server, client connecting from the other computer
Code:

[1848] local 192.168.2.105 port 5001 connected with 192.168.2.108 port 49243
[1832] local 192.168.2.105 port 5001 connected with 192.168.2.108 port 49244
[ ID] Interval      Transfer    Bandwidth
[1848]  0.0-30.3 sec  521 MBytes  144 Mbits/sec
[1832]  0.0-30.0 sec  469 MBytes  131 Mbits/sec
[SUM]  0.0-30.3 sec  989 MBytes  274 Mbits/sec

seems I need to tweak that laptop!!

Kerrisis 07-26-2009 07:17 AM

I'm the friend NeoLojik mentioned as also having the same problem. I run Ventrilo 3.0.4 (64bit) on Windows Vista Ultimate 64-bit. I've performed the same tests as him, using Iperf for Windows v1.70. The destination PC is on the same switch, and is running Windows Vista Business 32bit. This is what happens when I run Iperf with Vent running:

Code:

[116] local 192.168.0.3 port 49787 connected with 192.168.0.2 port 5001
[108] local 192.168.0.3 port 49786 connected with 192.168.0.2 port 5001
[ ID] Interval      Transfer    Bandwidth
[116]  0.0-30.0 sec  204 MBytes  56.9 Mbits/sec
[108]  0.0-30.0 sec  206 MBytes  57.6 Mbits/sec
[SUM]  0.0-30.0 sec  410 MBytes  115 Mbits/sec

And this is without Vent running:
Code:

[116] local 192.168.0.3 port 49794 connected with 192.168.0.2 port 5001
[108] local 192.168.0.3 port 49793 connected with 192.168.0.2 port 5001
[ ID] Interval      Transfer    Bandwidth
[108]  0.0-30.0 sec  1.14 GBytes  325 Mbits/sec
[116]  0.0-30.0 sec  1.10 GBytes  314 Mbits/sec
[SUM]  0.0-30.0 sec  2.23 GBytes  639 Mbits/sec

As you can see, it's a pretty significant performance hit. Bear in mind that this is only a *one-way* performance hit, too; if I rerun the tests in the other direction, it all works fine. This implies to me that Ventrilo is somehow affecting the RX (IE Download) bandwidth of the machine it's running on...

Further information if it's needed.
- The PC running Vent has an Asus P5K Premium Black Pearl edition mobo with an onboard Realtek RTL8110SC PCI Gigabit LAN controller. The drivers are the latest I could find, and are dated 25th May 2009. The problem occured with earlier driver versions.
- The destination PC has an Asus P5Q-EM mobo, running a Realtek 8111C PCI-E Gigabit LAN controller, drivers dated 3rd July 2009. Again, updating drivers update did not affect the problem.
- Both systems are connected to a a Netgear GS605 v2 gigabit-capable switch via known-working moulded-connector Cat5e.
- Both PCs show as connecting to the network at 1.0Gbit, and are perfectly stable at that speed.
- The RX and TX on both systems will allow for ~75-80MB/sec transfers (SATA II bottlenecks permitting) in both directions under normal operations.When the problem occurs, RX is limited to 14MB/sec.

So, does anyone have any ideas what NeoLojik and I could to to narrow down or solve this? Any help would be greatfully received.


Edit: I've updated to Ventrilo 3.0.5 (64bit), with no change to the problem.

Kerrisis 07-27-2009 06:11 PM

Apologies if this is seen as rude on these forums, but consider this a thread bump, in the hope that someone may be able to help.

mjgraf 07-29-2009 02:36 PM

what is the port number for the ventrilo server you are connecting to when you test this?

NeoLojik 07-30-2009 07:51 AM

The server is the free 8 client version and uses the standard ventrilo port, which is 3784 I believe?

mjgraf 07-31-2009 02:52 PM

are you connecting to a locally hosted server?
more detail would be helpful.

NeoLojik 07-31-2009 04:00 PM

The Ventrilo server is not on the local LAN, it's running on a remote machine in Newark, NJ, USA. It's a Linux machine, running version 3.0.3 of the Ventrilo server.

Internet access is provided by a Netgear DG834GT router, but that's connected to the network via the gigabit switch, local network traffic doesn't go through it. The actual internet line is 8mbps ADSL, not that I think that matters.

I'm not sure what other details I can give that are relevant. The iperf results show the issue. Copying files to and from the gigabit enabled PC's on the LAN usually results in 65-70MB/sec of throughput, but it's considerably reduced when the Ventrilo client is running and connected.

It seems unlikely that something else on the system is causing the issue as it happens only when Ventrilo is open and connected. It's 100% reproducible.

I've also tried disabling both QoS and IPv6 with no change.

Prog-Rocker 08-06-2009 10:57 PM

finally got all the hardware configured.

Here's the results:
1st run is without vent running
2nd run is with vent connected to external server on both systems

C:\Xfer>iperf -c 192.168.0.90 -w 128k -P 2 -t 30
------------------------------------------------------------
Client connecting to 192.168.0.90, TCP port 5001
TCP window size: 128 KByte
------------------------------------------------------------
[1896] local 192.168.0.15 port 1289 connected with 192.168.0.90 port 5001
[1912] local 192.168.0.15 port 1288 connected with 192.168.0.90 port 5001
[ ID] Interval Transfer Bandwidth
[1896] 0.0-30.2 sec 1.65 GBytes 469 Mbits/sec
[1912] 0.0-30.2 sec 1.56 GBytes 445 Mbits/sec
[SUM] 0.0-30.2 sec 3.21 GBytes 914 Mbits/sec
C:\Xfer>iperf -c 192.168.0.90 -w 128k -P 2 -t 30
------------------------------------------------------------
Client connecting to 192.168.0.90, TCP port 5001
TCP window size: 128 KByte
------------------------------------------------------------
[1896] local 192.168.0.15 port 1295 connected with 192.168.0.90 port 5001
[1912] local 192.168.0.15 port 1294 connected with 192.168.0.90 port 5001
[ ID] Interval Transfer Bandwidth
[1896] 0.0-30.2 sec 1.61 GBytes 458 Mbits/sec
[1912] 0.0-30.4 sec 1.56 GBytes 441 Mbits/sec
[SUM] 0.0-30.4 sec 3.17 GBytes 896 Mbits/sec

both systems running Vent v305
Netgear GS108 Gigabit switch
server pc - Windows 7 Ultimate 32bit, nVidia nForce adapter, auto negotiate, microsoft driver 2007
client pc - Windows XP SP2 32bit, Marvell Yukon 88E8053 adapter, forced 1000 Mbit full duplex, Marvell driver 2009
no firewalls/AV on either system
same results on 3 more test runs

NeoLojik 08-07-2009 10:35 PM

Well, I'm officially stumped at what the cause could be.

Kerrisis has tried using the on-board Marvell NIC and has the same issue, so that rules out it being a Realtek specific problem. He also suspected that maybe it could be the gigabyte switch we both use, but Prog-Rocker uses the same, bar his being 8 port and both of ours only being 5. So it seems unlikely the problem is with that.

I've run ventrilo.exe through process monitor and checked every single item reported where it touched system files or registry entries, and again turned up nothing.

I did discover that the drop in speed happens on transfers already in progress. I started a 2GB file copying to my machine and it managed 70MB/sec, the second I clicked connect in Ventrilo the speed plummeted. As soon as I disconnected, it shot back up again.

I've even gone as far as to connect to other public Ventrilo servers, both free (8 client limit) and commercial, with no change.

My last attempt was to run a ventrilo server on the same machine as the client and connect to localhost, ruling out Ventrilo traffic affecting the external network entirely.

Again, throughput dived the second the client was connected.

I'm now at a complete loss, and from the silence on this thread, I'm assuming the people in charge are too. Short of debugging this one line of code at a time (I'm not that skilled in programming, nor is the source available), there's nothing more I can do.

Prog-Rocker 08-07-2009 10:55 PM

don't give up yet.

there must be something we're missing in a config somewhere.

unfortunately, i will be away from the house for 2 weeks but i have this thread flagged and will get back to it when i get back.

i will try to adjust some settings to see if i can duplicate this.

also, maybe Graf might come up with an idea or 2 since he duplicated it somewhat.

NeoLojik 08-10-2009 05:53 AM

Ok, I've finally solved it, and it's not a Ventrilo issue, but rather a Vista-only one.

The problem:
With the release of Vista, Microsoft added a system service to monitor the multimedia subsystem for streaming audio and video, called the Multimedia Class Scheduler (MMC). When this service detects streaming media (and a lot of applications trigger this), it throttles network traffic to ensure smooth network playback.

Unfortunately, this was conceived in the days where single-core CPU's and 100MBit LAN's were common, and their solution drastically affects gigabit speeds.

I'll explain why this problem isn't far more widespread in a moment.

MMC is nothing more than a packet limiter. When certain applications run, MMC puts a hard-cap on the number of packets which can pass over the network interface per second.

At a standard TCP/IP frame size, that's ~1,500 bytes x 10,000 packets (the limit imposed by MMC), which after overhead works out at about 11-12MB/sec.

From SP1 in Vista, you can tweak this limit via the registry. Initially I tried setting it to 30 (for 30,000 packets), but only managed half network speed. Upping it to 60 or 70 made no difference.

Instead I set it to FFFFFFFF (Hexidecimal) or 4294967295 (Decimal) which effectively sets the cap at such a high limit that you'll never hit it.

Network transfer speeds are now the same when either connected, or disconnected on Ventrilo, with no degradation to Ventrilo quality (what's 4-5KB/sec when the interface can do 65-75MB/sec =P ).

Jumbo frames:
Before I post the how-to for changing the registry, here's the reason why some people haven't noticed it.

As the MMC limit is a *packet* limit, those on a gigabit network with jumbo-frames are far less likely to see the issue. Jumbo frames increase the TCP frame size from ~1,500 to a maximum of 9,000 bytes. As you can imagine, 10,000 frames at 9,000 bytes per frame is roughly 80MB/sec after overhead.

My machines here don't enable jumbo frames by default, but I'm willing to bet that would also solve the issue.

The solution:
So, here's how to tweak the limit yourself. Perhaps this should be added to the Ventrilo FAQ?

1) Open Regedit.exe
2) Navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile
3) In the right pane, double click NetworkThrottlingIndex
4) Set the value to FFFFFFFF, Hexadecimal
5) Close Regedit
6) Reboot

Once the machine is rebooted, the limit will be removed. To undo the change, follow the instructions above, but set the value to: a (Hexadecimal), or 10 (Decimal). Reboot again and the limit will be put back.

This issue doesn't exist in Windows XP as the service doesn't exist. Apparently Windows 7 has corrected this particular flaw, so it's a Vista -only- issue.

Hopefully this will help someone who has been suffering the same problem. It's nice to finally have it solved.

Prog-Rocker 08-10-2009 12:26 PM

Very Nice investigative work. That explains why I couldn't duplicate it as I wasn't using Vista.

I can't check right now but I think the Jumbo Frames settings in the advanced network card properties was set the the default of 1500.

NeoLojik 08-11-2009 08:15 AM

Just a little update.

Setting NetworkThrottlingIndex to FFFFFFFF actually disables the limiter entirely, but regardless, I've found no negative effects when it's disabled. I can still happily stream HD video over my LAN whilst copying files.

I'm yet to try enabling jumbo frames, but as they're disabled by default and this solution works, I'm going to leave things as they are.

Thanks to everyone who contributed data and ideas. It's great to finally have this solved.

mjgraf 09-01-2009 04:56 PM

very interesting work. thanks for all your effort and information!!

MattTHM 11-07-2009 04:23 PM

If this is only happening on Vista with Gb ethernet, it might be related to this problem: http://blogs.technet.com/markrussino...7/1833290.aspx

Apparently Vista drops network throughput to about 15% when media/sound functions are used. The link contains all the (seriously in-depth) info on it.

MattTHM 11-07-2009 04:24 PM

D'oh, didn't see page 2, it was already answered!


All times are GMT -5. The time now is 06:08 AM.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2017, vBulletin Solutions, Inc.