Ventrilo Severely Degrading Network Performance - Page 2

NeoLojik · 07-30-2009, 06:51 AM

The server is the free 8 client version and uses the standard ventrilo port, which is 3784 I believe?

mjgraf · 07-31-2009, 01:52 PM

are you connecting to a locally hosted server?
more detail would be helpful.

NeoLojik · 07-31-2009, 03:00 PM

The Ventrilo server is not on the local LAN, it's running on a remote machine in Newark, NJ, USA. It's a Linux machine, running version 3.0.3 of the Ventrilo server.

Internet access is provided by a Netgear DG834GT router, but that's connected to the network via the gigabit switch, local network traffic doesn't go through it. The actual internet line is 8mbps ADSL, not that I think that matters.

I'm not sure what other details I can give that are relevant. The iperf results show the issue. Copying files to and from the gigabit enabled PC's on the LAN usually results in 65-70MB/sec of throughput, but it's considerably reduced when the Ventrilo client is running and connected.

It seems unlikely that something else on the system is causing the issue as it happens only when Ventrilo is open and connected. It's 100% reproducible.

I've also tried disabling both QoS and IPv6 with no change.

Prog-Rocker · 08-06-2009, 09:57 PM

finally got all the hardware configured.

Here's the results:
1st run is without vent running
2nd run is with vent connected to external server on both systems

C:\Xfer>iperf -c 192.168.0.90 -w 128k -P 2 -t 30
------------------------------------------------------------
Client connecting to 192.168.0.90, TCP port 5001
TCP window size: 128 KByte
------------------------------------------------------------
[1896] local 192.168.0.15 port 1289 connected with 192.168.0.90 port 5001
[1912] local 192.168.0.15 port 1288 connected with 192.168.0.90 port 5001
[ ID] Interval Transfer Bandwidth
[1896] 0.0-30.2 sec 1.65 GBytes 469 Mbits/sec
[1912] 0.0-30.2 sec 1.56 GBytes 445 Mbits/sec
[SUM] 0.0-30.2 sec 3.21 GBytes 914 Mbits/sec
C:\Xfer>iperf -c 192.168.0.90 -w 128k -P 2 -t 30
------------------------------------------------------------
Client connecting to 192.168.0.90, TCP port 5001
TCP window size: 128 KByte
------------------------------------------------------------
[1896] local 192.168.0.15 port 1295 connected with 192.168.0.90 port 5001
[1912] local 192.168.0.15 port 1294 connected with 192.168.0.90 port 5001
[ ID] Interval Transfer Bandwidth
[1896] 0.0-30.2 sec 1.61 GBytes 458 Mbits/sec
[1912] 0.0-30.4 sec 1.56 GBytes 441 Mbits/sec
[SUM] 0.0-30.4 sec 3.17 GBytes 896 Mbits/sec

both systems running Vent v305
Netgear GS108 Gigabit switch
server pc - Windows 7 Ultimate 32bit, nVidia nForce adapter, auto negotiate, microsoft driver 2007
client pc - Windows XP SP2 32bit, Marvell Yukon 88E8053 adapter, forced 1000 Mbit full duplex, Marvell driver 2009
no firewalls/AV on either system
same results on 3 more test runs

NeoLojik · 08-07-2009, 09:35 PM

Well, I'm officially stumped at what the cause could be.

Kerrisis has tried using the on-board Marvell NIC and has the same issue, so that rules out it being a Realtek specific problem. He also suspected that maybe it could be the gigabyte switch we both use, but Prog-Rocker uses the same, bar his being 8 port and both of ours only being 5. So it seems unlikely the problem is with that.

I've run ventrilo.exe through process monitor and checked every single item reported where it touched system files or registry entries, and again turned up nothing.

I did discover that the drop in speed happens on transfers already in progress. I started a 2GB file copying to my machine and it managed 70MB/sec, the second I clicked connect in Ventrilo the speed plummeted. As soon as I disconnected, it shot back up again.

I've even gone as far as to connect to other public Ventrilo servers, both free (8 client limit) and commercial, with no change.

My last attempt was to run a ventrilo server on the same machine as the client and connect to localhost, ruling out Ventrilo traffic affecting the external network entirely.

Again, throughput dived the second the client was connected.

I'm now at a complete loss, and from the silence on this thread, I'm assuming the people in charge are too. Short of debugging this one line of code at a time (I'm not that skilled in programming, nor is the source available), there's nothing more I can do.

Prog-Rocker · 08-07-2009, 09:55 PM

don't give up yet.

there must be something we're missing in a config somewhere.

unfortunately, i will be away from the house for 2 weeks but i have this thread flagged and will get back to it when i get back.

i will try to adjust some settings to see if i can duplicate this.

also, maybe Graf might come up with an idea or 2 since he duplicated it somewhat.

NeoLojik · 08-10-2009, 04:53 AM

Ok, I've finally solved it, and it's not a Ventrilo issue, but rather a Vista-only one.

The problem:
With the release of Vista, Microsoft added a system service to monitor the multimedia subsystem for streaming audio and video, called the Multimedia Class Scheduler (MMC). When this service detects streaming media (and a lot of applications trigger this), it throttles network traffic to ensure smooth network playback.

Unfortunately, this was conceived in the days where single-core CPU's and 100MBit LAN's were common, and their solution drastically affects gigabit speeds.

I'll explain why this problem isn't far more widespread in a moment.

MMC is nothing more than a packet limiter. When certain applications run, MMC puts a hard-cap on the number of packets which can pass over the network interface per second.

At a standard TCP/IP frame size, that's ~1,500 bytes x 10,000 packets (the limit imposed by MMC), which after overhead works out at about 11-12MB/sec.

From SP1 in Vista, you can tweak this limit via the registry. Initially I tried setting it to 30 (for 30,000 packets), but only managed half network speed. Upping it to 60 or 70 made no difference.

Instead I set it to FFFFFFFF (Hexidecimal) or 4294967295 (Decimal) which effectively sets the cap at such a high limit that you'll never hit it.

Network transfer speeds are now the same when either connected, or disconnected on Ventrilo, with no degradation to Ventrilo quality (what's 4-5KB/sec when the interface can do 65-75MB/sec =P ).

Jumbo frames:
Before I post the how-to for changing the registry, here's the reason why some people haven't noticed it.

As the MMC limit is a *packet* limit, those on a gigabit network with jumbo-frames are far less likely to see the issue. Jumbo frames increase the TCP frame size from ~1,500 to a maximum of 9,000 bytes. As you can imagine, 10,000 frames at 9,000 bytes per frame is roughly 80MB/sec after overhead.

My machines here don't enable jumbo frames by default, but I'm willing to bet that would also solve the issue.

The solution:
So, here's how to tweak the limit yourself. Perhaps this should be added to the Ventrilo FAQ?

1) Open Regedit.exe
2) Navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile
3) In the right pane, double click NetworkThrottlingIndex
4) Set the value to FFFFFFFF, Hexadecimal
5) Close Regedit
6) Reboot

Once the machine is rebooted, the limit will be removed. To undo the change, follow the instructions above, but set the value to: a (Hexadecimal), or 10 (Decimal). Reboot again and the limit will be put back.

This issue doesn't exist in Windows XP as the service doesn't exist. Apparently Windows 7 has corrected this particular flaw, so it's a Vista -only- issue.

Hopefully this will help someone who has been suffering the same problem. It's nice to finally have it solved.

Prog-Rocker · 08-10-2009, 11:26 AM

Very Nice investigative work. That explains why I couldn't duplicate it as I wasn't using Vista.

I can't check right now but I think the Jumbo Frames settings in the advanced network card properties was set the the default of 1500.

NeoLojik · 08-11-2009, 07:15 AM

Just a little update.

Setting NetworkThrottlingIndex to FFFFFFFF actually disables the limiter entirely, but regardless, I've found no negative effects when it's disabled. I can still happily stream HD video over my LAN whilst copying files.

I'm yet to try enabling jumbo frames, but as they're disabled by default and this solution works, I'm going to leave things as they are.

Thanks to everyone who contributed data and ideas. It's great to finally have this solved.

mjgraf · 09-01-2009, 03:56 PM

very interesting work. thanks for all your effort and information!!

07-30-2009, 06:51 AM	#11
NeoLojik Junior Member Join Date: Jul 2009 Posts: 8	The server is the free 8 client version and uses the standard ventrilo port, which is 3784 I believe? Last edited by NeoLojik; 07-30-2009 at 06:57 AM.

08-06-2009, 09:57 PM	#14
Prog-Rocker just tryin to help Join Date: Jul 2006 Location: Local Space/Time Continuum Posts: 23,460	finally got all the hardware configured. Here's the results: 1st run is without vent running 2nd run is with vent connected to external server on both systems C:\Xfer>iperf -c 192.168.0.90 -w 128k -P 2 -t 30 ------------------------------------------------------------ Client connecting to 192.168.0.90, TCP port 5001 TCP window size: 128 KByte ------------------------------------------------------------ [1896] local 192.168.0.15 port 1289 connected with 192.168.0.90 port 5001 [1912] local 192.168.0.15 port 1288 connected with 192.168.0.90 port 5001 [ ID] Interval Transfer Bandwidth [1896] 0.0-30.2 sec 1.65 GBytes 469 Mbits/sec [1912] 0.0-30.2 sec 1.56 GBytes 445 Mbits/sec [SUM] 0.0-30.2 sec 3.21 GBytes 914 Mbits/sec C:\Xfer>iperf -c 192.168.0.90 -w 128k -P 2 -t 30 ------------------------------------------------------------ Client connecting to 192.168.0.90, TCP port 5001 TCP window size: 128 KByte ------------------------------------------------------------ [1896] local 192.168.0.15 port 1295 connected with 192.168.0.90 port 5001 [1912] local 192.168.0.15 port 1294 connected with 192.168.0.90 port 5001 [ ID] Interval Transfer Bandwidth [1896] 0.0-30.2 sec 1.61 GBytes 458 Mbits/sec [1912] 0.0-30.4 sec 1.56 GBytes 441 Mbits/sec [SUM] 0.0-30.4 sec 3.17 GBytes 896 Mbits/sec both systems running Vent v305 Netgear GS108 Gigabit switch server pc - Windows 7 Ultimate 32bit, nVidia nForce adapter, auto negotiate, microsoft driver 2007 client pc - Windows XP SP2 32bit, Marvell Yukon 88E8053 adapter, forced 1000 Mbit full duplex, Marvell driver 2009 no firewalls/AV on either system same results on 3 more test runs Last edited by Prog-Rocker; 08-06-2009 at 10:10 PM.

08-07-2009, 09:55 PM	#16
Prog-Rocker just tryin to help Join Date: Jul 2006 Location: Local Space/Time Continuum Posts: 23,460	don't give up yet. there must be something we're missing in a config somewhere. unfortunately, i will be away from the house for 2 weeks but i have this thread flagged and will get back to it when i get back. i will try to adjust some settings to see if i can duplicate this. also, maybe Graf might come up with an idea or 2 since he duplicated it somewhat. Last edited by Prog-Rocker; 08-07-2009 at 09:58 PM.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

07-31-2009, 01:52 PM	#12
mjgraf Senior Member Join Date: May 2005 Location: Some say in my own little world Posts: 15,502	are you connecting to a locally hosted server? more detail would be helpful.

07-31-2009, 03:00 PM	#13
NeoLojik Junior Member Join Date: Jul 2009 Posts: 8	The Ventrilo server is not on the local LAN, it's running on a remote machine in Newark, NJ, USA. It's a Linux machine, running version 3.0.3 of the Ventrilo server. Internet access is provided by a Netgear DG834GT router, but that's connected to the network via the gigabit switch, local network traffic doesn't go through it. The actual internet line is 8mbps ADSL, not that I think that matters. I'm not sure what other details I can give that are relevant. The iperf results show the issue. Copying files to and from the gigabit enabled PC's on the LAN usually results in 65-70MB/sec of throughput, but it's considerably reduced when the Ventrilo client is running and connected. It seems unlikely that something else on the system is causing the issue as it happens only when Ventrilo is open and connected. It's 100% reproducible. I've also tried disabling both QoS and IPv6 with no change.

08-07-2009, 09:35 PM	#15
NeoLojik Junior Member Join Date: Jul 2009 Posts: 8	Well, I'm officially stumped at what the cause could be. Kerrisis has tried using the on-board Marvell NIC and has the same issue, so that rules out it being a Realtek specific problem. He also suspected that maybe it could be the gigabyte switch we both use, but Prog-Rocker uses the same, bar his being 8 port and both of ours only being 5. So it seems unlikely the problem is with that. I've run ventrilo.exe through process monitor and checked every single item reported where it touched system files or registry entries, and again turned up nothing. I did discover that the drop in speed happens on transfers already in progress. I started a 2GB file copying to my machine and it managed 70MB/sec, the second I clicked connect in Ventrilo the speed plummeted. As soon as I disconnected, it shot back up again. I've even gone as far as to connect to other public Ventrilo servers, both free (8 client limit) and commercial, with no change. My last attempt was to run a ventrilo server on the same machine as the client and connect to localhost, ruling out Ventrilo traffic affecting the external network entirely. Again, throughput dived the second the client was connected. I'm now at a complete loss, and from the silence on this thread, I'm assuming the people in charge are too. Short of debugging this one line of code at a time (I'm not that skilled in programming, nor is the source available), there's nothing more I can do.

08-10-2009, 04:53 AM	#17
NeoLojik Junior Member Join Date: Jul 2009 Posts: 8	Ok, I've finally solved it, and it's not a Ventrilo issue, but rather a Vista-only one. The problem: With the release of Vista, Microsoft added a system service to monitor the multimedia subsystem for streaming audio and video, called the Multimedia Class Scheduler (MMC). When this service detects streaming media (and a lot of applications trigger this), it throttles network traffic to ensure smooth network playback. Unfortunately, this was conceived in the days where single-core CPU's and 100MBit LAN's were common, and their solution drastically affects gigabit speeds. I'll explain why this problem isn't far more widespread in a moment. MMC is nothing more than a packet limiter. When certain applications run, MMC puts a hard-cap on the number of packets which can pass over the network interface per second. At a standard TCP/IP frame size, that's ~1,500 bytes x 10,000 packets (the limit imposed by MMC), which after overhead works out at about 11-12MB/sec. From SP1 in Vista, you can tweak this limit via the registry. Initially I tried setting it to 30 (for 30,000 packets), but only managed half network speed. Upping it to 60 or 70 made no difference. Instead I set it to FFFFFFFF (Hexidecimal) or 4294967295 (Decimal) which effectively sets the cap at such a high limit that you'll never hit it. Network transfer speeds are now the same when either connected, or disconnected on Ventrilo, with no degradation to Ventrilo quality (what's 4-5KB/sec when the interface can do 65-75MB/sec =P ). Jumbo frames: Before I post the how-to for changing the registry, here's the reason why some people haven't noticed it. As the MMC limit is a packet limit, those on a gigabit network with jumbo-frames are far less likely to see the issue. Jumbo frames increase the TCP frame size from ~1,500 to a maximum of 9,000 bytes. As you can imagine, 10,000 frames at 9,000 bytes per frame is roughly 80MB/sec after overhead. My machines here don't enable jumbo frames by default, but I'm willing to bet that would also solve the issue. The solution: So, here's how to tweak the limit yourself. Perhaps this should be added to the Ventrilo FAQ? 1) Open Regedit.exe 2) Navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile 3) In the right pane, double click NetworkThrottlingIndex 4) Set the value to FFFFFFFF, Hexadecimal 5) Close Regedit 6) Reboot Once the machine is rebooted, the limit will be removed. To undo the change, follow the instructions above, but set the value to: a (Hexadecimal), or 10 (Decimal). Reboot again and the limit will be put back. This issue doesn't exist in Windows XP as the service doesn't exist. Apparently Windows 7 has corrected this particular flaw, so it's a Vista -only- issue. Hopefully this will help someone who has been suffering the same problem. It's nice to finally have it solved.

08-10-2009, 11:26 AM	#18
Prog-Rocker just tryin to help Join Date: Jul 2006 Location: Local Space/Time Continuum Posts: 23,460	Very Nice investigative work. That explains why I couldn't duplicate it as I wasn't using Vista. I can't check right now but I think the Jumbo Frames settings in the advanced network card properties was set the the default of 1500.

08-11-2009, 07:15 AM	#19
NeoLojik Junior Member Join Date: Jul 2009 Posts: 8	Just a little update. Setting NetworkThrottlingIndex to FFFFFFFF actually disables the limiter entirely, but regardless, I've found no negative effects when it's disabled. I can still happily stream HD video over my LAN whilst copying files. I'm yet to try enabling jumbo frames, but as they're disabled by default and this solution works, I'm going to leave things as they are. Thanks to everyone who contributed data and ideas. It's great to finally have this solved.

09-01-2009, 03:56 PM	#20
mjgraf Senior Member Join Date: May 2005 Location: Some say in my own little world Posts: 15,502	very interesting work. thanks for all your effort and information!!