Redhat 9.0 and Gigabit woes...

[Godlike567]

I just finished installing Redhat 9 on a new system designed specifically for LAN file storage. I have about 600 gigabytes of data to backup and im expierencing preformance issues transfering the data from XP to Linux.

For example I logged into the linux box via ftp and was only getting 6 - 8 mbp/s from both directions. After playing with the MTU a bit on both systems I found that 9000 gave these results. Now its transfering at

XP >> Linux 48.08 Mb/s
XP << Linux 11.01 Mb/s

Both system have identical hardware. Only the OS is different. Is there anyway to find the "perfect" MTU and TCP window sizes for this setup? Also how do I change TCP send/recieve window in Redhat 9? I use drtcp in XP and it worked.

The systems are connected via CAT5E crossover cable (thats about 20 ft. long) there are no problems with the cable as I have tried several different brands and lengths with the same results.

My knowledge of linux is very limited and I would appreciate any insight.


Thanks in Advance.



__________________
RedHat Linux 9.0
TYAN S2466N MPX
Dual 2400 Athlon MP's
AGP Voodoo Banshee
512 DDR Unbuffered
Highpoint 404 Raid Controller
600 gigabyte Raid 0 array
Intel Pro 1000 MT
Onboard 3c905c 100baseT
No CD or Floppy

[Magueta]

Could it have something to do with your ethernet cards? I see you have 100 Mbps on your machine and I presume it's the same on the other one because they both have the same hardware, but are they setup for 100 Mbps? Are they full duplex? Does the information go through a hub? A hub will reduce your bandwidth in direct proportion to the number of connections. If it goes through a hub, router or switch you should check that they are functioning at 100 Mbps. I don't know if these are the source of your problem but they're worth checking.

Joe

[Godlike567]

Thanks for the response Joe. I double check the link speeds on all the NIC's including the 3coms. Everything seems in perfect order. I just drew up a network Diagram to explain how my setup is. I appreciate any feedback.


http://www.godlike.org/network/mynetwork.gif

[Magueta]

The diagram isn't bad, it seems clear enough. When you ftp back and forth how do you know when you're using the crossover cable and when you're using the cable through the switch? I guess it's the different subnets. What are your ping times like through the crossover cable and through the switch? I don't know why it caps off at 45 Mbps but the 10 Mbps seems to suggest that the limitation is the switch. I can't tell you if there's some problem between XP and Linux on a network but I can tell you that I get very close to 100 Mbps on my own network with Win2K and RH Linux 9. In fact I can burn CDs from my mounted Samba shares.
Once we've eliminated hardware problems all that's left is the software. What else could it be? I know that you checked the NICs but could it be that one of them is malfunctioning and causing a broadcast storm or at least transmitting extra packets? Do you have the same problem on both subnets? A long shot is that the drives are your bottleneck but it seems highly unlikely because you have a very recent motherboard and are unlikely to use any drive that would cause a bottleneck like that (good choice, I have a Tyan Thunder K7 S2462UNG).
Let me know if any of these suggestions help I'm very interested in what the problem might be.

Joe

[Godlike567]

>> how do you know when you're using the crossover cable and when >>you're using the cable through the switch?

When I login using FTP - the address I connect to is on the gigabit side. Also, my switch isnt registering any traffic on the port.

>> What are your ping times

Less than 0.5 ms with either cards.

I swapped Nics in the systems to see if that was the problem. Same throughput issue. I've ruled out the hardware for the most part. How would I go about checking if there is any broadcast storms or packet errors? When I use ifconfig in linux it shows 0 errors. Also, the Intel NIC software hasnt reported any problems.

I think its an XP issue. I ran some test using iperf which generates traffic over a network and its ram based (to rule out the harddrives). I was sending a sustained 117 MB/s. But on the recieving end it was only around 12 MB/s.

What do you think would be ideal MTU, RWIN, TCP Recieve Window... etc setting for short distance Gigabit to Gigabit? Most of them seem to be calculated by latency but since im on a lan it isnt an issue.

Thanks again Joe. :)

[Magueta]

I don't know what the numbers are because I've never gone through that tweaking process but if you're interested in how it's done there's a way of finding the max MTU, rwin, and tcp receive window using ping. There are a bunch of things you can do with ping like specify the packet size for example. I'm giving you a website that is actually for windows but it explains to some degree what the process is for finding the values. You can find it here (http://snakefoot.fateback.com/tweak/windows/network.html)
Another option is to use another protocol, if you don't need it to be routable you can just use UDP which is much faster than TCP/IP anyday.
Good luck!

Joe

[tecknophreak]

The MTU for gigabit should be set at 9000. It may be possible for you to increase this if the network cards let you. However this should not be the problem with the simptoms you're reporting.

Run this test, take that gigabit cable and connect the two 10/100 Mb/s cards up and run that test. If they report slow, i.e. 10 Mb/s links, then there's something seriously wrong with your comps.

Ok, let's see. Are these the 32-bit or 64-bit pci gigabit cards?

Window size = bandwidth * link delay, or BDP. So you've got 1000000000 b/s of bandwidth and a link delay of say 1 ms. So it's just 1000000000 * 0.01 = 10000000 bit window. or 1,250,000 MB window will top out your link. Of course, you'll need to change your settings.

The attached file should increase your tcp/ip settings on your linux box. Run that then run iperf again.

[fredg]

Not sure how much speed you are really expecting to get here.

Even if you are sure the gigabit link is properly configured and actually being used in both directions in all cases, that only covers what's happening on the wire.

You still have to get the data on and off the wire. And that means reading it from a disk and writing it to another disk.

I don't think you're going to get gigabit rates on and off typical single hard disks on large transfers. You'd have to consider running stripe sets across multiple disks on both ends to get the combined HD bit rates up high enough to make gigabit networking worthwhile.

[tecknophreak]

I haven't seen a harddrive that can't read/write at least 100 Mb/s for a while now. Since he's using gigabit cards, I'm guessing he has pretty decent hard drives. Now if the application sucks at sucking data off the drives and writing data to the drives, thats another story.

Besides, the recieve side on the linux machine was only getting 12 mbps from iperf. As far as I'm aware, it only uses what's in RAM as it's data.

[fredg]

Well, he didn't say what he's using for drives, or how many. It's clear that his controller is IDE though, and one way to cripple one of those is to hang two drives off one channel.

I'm not aware of any drives that will write as fast as they will read, so writing speed is the bottleneck that must be considered.

And like I said, he didn't say how much speed he was expecting.

But if you want to throw rough numbers at it, then take the write speed for a single drive and stripe as many of those as you can afford. You'll think you should get about N times single drive speed off the wire, assuming another bottleneck doesn't appear somewhere else first.

But it rarely works that well off a single controller. To really get the most out of the drives, you'll have to run multiple controllers, and then you may run out of PCI buss bandwidth before filling up a gigabit pipe.

[Godlike567]

>>Not sure how much speed you are really expecting to get here.

I was looking to achieve 30 - 50 MB/s range. I know the hardware can handle it. I've had sustained Reads in Writes within that range. Im using
5 x 120 Gig Hitachi Deskstars w/ 8 megs of cache on both systems. The network cards are 32 bit PCI in the Tyan's 64 bit slots.

I took out the gigabit cards and let the 3coms handle it on there own. using iperf I hit 94 % network usuage and using the raid array it was in the high 80's. respectable with speeds with overhead and all.

Tecknophreak Thank you for the linux setting as they seems to stabilized the transfers quite a bit. I really am apreciative of everyone help. :)

Ive noticed that when sending data back and forth on both systems there seems to be an excessive amount of ACK requests (I believe im using the correct term). For example when sending a file over the 10/100 cards I would transmit near 9.8 MB/s and recieve about 20k worth of data. When I trasmit using the gigabit at 30 MB/s Im recieving around 600k as ACK back from the linux box. Is this figure normal, it seems pretty large to me.

Thanks again for everyone's input, I look forward to more.

[fredg]

3% ACKs on a TCP/IP network are not unusual. You can raise the TCP RWIN value up, and that will lower the amount of ACKs. But don't overdo it, retransmits get more expensive since they are larger.

[saithan]

I don't know if this will help or not but something I did to increase network performance on my redhat9 was to edit the sysctlconfig.

I had to adjust the and the rwin settings.

I use KDE on redhat9 so to alter these settings in the gui i use the kernel tuning tool in :

start menu->Sytem tools->More Sytem tools->Kernel tuneing

the Core heading holds the Rwin settings.
after a few tries the network performance was more than noticable.

[tecknophreak]

Godlike567, you'll have to put those settings in to /etc/rc.local or run tcptune each time you reboot.

what are your settings for
/proc/sys/net/ipv4/tcp_sack
/proc/sys/net/ipv4/tcp_fack
/proc/sys/net/ipv4/tcp_dsack
?
to find these out just type echo /proc/sys/net/ipv4/setting