Issue:<br />
Latency in processing information over NFS between client and server (specifically IP 10.78.159.32)<br />
See send queues on server always holding data awaiting client to take<br />
Receive queues on Server are processing and never waiting<br />
Isilon Server serving NFS data for workload<br />
CentOS 2.6.32 kernel <br />
<br />
What are the best practices for CentOS tuning for NFS traffic with 1500MTU and 10 gig cards?<br />
What are the defaults?<br />
Could this be a bug in CentOS?<br />
What are other potential causes of CentOS clients not taking in data? (Network card firmware/etc)<br />
What other info is needed to move forward?<br />
<br />
CentOS does not seem to be hammered on CPU or Memory<br />
Client is sending initial receive buffer size of 14848 to server <br />
Also 14848 is max size for client receive buffer advertised from client<br />
I have attached the non-empty TCP socket list to show the send buffers on Server side so you can see why I am asking how to get the CentOS side to take in data faster through buffers.<br />
<br />
What we have found so far:<br />
If we look at the 3rd column at the max send window we see that 159.32 always has a max limit of 14848. The 7th column is the same as initial advertised window size in each case. <br />
As we see there are other IP’s that are able to have window sizes of 10X this (leads me to believe Windows clients as it is SMB traffic with the exception of one connection to 2049 NFS). (Port 304 is used for the NFS TCP locking)<br />
<br />
The command below will tell you on a Linux (client) side what all receive buffers are set to.<br />
# sudo sysctl -a | grep rmem<br />
<br />
From a quick review on internet sources I am seeing that it is common from Linux clients to have an initial buffer size advertised even though the sysctl’s are likely set differently (a few cite 14848 as the byte size as well when using kernel 2.6.32). This should not be the same as the max however (client would not increase the window size the way it is acting now). I am hearing we are not seeing CPU or Memory load on the client side so I would say, as we see the send buffers not draining from the server side, that the client side receive buffers need to increase either initially or incrementally to max in order to handle the load trying to be sent from the Isilon. <br />
<br />
SndWnd - The amount of available buffer space that is advertised by the remote side into which data can be sent. (coming from client side)<br />
<br />
Local Address Foreign Address mxswnd maxseg srtt srtvar sndwnd sncwnd rcvwnd rttimo SR SS ND AS<br />
10.78.159.21.445 10.78.159.41.58486 131328 1460 1 19 129792 2920 1049740 5 X X X X<br />
10.78.159.21.304 10.78.159.36.939 14848 1460 68 16 14848 10220 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.914 14848 1460 1 19 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.684 14848 1460 1 23 14848 14600 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.32.967 14848 1460 5 26 14848 7300 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.713 14848 1460 13 21 14848 249660 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.854 14848 1460 16 32 14848 8760 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.790 14848 1460 1 21 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.52.760 14848 1460 23 29 14848 151954 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.701 14848 1460 1 20 14848 10220 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.952 14848 1460 1 20 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.32.877 14848 1460 21 31 14848 9694 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.681 14848 1460 1 20 14848 8760 525600 0 X X X X<br />
10.78.159.21.445 10.78.159.45.57331 131328 1460 1 16 130304 2920 1049740 2 X X X X<br />
10.78.159.21.304 10.78.159.35.867 14848 1460 1 19 14848 13140 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.1000 14848 1460 13 24 14848 85675 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.778 14848 1460 1 19 14848 7300 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.819 14848 1460 1 19 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.957 14848 1460 13 24 14848 999999 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.912 14848 1460 1 19 14848 8760 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.32.848 14848 1460 1 19 14848 9239 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.32.867 14848 1460 1 22 14848 7300 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.685 14848 1460 13 21 14848 999999 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.865 14848 1460 1 19 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.979 14848 1460 13 22 14848 999999 525600 0 X X X X<br />
10.78.159.21.445 10.78.159.46.56986 131328 1460 1 19 130816 2920 1049740 1 X X X X<br />
10.78.159.21.445 10.78.159.46.56985 131328 1460 1 16 130048 2920 1049740 1 X X X X<br />
10.78.159.21.445 10.78.159.28.56301 131328 1460 0 0 131328 37641 1049740 0 X X X X<br />
10.78.159.21.304 10.78.159.35.899 14848 1460 1 19 14848 10220 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.908 14848 1460 1 21 14848 8760 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.687 14848 1460 1 19 14848 10220 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.32.841 14848 1460 1 19 14848 9003 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.684 14848 1460 1 19 14848 8760 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.32.999 14848 1460 1 19 14848 9469 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.684 14848 1460 1 21 14848 29200 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.32.999 14848 1460 3 22 14848 15476 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.953 14848 1460 13 23 14848 999999 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.799 14848 1460 1 19 14848 8760 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.744 14848 1460 13 20 14848 999999 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.780 14848 1460 1 19 14848 8760 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.1006 14848 1460 13 22 14848 101443 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.842 14848 1460 1 19 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.812 14848 1460 13 21 14848 999999 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.842 14848 1460 1 19 14848 29200 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.843 14848 1460 17 31 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.843 14848 1460 1 19 14848 10220 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.995 14848 1460 13 20 14848 797160 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.1018 14848 1460 1 16 14848 8760 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.52.760 14848 1460 21 32 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.977 14848 1460 1 19 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.752 14848 1460 1 15 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.791 14848 1460 13 22 14848 131710 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.752 14848 1460 1 17 14848 29200 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.790 14848 1460 1 19 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.744 14848 1460 14 25 14848 999999 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.731 14848 1460 1 19 14848 8760 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.736 14848 1460 11 20 14848 83592 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.866 14848 1460 1 19 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.959 14848 1460 1 19 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.937 14848 1460 1 19 14848 10220 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.737 14848 1460 13 28 14848 999999 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.909 14848 1460 13 23 14848 213160 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.849 14848 1460 1 18 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.884 14848 1460 13 24 14848 999999 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.965 14848 1460 3 24 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.710 14848 1460 13 24 14848 213160 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.59.945 14848 1460 40 38 14848 16060 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.820 14848 1460 1 19 14848 8760 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.36.988 14848 1460 108 44 14848 10220 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.820 14848 1460 1 19 14848 29200 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.749 14848 1460 1 19 14848 8760 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.32.742 14848 1460 1 23 14848 8760 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.806 14848 1460 1 19 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.36.823 14848 1460 77 45 14848 14600 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.152.666 14848 1460 13 27 14848 129422 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.713 14848 1460 13 27 14848 11680 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.762 14848 1460 1 19 14848 8760 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.59.969 14848 1460 24 35 14848 16060 525600 0 X X X X<br />
10.78.159.21.2049 10.78.159.35.997 64512 1460 20 26 64512 7300 3728270 0 X X X X<br />
10.78.159.21.304 10.78.159.35.867 14848 1460 8 23 14848 13140 525600 0 X X X X<br />
10.78.159.21.304 10.78.159.35.908 14848 1460 1 19 14848 13140 525600 0 X X X X<br />
<br />
Am I correct that these are the tuning parameters needing adjustment?<br />
#sysctl net.core.rmem_max <br />
max OS receive buffer size for all types of connections.<br />
<br />
#sysctl net.core.rmem_default <br />
OS receive buffer size for all types of connections.<br />
<br />
#sysctl net.ipv4.tcp_rmem <br />
memory allocation min/pressure/max for read buffer<br />
TCP Autotuning setting. "The first value tells the kernel the minimum receive buffer for each TCP connection, and this buffer is always allocated to a TCP socket, even under high pressure on the system. ... The second value specified tells the kernel the default receive buffer allocated for each TCP socket. This value overrides the /proc/sys/net/core/rmem_default value used by other protocols. ... The third and last value specified in this variable specifies the maximum receive buffer that can be allocated for a TCP socket."<br />
<br />
#sysctl net.ipv4.tcp_mem <br />
memory allocation min/pressure/max for buffer space<br />
TCP Autotuning setting. "The tcp_mem variable defines how the TCP stack should behave when it comes to memory usage. ... The first value specified in the tcp_mem variable tells the kernel the low threshold. Below this point, the TCP stack do not bother at all about putting any pressure on the memory usage by different TCP sockets. ... The second value tells the kernel at which point to start pressuring memory usage down. ... The final value tells the kernel how many memory pages it may use maximally. If this value is reached, TCP streams and packets start getting dropped until we reach a lower memory usage again. This value includes all TCP sockets currently in use."<br />
<br />
<br />
<br />
Info:<br />
Internet Engineering Task Force doc on increasing initial window sizes. - <a href="http://tools.ietf.org/html/rfc3390">http://tools.ietf.org/html/rfc3390</a> [<a href="http://tools.ietf.org/html/rfc3390" target="_blank">^</a>]<br />
Internet Engineering Task Force doc on window size acknowledgement - <a href="http://tools.ietf.org/html/rfc813">http://tools.ietf.org/html/rfc813</a> [<a href="http://tools.ietf.org/html/rfc813" target="_blank">^</a>]
↧