PDA

View Full Version : NFS4 server performance



interele
06-Jul-2015, 11:47
Is there a 'simple' guide to getting the most from a NFS4 server
Our servers generally have 60-90 users hanging off them and they
are fine but occasionally things conspire so that the full 180 users
are attached to each one and they get very slow.

I have fiddles with the use_kernel_nfsd_number' and increasing it from
the default 4 to 64 makes a lot of difference but over 128 makes it
slower- I am assuming all those processes exhaust something else

These servers just sit there being NFS4 servers ( for openSUSE
clients ) and nothing else.

Has anyone some guides

Ta

M

jmozdzen
07-Jul-2015, 17:30
Hi M,

the typical bottle necks apply here. Server-wise these are disk i/o, CPU and memory. Network (not only bandwith: NFS has pretty small window sizes, so latency is a factor, too). Protocol overheads, additional services (i.e. DNS delays) and NFS4-specific side effects (i.e. Kerberos performance) will influence, too.

What you haven't told is the work load these users run across the NFS mounts. Are these home directories of up to 180 users, with all the tempfiles and databases (KDE, Firefox,...) running via NFS?

The main factors in our environment were disk I/O throughput and latencies (since switching to bcache, that has been no problem anymore), and setting the "async" options server-side.

> use_kernel_nfsd_number

It depends on the number of requests that come in in parallel - "4" is a very conservative number, more suitable for test setups ;) We're running at 128. Take a look at /proc/net/rpc/nfsd (the "th" values), the first value is the number of threads currently active and the second number counts the number of times all threads were busy. Though, depending on your kernel NFS server version, you may not have reasonable values there (all zeros), then /proc/fs/nfsd/pool_stats is the file to look at (see knfsd-stats.txt for a description).

Regards,
Jens

interele
08-Jul-2015, 08:46
Hi, Thanks for replying

Yes, they are home directories ( for openSUSE 13.2 clients )
Looking at /proc/net/rpc/nfsd shows 0 in the second column
I was wondering if nfsd is running out of buffer space - The machines have 12G of ram so it could have most of it
It seems to happen more often when 180 users are all using LibreOffice - I am going to investigate LO's file locking as a suspect

Ta

M

jmozdzen
08-Jul-2015, 11:49
Hi M,

> It seems to happen more often when 180 users are all using LibreOffice - I am going to investigate LO's file locking as a suspect

and try to run some long-term statistics on server memory/cpu/network/disk i/o. If you have no systems management tool set up for this, fetch MRTG or alike and monitor the appropriate SNMP variables. The resulting graphs may shed some light on the actual bottle neck.

Regards,
Jens

interele
08-Jul-2015, 15:23
I looked at /proc/fs/nfsd and the 'sockets-enqueued' is significantly non-zero
- as in - about half a million... hmmm

M

jmozdzen
08-Jul-2015, 16:49
Hi M,

I looked at /proc/fs/nfsd and the 'sockets-enqueued' is significantly non-zero
- as in - about half a million... hmmm

M

I guess the more important question is - since when? You wrote that you started with 4 threads... if no server reboot was inbetween, then those large numbers could have their origin in those times.

It's more important to monitor that number (as in "increase per time unit") and correlate that with reports of bad overall performance.

Regards,
Jens