I was surprised at the terrible performance quoted at yesterday's
strategy meeting for NFS performance, as well as the recommendation
of abandoning NFS for clusters. I've typically seen very acceptable
results on the several Linux clusters I've adminstrated and/or used
(pcfarms, rip, and the CDF Level 3 prototype).
So, I did some quick measurements on the RIP cluster. The nodes were
rip8, the NIS and NFS server, and rip7. Both are 400 MHz P-II nodes,
connected with fast ethernet via a Foundry switch.
Writing a 100 MB file:
rip7:/raid$ time dd if=100MB.file of=~/100MB.copy bs=8k
12800+0 records in
12800+0 records out
0.02user 2.87system 0:29.54elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (299major+12minor)pagefaults 0swaps
Here /raid is a 2-way stripe on rip7, and my home area is mounted from
rip8 (i.e., where ~/100MB.copy goes). I took care to have the 100MB.file
in rip7's cache. Also, note that I used a block size matching the
read/write block size specified in the NFS mount.
NFS writing rate (big file) = 3.38 MB/sec
The local writing rate to the home area on rip8 was about 6 MB/sec.
For a more rigorous test, I copied the linux kernel tree from rip7 to
the NFS mounted home area on rip8:
rip7:/usr/src$ cd linux
rip7:/usr/src/linux$ du -s
36113 .
rip7:/usr/src/linux$ find . -print | wc
2773 2773 68532
rip7:/usr/src/linux$ time cp -d -r . ~/linux/.
0.07user 2.96system 0:44.81elapsed 6%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (10056major+70minor)pagefaults 0swaps
So, the copy of 2773 files with a total length of 36113K was achieved at
a rate of 0.79 MB/sec.
I believe that these are very acceptable rates and disagree with the
recommendation to not use NFS. I am aware of very poor NFS writing
performance from Linux nodes to other UNIX's, particularly OSF/1. However,
all Linux clusters in my experience work well with NFS.
There was also mention in the slides of NIS being immature and not recommended.
Again, in my experience with all Linux clusters NIS has been trouble-free.
Further, on some heterogenous clusters (SDSS, FNDAUx, HPPC) with IRIX NIS
servers, I've not seen any problems with Linux NIS clients. So I can
recommend NIS for Linux clusters as well.
Don Holmgren