Eisler's NFS Blog

Red Hat Releases RHEL 6.2 with pNFS support

2011-12-06T13:55:00.000-08:00

Hat Tip to Trond Mylkelbust, the Official NFS Client maintainer for Linux.

Today the NFS Industry received an early Christmas present from Red Hat.

Today Red Hat announced the availability of RHEL 6.2.

The RHEL 6.2 Features list from RedHat explicitly mentions pNFS (for the files layout only) as supported in RHEL 6.2.

This pNFS client capability complements NetApp's release last month of its Data ONTAP pNFS server.

Note that the Linux community has achieved a first here: this is the first time a Linux vendor of an enterprise-class distribution beat all other commericial operating systems to market with an NFS client. Historically, Linux was way behind commercial operating systems in delivering NFSv3 and NFSv4.0 (by 5 to 10 years). Note that this was acheived slightly less than 3 years after the NFSv4.1 and pNFS standards were ratified, and less than two years after those standards were published as RFC5661.

I want to extend my congratulations and thanks to Trond, Red Hat, the NetApp Linux NFS client engineering team, and Linux development community for the hard work of the past several years that went into this milestone.

NetApp has shipped its pNFS server

2011-11-21T08:46:00.000-08:00

This week is Thanksgiving in the USA, and today the NFS industry has much to be thankful. Because today, Release Candidate 2 of Data ONTAP 8.1 was posted to NetApp's now.netapp.com site and is available for download now. Release Candidate 2 introduces NetApp's pNFS server for Data ONTAP Cluster Mode, as well as the NFSv4.1 server necessary to enable pNFS functionality. This pNFS server supports the files-based layout type, aka LAYOUT4_NFSV4_1_FILES.

As described in Red Hat's documentation, the RHEL 6.2 beta release includes a tech preview of RedHat's upcoming pNFS client for Linux. You can also go the Fedora route.

Two questions that we often get about our pNFS server are:

Is there is a single pNFS metadata server? Answer: no, every node in a Data ONTAP 8 Cluster Mode storage cluster is capable of being a metadata server. The NFSv4.1 client simply NFS mounts a volume via any node of the storage cluster, and that node acts as a metadata server.
What happens when a node hosting a metadata server encounters a failure? The Data ONTAP 8 Cluster Mode system is designed to be fault tolerant if there are two or more nodes in the cluster. Another node will be assigned the network interfaces (essentially, IP addresses) of the failed node, and the NFSv4.1 client will re-connect to the new node, discovering that there has been a metadata server failure, and if necessary, obtain new layouts to any open files that were being accessed over pNFS.

Enjoy NetApp's pNFS server, and have a great Thanksgiving.

www.nasconf.com and www.nfsconf.com offline, material available at nfsv4bat.org

2011-09-30T13:08:00.000-07:00

I've received inquiries about nasconf.com (formerly nfsconf.com) going off line.

Via the wayback machine, Tom Haynes has restored hopefully all of the material, which is mostly presentations made at this valuable, but unfortunately now defunct, conference between the years 2000 and 2005, inclusive.

Tom, thank you, for providing this community service. I hope the time you spent qualifies under NetApp's volunteer benefit.

Will be blogging about NFS here for a while

2011-08-02T14:54:00.000-07:00

NetApp is going in a different direction with corporate blogging, and until that gets resolved with respect to the NFS blog I used to post there, I will be posting here. If the blog location changes I again, I will delete this post, and replace it with one that links to the new spot.

I am working on getting the posts that used to be at blogs.netapp.com/eisler back on line.

Thanks in advance for your understanding.

pNFS client is now part of Fedora 15

2011-08-02T14:46:00.000-07:00

Trond Myklebust, the official Linux NFS client maintainer, told me today:

FYI: as of this morning, Fedora 15 is shipping with 'kernel-2.6.40' which is basically a renamed 3.0 kernel (presumably to avoid trouble with shell scripts that check for the '2.6.x').

The kernel is shipping with both 'files' and 'objects' pNFS modules.

VMware over NFS?

2007-09-11T16:06:00.000-07:00

Nick Triantos of NetApp blogs about blocks-based storage protocols and the NetApp perspective. In as much as I am a public face of NFS zealotry for NetApp, Nick as assumed a similar role for NetApp's blocks protocols. I've been busy with the NFS track (more in another post coming up) at the SNIA conference this week, but just read Nick's blog post about using NFS over VMware, where he makes a strong case for using VMware over NFS.

There is also a presentation at VMworld from Peter Learmonth and Kim Weller of NetApp, and Bud James of BEA that delves more deeply into this notion. The presentation is password protected, but the web page that presented the link to the presentation also provided this password information:

user name: cbv_rep
password: cbvfor9v9r

If the link and password don't work, you can get to the presentation by:

http://www28.cplan.com/cc166/sessions_catalog.jsp

Search for Learmonth in the Speaker Name field.

Enjoy.

The actual presentation will be Wednesday, September 12, 2007 at the Moscone convention center in San Francisco.

NFSv4.1 at the SNIA Developers Conference

2007-08-31T05:15:00.000-07:00

NetApp's PR department issued a press release telling the world I, and my co-editors, Dave Noveck of NetApp and Spencer Shepler of Sun, will be presenting the NFSv4.1 protocol at SNIA's Storage Developer's Conference on September 11, 2007.

That's my signal to finish up some slides. :-)

An NFSv4 ACL editor

2007-07-26T09:00:00.001-07:00

Let's say you have to use NFSv3 but need Access Control Lists (ACLs). Let's say your NFSv3 server does not support one of many proprietary Draft POSIX ACL protocols, but your server does have NFSv4 support and NFSv4 ACLs. Let's also say that an NFSv4 ACL on your server is enforced on NFSv3 access. Is there a way to use NFSv4 ACLs without having an NFSv4 client?

Yes. The idea is to use a user-level NFSv4 client that implements enough of the NFS protocol to read and write NFSv4 ACLs.

A while back I wrote such a beast and it is available at:
http://sourceforge.net/projects/nfsv4-acl-edit/

It has been ported to Solaris and Linux.

The user interface isn't as nice as I'd like, nor does it support Kerberos V5 authentication. But rather than wait for such things to get done in my "ample spare time", I think it is worthwhile to make it more widely know this software exists. Feedback welcome. If this proves popular, I'll find time to add requested features and bug fixes.

NFSv4.1 Bakeathon and pNFS

2007-06-19T19:19:00.000-07:00

Last week I was at Sun Microsystems' campus in Austin, Texas for the NFSv4.1 bakeathon, where various implementors tested NFSv4.1 against each other. The terms of Sun's confidentiality agreement don't allow me to provide details about companies and organizations that attended and how their code did. What I can say is that a total of 7 organizations, including NetApp, brought implementations to Austin, and all implementors had success with interoperability testing.

NFSv4.1 has two big chunks of functionality: sessions and pNFS. Sessions is a new infrastructure that enables exactly once semantics and trunking. By "exactly once" we mean that NFSv4.1 will be able to guarantee that every operation is executed exactly once. This is important for "non-idempotent" operations: operations that if executed twice return different results, for example the file REMOVE operation. Overcoming non-idempotency is necessary for all filesystems, but it is a significant practical problem when the filesystem and the storage are separated by a potentially unreliable communications link as is the case with NFS. Because sessions is a large piece of infrastructure, several implementors in Austin focused on getting sessions to work.

PNFS is parallel NFS: the striping of regular files across several data servers. NFSv4.1 entertains several types of data servers:

Blocks-based, where the pNFS client accesses data via Fibre Channel or iSCSI.
Object Storage-based, where the pNFS client accesses data via the OSD protocol.
File-based, where the pNFS client accesses data via the NFSv4.1 protocol.

Operations to create and delete files, and access directories are always done to a metadata server, regardless what type of data server is used to store regular files.

At Austin, all three pNFS server/data server flavors were there.

Recently Panasas had a press release or two on pNFS, and serveral articles were written. From my perspective, the Byte and Switch article is perhaps the most interesting one to use as fodder for the rest of this blog post, because it expresses opinions that are easy to take issue with.

"You could say NFS was invented by Bill Joy at Sun back in 1983, and the thing hasn't had a major performance upgrade in two decades,"

Welll NFSv3 did add asynchronous I/O, and NFSv4.0 added delegations. In addition, NFS/RDMA adds significant performance wins. I consider pNFS to be yet another step in NFS performance improvements, and doubt it will be the last one either.

When the IETF approves the new standard, which is anticipated by year's end, Panasas will have a significant first-mover advantage.

Note that pNFS has three flavors of data servers, so this is not necessarily the case. Panasas is backing the OSD data server. Whereas, EMC and NetApp are backing the blocks and NAS-based data servers, respectively. Given the amount of storage that is accessible via blocks protocols and NFS, versus object protocols, I would expect some impedance in the market to pNFS over OSD, unless EMC, NetApp, and others have no story for moving on blocks and NFS-based data servers.

The beauty of the files-based data server is that is uses the same protocol as that used to talk to the pNFS metadata server: NFSv4.1. Proponents of other data server protocols might come to appreciate this beauty, and wrap an NFSv4.1 front end onto their data servers.

At least one analyst thinks enterprises don't really need pNFS to improve the performance of clustered systems. "All of the clustered file system NAS vendors have at some fundamental level data coming in over Ethernet that's served by different nodes," says Arun Taneja of the Taneja Group consultancy. "They all do it differently. Panasas does it in a very different way, and I'd call them the odd duck of the group."
But Taneja acknowledges that, if large storage players get behind pNFS, the power of standardization could take over. Then, vendors like BlueArc, Exanet, Isilon, Polyserve (through its alliance with HP), and others would probably look to support it, he says.

While Panasas hopes to widen its appeal through pNFS, another expert says that, for now, pNFS solves problems very specific to HPC environments. "It's not just for big files, it's for multiple, time-sensitive computations," says analyst Mike Karp of Enterprise Management Associates. Where calculations are independent and require lots of instantaneous processing, pNFS could serve a big need.

These analysts are taking an extremely narrow view. "HPC" implies a narrow range of computing such as what United States National Laboratories use. In fact, the need for pNFS is much broader. For example, I constantly get asked by people doing analysis is seismic data (i.e. oil exploration) about pNFS. These folks have requirements for I/O to large files that needs to be accelerated. Storage clusters with clustered parallel filesystems will help (e.g. Data ONTAP GX High Performance Option), but to completely eliminate bottlenecks, pNFS is necessary. Even people doing grid computing with small files see the need for pNFS, because a nice by product of pNFS is that even if a single file is too small to stripe across data servers, lots of little files can be automatically distributed across lots of data servers, thereby removing hot spots and keeping load balanced.

Talk is cheap though; what concrete things are NetApp doing in the pNFS space?

In Data ONTAP GX, the High Performance Option is a parallel, clustered version of WAFL that will the ideal back end filesystem for pNFS. Data ONTAP GX HPO is real and is available now for NFSv3 and CIFS.
NetApp employees are active in finishing the actual standards document. Garth Goodson of NetApp wrote much of the text for first pNFS draft. Since then, it was integrated into the NFSv4.1 document, and I've done significant editing on it. The NFSv4.1 working group has had formal inspections of pNFS and other parts of NFSv4.1. Right now, Spencer Shepler of Sun is folding in the inspectors' comments on the generic pNFS chapter of NFSv4.1, and I will be folding in the comments for the files-based pNFS chapter. Many of the inspectors for the pNFS chapters were NetApp employees.
NetApp wants to ensure that there is a robust pNFS client for the files-based data server in Linux, and to that end, a team of NetApp developers is working with the Linux NFS developer community. Indeed, NetApp brought that client to the Austin bakeathon last week. To be clear, this is work that complements the pNFS work being done at CITI. Think of CITI has the "owner" for the generic pNFS Linux code, and NetApp as driving the file-based-specific parts of the pNFS in Linux.
What might be surprising is that the aforementioned group of NetApp Linux NFS developers are also working on a Linux pNFS data server that is files-based. Why would NetApp care, especially since the Linux NFS server is in theory competition to NetApp storage? In order to have a working client, it needs to have something to test against.
#4 brings up the obvious: what about the Data ONTAP pNFS server as something to test against? NetApp is working on an NFSv4.1 server for Data ONTAP and it will have pNFS support in it, including a files-based data server. However, by deliberate choice, work on a production Data ONTAP NFSv4.1 server (note, NetApp has already demonstrated a prototype pNFS server for Data ONTAP in order to prove the viability of the technology) started after work on the Linux pNFS client and server. The rationale is simple: releasing a Data ONTAP pNFS server into the market well before there is a production quality Linux client it can interoperate with just causes frustration on the part of customers and sales teams.

I have not mentioned release names or actual timing here because (1) roadmaps are never fixed, and (2) even if they were, I'm not allowed to discuss such things. Instead, I'm summarizing activities that are deducible by anyone who reads blogs or mailing lists (Linux and IETF).

NAS Conference Web Site

2007-06-07T15:29:00.000-07:00

The NAS conference (aka the NFS Industry Conference) was a tradition Sun started in the 1980s during the beginnings of NFS but stop having by the early 1990s. In the mid 1990s it was brought back. In the 2000s it got much bigger and was expanded to include CIFS. A year or two ago Sun and SNIA agreed to give SNIA the conference. Anyway in the last month or so, the nasconf.com domain briefly expired, and so all the presentations from the 2000s were offline. After some whining on my part, I'm happy to report nasconf.com is back up.

A Database on NetApp Storage blog

2007-06-07T15:10:00.000-07:00

Sanjay Gulabani is a performance engineer at NetApp who focuses on databases using NetApp storage. He's recently started a blog to discuss ideas and issues on this topic. I expect he'll write a great deal about using databases over NFS, and Oracle over NFS in particular.

NFSv4.1 Bakeathon in Austin next week

2007-06-07T15:03:00.000-07:00

I'm going to Austin, TX next week to attend the NFSv4.1 interoperability testing event at Sun's facility. PNFS (parallel NFS) will be tested by several companies. I'll talk more about pNFS after the testing event. Feel free to post some questions now, and I'll follow up next week.

My reason for going is to get feed back on the NFSv4.1 draft specification which I've been editing along with Spencer Shepler of Sun, and Dave Noveck of NetApp. Back to my editing.

Data ONTAP GX paper summarized in ;login:

2007-06-07T14:49:00.000-07:00

The June 2007 issue of ;login: has a summary from Avishay Traeger (of cs.sunysb.edu) of the GX paper on I co-authored for the 2007 USENIX FAST Conference.

http://usenix.org/publications/login/2007-06/openpdfs/fast07sums.pdf

Storage Virtualization and why blogs beat traditional journalism

2007-04-27T07:14:00.000-07:00

I was searching Google news for various key words, and imagine my astonishment went I came across:

NetApp VP says storage virtualization overrated
NetApp's VP of emerging products Jay Kidd on staying off the storage virtualization bandwagon, competition with Isilon and NetApp's current identity crisis.
searchstorage.techtarget.com/originalContent/0,289142,sid5_gci1252978,00.html - Apr 26, 2007 - Similar pages - Note this

What gives? Is my fellow employee (who is a smart articulate guy) disrespecting my pride and joy, ONTAP GX? If GX is not storage virtualization, then what is? Considering my previous blog post was on GX, I couldn't let this one slide.

If you read the article, you'll find an excellent conversation between the interviewer Beth Pariseau, and Jay Kidd, Senior VP of the Emerging Products Group at NetApp. No where does Jay say "storage virtualization [is] overrated". He does discuss file virtualization that NeoPath, Acopia, and Rainfinity do, and expresses his belief that that those businesses are not profitable.

File virtualization != Storage Virtualization. NetApp sells storage controllers, and products like V-Series and GX are examples storage virtualization at the storage controller level.

So why are blogs better than traditional journalism? Because bloggers get to pick their own headlines. I know Jay didn't pick NetApp VP says storage virtualization overrated as the headline, and doubt the interviewer, Beth Pariseau, did either.

Report from FAST 2007: Data ONTAP GX Paper

2007-02-20T17:49:00.000-08:00

The night before the presentation, Peter Corbett, Dan Nydick, and I worked on the slides Peter was to present. Peter then fine tuned them, arrived exactly on time to present the slides (much to the relief of everyone involved). But the wait was worth it as Peter definitely improved the product (I later presented the paper to data storage class at a university in northern California. You can view that version of the slides [sans performance data, for now at least] on my personal web site).

At the FAST presentation, there were several questions, which I feverishly attempted to paraphrase. Here they are, with the answers given, and in some cases, my color commentary (in italics):

Q: Was a single file system used in the performance charts (given during the presentation)?

A: A single namespace, at least one volume per D-blade, was used.

Q: Why doesn't it scale beyond 24 nodes? What happens at 25?

A: We stopped at 24 because we achieved our initial one million operations/second goal. We believe it will scale beyond 24.

Q: What can limit scaling?

A: The replicated coherent database can potentially be a limiter.

Also, I think the other limiter can potentially be the cluster interconnect, but so far switch vendors can build devices more than capable of switching dozens to low hundreds of nodes.

Q: What benchmark is used for CIFS numbers?

A: Currently there is no standard CIFS benchmark, and we didn't prepare CIFS number for the presentation.

Also our CIFS benchmark numbers use aggregate read and write as NFS do, and will be similar. Note that SFS 4.0 will provide CIFS performance measurements.

Q: Why is write throughput half the read throughput?

A: READs are faster because the benchmark uses sequential I/O, and READs can benefit from read ahead.

Q: For the load balancing mirror feature, aren't you worried about writing multiple mirrors?

A: The load balancing mirrors are read-only. Only the master of a mirror family is writeable.
In the presentation slides I've posted, I've attempted to make this
clearer.

You can read the paper at my personal website.

Data ONTAP GX paper at FAST 2007 this week

2007-02-12T11:01:00.000-08:00

With Peter Corbett, Mike Kazar, Dan Nydick, and Chris Wagner, I submitted a paper on NetApp's Data ONTAP GX architecture and it was accepted for this week's FAST Conference. Peter is scheduled to present our paper this Thursday at 1:30 pm. (Apparently the venue is at the San Jose Marriott).

I'll follow up with a summary of audience questions and reactions.

Connectathon 2007

2007-02-12T10:44:00.000-08:00

I'm trying (with mixed success) to travel less this year, and was going to skip Connectathon this year. However, I currently own the sessions portion of the NFSv4.1 spec, and several developers had issues and questions so I showed up for a few days. I didn't catch many presentations. Three presentations you might look at are Dave Noveck (one of my fellow NFSv4.1 specification editors) via his proxy Tom Talpey presented an excellent summary of new stuff in NFSv4.1 versus NFSv4.0. Ben Rockwood (of the cuddletech storage blog) discussed how he and his employer use NFS in what seems to be an OpenSolaris-only shop. Interestingly, Ben seems to be using bleeding edge OpenSolaris code which is a sharp contrast from my experience with how customers use Linux. Finally, Brent Callaghan of Apple discussed the NFS client and server changes in the upcoming Leopard release of Mac OSX. Brent's talk is a good reminder why a monoculture in the desktop computing space is bad thing, because Brent and his team produced a lot of interesting ideas and innovations. For example, Leopards adds Kerberized NFS support, joing Solaris, Linux, and AIX among the UNIX-like NFS clients, but rather than stick Keberos credentials in a ticket file, the tickets are kept per-user instance of the gssd daemon. BTW, Leopard will have a rudimentary NFSv4 client.

slides from my LISA 2007 presentation

2006-12-29T15:05:00.000-08:00

My slides are available now at eisler.com.

I will be at the USENIX LISA Conference in D.C. this Thursday

2006-12-05T21:13:00.000-08:00

I am scheduled to present on NFSv4 again this year at LISA, this Thursday in morning. I'll post slides sometime after.

Review of "Why NFS Sucks" Paper from the 2006 Linux Symposium

2006-10-27T13:28:00.000-07:00

Olaf Kirch of SUSE/Novell, a major Linux distributor gave a talk on NFS July 26, 2006, at the Linux Symposium. There were some press reports of his presentation, which were stunning in their inaccuracies (e.g Sun invented RFS). In fairness, Olaf's paper has fewer errors, and I'll presume, since I wasn't there, that his presentation was no less accurate than his paper. Also according to first hand accounts of engineers I've exchanged email with, his presentation was far less critical of NFS than the paper. One attendee told me:

The parts of his talk that I did hear, though, left me with the impression that NFSv4 is the best thing since sliced bread since it fixes all the nits and problems with NFSv2/v3.

There were a few inaccuracies, but overall it was actually rather positive.

Kirch's paper pokes lots of holes, some accurate, without always explaining why those holes are there, or how hard it would be to fill them. You might get more information understanding NFS warts by reading the original NFSv2 USENIX paper.

The first section on History claims that AT&T's RFS predated NFS, and Sun designed NFS in reaction to weaknesses of RFS. That is reversed. Sun released NFS with SunOS 2.0 in 1985. RFS arrived in System V Release 3, which came in 1987. I was an employee of Lachman Associates, Inc. at the time, when Lachman obtained early access to System V Release 3 source code, and ported NFS from SunOS 2.0 to System V Release 3 during 1985 (Lachman also ported NFS to System V Release 2 in the same time frame). RFS was, if anything, a reaction to NFS, and is a classic example of the problems one will get if 100% adherence to POSIX semantics is the primary goal of a remote file access protocol. Kirch's explanation of the problems with RFS are correct, but later in his paper he criticizes NFS for not going down the same road.

The paper claims that in the NFSv3 specification was written mostly by Rick Macklem, and published in 1995. RFC1813 , published indeed in 1995, documents the specification, but it was made available in a PostScript form by Sun in 1993. The primary contributors to the specification were Brian Pawlowski, Peter Staubach, Brent Callaghan, and Chet Juszczak (Chet being the catalyst for finally getting the NFS industry to sit down at the 1992 Connectathon and get serious about NFSv3). Rick certainly contributed to NFSv3 specification, but so did several others, and they are listed in the acknowledgements of RFC 1813. For what it is worth, Rick's contributions to NFSv3 out weighed mine.

Regarding the claim that WebNFS gained no real following outside of Sun, I know of many NetApp customers that use it from Solaris clients to NetApp filers. Without NFSv4, it is the most practical way to use NFS through a firewall. It is certainly the case that web browsers unfortunately don't support nfs:// URLs, though I noticed Mac OS X uses nfs:// syntax for some applications. In the Linux world there's no WebNFS following, but that is a function of no support for it in Linux.

Kirch states that the NFSv4 WG formed in reaction to Microsoft rebranding SMB as CIFS. Actually, the rebranding took place after Sun announced WebNFS. The Sun-hosted NFSv4 BOF at the 1996 San Jose IETF meeting took place after the Microsoft-hosted SMB BOF at the 1996 Montreal IETF meeting. I was at the SMB BOF in Montreal, and then co-chaired (with Brent Callaghan) the NFSv4 BOF at San Jose. Readers are free to connect the dots.

In the section on NFS file handles, Kirch notes the difficulties the Linux dentry model poses for NFS. NFS was around for years before Linux arrived. I submit that it "sucks" to design a VFS layer that didn't account for most popular remote file access protocol at the time. Few UNIX systems at the time or now with a VFS layer shares the problems the Linux VFS layer has with NFS.

In the section on write performance, Kirch claims "virtually all" NFS server implementations provide an option to turn off stable writes. Actually Solaris never did, and NetApp's ONTAP never has either. Those two are rather significant servers, and so "virtually all" is a stretch. Actually I'm not even sure most servers had such an option.

At any rate, it is hard to understand what Kirch is arguing, when he claims that even the safe unstable writes of NFSv3 are unsatisfactory. He doesn't offer any alternatives for the problem of ensuring data reliability in face of server or client crash. Once the storage is decoupled from the application, and the storage and computer environment can independently fail, one has this problem.

As for his claim that the performance gain of NFSv3 safe unstable writes is a mirage due to internal write buffers in modern disk drives that don't actually flush data, in my experience, NFS vendors are well aware of the issue, and spend a lot of engineering resources to keep those disk buffers stable or force them to disk. The SPEC SFS committee reviews benchmark all the time, and rigorously enforces the requirement that committed NFSv3 writes go to stable storage.

In the section on NFS over UDP, Kirch makes some concise and excellent arguments for why you should use NFS over TCP.

Kirch's criticisms in the Retransmitted Requests section are dead on accurate. This is why NFSv4.1 will support true exactly once semantics (I spent much of the summer getting the NFSv4.1 spec in shape for the exactly once semantics description which is why my blogging output has been pathetic of late).

There are some inaccuracies in the Cache Consistency section. Kirch claims the client at regular intervals revalidates the cache. Actually clients set a time to live on the cache for a certain interval, and the next time the cache is accessed, the cache is revalidated if the time to live has expired. So if a file is cached, but not actively in use (no process is issuing read or write system calls to it), no over the network revalidation requests (GETATTRs) occur.

Kirch also claims "most file systems store time stamps with second granularity". Perhaps in Linux this is the case. Outside the Linux world, file systems have been storing time stamps with microsecond or finer resolution for at least a decade, probably closer to two decades. It is certainly a huge problem if you are using Linux as your NFS server.

Kirch also glosses over the fact that applications that concurrently access the same file need to have a synchronization method, and that this method is usually byte range file locking. He mentions that NFS clients that set a byte range file lock will either bypass the cache for reads or writes, or invalidate the cache before each read, and flush the cache after each write. But he doesn't note that even if an application were doing concurrent I/O to the same file on a local file system, synchronization would be necessary. This is no different than a multi-threaded application accessing a shared data structure. Synchronization primitives like spinlocks are needed, even the the data structure is kept in local memory.

It is hard to tell whether Kirch considers cache consistency a performance problem, or a correctness problem, as he dimisses NFSv4 delegations which are not available when there is contention, and in a later section notes that cluster file systems he touted earlier as possible solutions have their own problems, including scaling "beyond a few hundred nodes". NetApp has many customers with grid computing farms of thousands to tens of thousands of NFSv3 clients accessing as few as one filer. For some, NFSv4 delegations will be very appopriate.

The section on POSIX Conformance is accurate.

The section on Access Control Lists is mostly accurate. Note that when the ACCESS procedure was introduced in NFSv3, ACLs weren't widely used in UNIX at all. ACCESS was needed anyway to deal with the situation where NFS servers mapped superuser (uid 0) to "nobody", but clients would let superuser open the file anway, resulting in user surprises, like being able to read the parts of a file with mode 0000 that were in local cache but not the uncached bits. The deficiencies he states are issues in NFSv4 ACLs are actually problems with the Linux implementation not the protocol itself. Kirch is accurate that mapping NFSv4 ACLs to draft (but never standardized) POSIX ACLs is not always possible. That is by intent; it was never a goal to provide a perfect mapping. NTFS ACLs have won, and it is time to move on from draft POSIX ACLs.

The section on NFS Security is accurate.

In the section on NFS File Locking, Kirch states that no one has explained why NFS originally did not support file locking. The explanation is SunOS 2.0 was based on a 4.2 BSD kernel, and 4.2BSD had very limited support for file locking. Only when SunOS added support for System V APIs, and complied with the System V Interface Definition (SVID) did Sun acknowledge the requirement to support byte range locking on NFS and local file systems. This section is mostly accurate, but skips noting the vast improvements NFSv4 makes over NFSv3 in terms of lock recovery.

Kirch's appraisals of AFS and CIFS are fairly accurate, though I cannot reconcile his accurate statement that "crash recovery" in CIFS is the "job of the application" with his opinion "CIFS could be serious competition to NFS in the Linux world". Without real crash recovery, except perhaps for desktops, CIFS isn't a viable competitor to NFS. For example you don't see Oracle recommending its database be used over CIFS. If CIFS had crash recovery, there might never have been an NFSv4.

In the Future NFS trends section, Kirch doubts whether NFSv4 will meet its goal of interoperability with the Windows world. It already has. Not in the sense that NFSv4 is widely deployed on Windows (even though Hummingbird has an NFSv4 client for Windows), but in the sense that with state, on multiprotocol servers like filers, NFSv4 clients can coordinate much better with CIFS clients, and a CIFS open cannot suddenly stop NFSv4 I/O to previously opened files, unlike NFSv3 I/O.

In the section So How bad is it really, Kirch says NFSv4 ACLs aren't CIFS compatible. News to those of us at NetApp. Our NFSv4 and NTFS ACLs are pretty much the same. As for there being "no mechanism to enforce NFSv4 ACLs locally, or via NFSv3", filers and other NFS servers enforce NFSv4 ACLs just fine, as do local filesystems on conventional systems like ZFS on Solaris. Perhaps he is talking about issues in Linux.

Kirch is correct that the inability to perform callbacks over an established TCP connection is an issue. NFSv4.1 will address it (another area of the NFSv4.1 spec that I've been hammering on). He also suggests NFS should have a better session protocol to enable a more efficient and robust replay detection cache. Again, to be fixed in NFSv4.1.

OSDL's NFSv4 Press Release

2006-10-27T11:32:00.000-07:00

I got a question about the implications about this excerpt from OSDL's NFSv4 press release:

The Open Source Development Labs (OSDL), the global consortium dedicated to accelerating the adoption of Linux® and open source software, today announced that the Network File System v4 (NFSv4) for Linux is available in Red Hat Enterprise Linux from Red Hat and SUSE Linux Enterprise from Novell. This milestone reflects the maturity of NFSv4 for Linux in the enterprise and coincides with Network Appliance’s latest donation of $100,000 to the NFSv4 testing community.

''NFS testing has been a key priority for OSDL and the Linux development community, and we have passed a significant milestone for it to be ready for enterprise validation,'' said Stuart Cohen, CEO of OSDL.

First, this is all good news, and it is consistent with the claims I've made last year at SNIA and LISA that, unlike the history with NFSv3, Linux is not lagging the industry on NFSv4. There are several commerical NFS vendors that are behind Linux in NFSv4 support.

Second, given the juxtaposition of "test", "significant milestone" , "Enterprise", and "Linux", a reasonable reader might conclude that OSDL is stating that Red Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise (SLE) have passed all of OSDL's NFSv4 tests, and OSDL is stating NFSv4 on the current releases of those two distributions are enterprise ready.

I asked around and apparently OSDL did its testing in Linux kernel code from kernel.org, and not RHEL or SLE. RHEL and SLE at the time this blog post was written did not have all the necessary NFSv4 updates. I'm told that RHEL and SLE will need several of updates from the mainline (kernel.org) code before both distributions have an NFSv4 implementation that is "ready for enterprise validation."

XDR is now a full Standard!

2006-05-09T18:28:00.000-07:00

Why am I using fixed width font? Because today the RFC editor published RFC 4506, the specification for XDR, and it is only fitting to use IETF's preferred character spacing to note this event. XDR is the data encoding standard for ONC RPC and NFS.

This is the culmination of a long process that started when RFC 1014, an informational RFC for XDR, was published in 1987. For me, the process started in 1997, when Bill Janssen and I submitted implementation reports showing that XDR qualified as a Draft Standard.

New IETF full Standards are rare beasts these days. RFC 4506 is assigned Standard number 67. Standard number 66 - RFC 3986 - was published January of last year.

Thanks to Bob Lyon for inventing XDR. And thanks to Kevin Coffman, Benny Halevy, Jon Peterson, Peter Astrand and Bryan Olson for helping to cross the Ts and dot the Is on the final document.

NFSv3 Exclusive Create and NTFS Qtrees

2006-04-11T14:43:00.000-07:00

A customer recently was having trouble using NTFS qtrees in Data ONTAP, when using NFSv3 or NFSv4 to gunzip some files. No such problem with NFSv2. It was narrowed down to the fact gunzip, or at least the gunzip being used by the customer, creates files with the exclusive create flag set.

A file created by the open() system call with the O_EXCL flag present tells the kernel (UNIX or Linux), that if the specified file already exists, return an error, otherwise create the file. This allows applications that want to use lock files to work correctly. However, NFSv2 doesn't do anything special with exclusive create; its CREATE procedure is used for exclusive and non-exclusive create. If the file already exists, then NFSv2 CREATE just returns success from the NFSv2 server to the NFSv2 client. NFSv2 clients simulate the O_EXCL semantic by doing an over the network NFSv2 LOOKUP procedure to see if the file exists, and if it does, return an error to the process attempting the open(), otherwise, it issues the CREATE, and returns the result from the NFSv2 server for the CREATE (which will likely be success, barring permissions issues, out of space issues, or other issues). Clearly this isn't useful for creating lock files from multiple NFS clients because two clients could both find that a file doesn't not exist, and both issue the CREATE operation and both get success.

Enter NFSv3 CREATE. The designers of NFSv3 (BTW, I'm a credited designer, but I can't take any credit for NFSv3 CREATE) produced a very clever yet simple algorithm for implemting exclusive create. Here are the arguments to NFSv3 CREATE:

CREATE3res NFSPROC3_CREATE(CREATE3args) = 8;

enum createmode3 {
UNCHECKED = 0,
GUARDED = 1,
EXCLUSIVE = 2
};

union createhow3 switch (createmode3 mode) {
case UNCHECKED:
case GUARDED:
sattr3 obj_attributes;
case EXCLUSIVE:
createverf3 verf;
};

struct CREATE3args {
diropargs3 where;
createhow3 how;
};

The key thing to understand is that if a non-exclusive create is done, the client provides an initial set of attributes, most likely consisting of the permission bits. However if an exclusive create is done, the client provides not attributes, but does offer a 64 it verifier. What happens in an exclusive CREATE is that the verifier is recorded in one of new file's attributes. If for some reason the client has to retry the request due to a timeout, or server re boot, the retry uses the same verifier. Because the verifier in the request matches what is stored in the file, the server returns success to the client, rather an NFS3ERR_EXIST error. If another client tries to do an exclusive CREATE around the same time, its verifier won't match what the server has recorded in the file, and so the other client gets NFS3ERR_EXIST. So now we have a perfect implementation of POSIX exclusive file create semantics. But we aren't quite done because the recall that the client didn't get set the desired permission bits. The NFSv3 protocol requires the "winner" of the exclusive create to follow up with an NFSv3 SETATTR operating to set all the attributes, including the mode bits.

Here is where we get into trouble with NTFS qtrees in ONTAP. With an NTFS qtree, CIFS and CIFS alone owns the security attributes of a file. So when the NFSv3 client issues the SETATTR to set things like owner, group, and mode bits, ONTAP returns an error. This causes an error to be returned to the process on the NFSv3 client that issued the open() with the O_EXCL|O_CREAT flags.

NFSv4 uses an OPEN operation, but OPEN implements exclusive create the same way.

The vexing thing is that the SETATTR is unnecessary because this is an NTFS qtree; NTFS has already filled in reasonable attributes for the file. But there's nothing in the NFSv[34] protocols to tell the client that.

What do to besides switching to NFSv2 or UNIX qtrees? You can enable the

cifs.ntfs_ignore_unix_security_ops

option on your filer. This option will cause ONTAP to ignore any NFS SETATTR requests, but return success instead of an error.

What I find very interesting is how rare this situation comes up. Very few UNIX utilities apparently attempt exclusive creates. It is curious that gunzip does an exclusive create at all. But if you are depending on gunzip to fail when it attempts to overwrite an existing file, avoid NFSv2.

Connectathon 2006

2006-03-11T08:18:00.000-08:00

I was at Connectathon last week, and gave a presentation, "NFS over TCP, Again". The slides are now posted at the Connectathon web site. The material should hopefully be self-explanatory, but I'll annotate some it here based on the questions and discussions.

Slide 3 asks "Why NFS/TCP?" In addition to the reasons I gave, Max Matveev pointed out that even though both TCP and UDP have the same weak 16 bit checksum algorithm (a topic discussed in more depth by Alok Aggarwal), it turns out NFS over UDP/IP is much more prone to data corruption than over TCP/IP. NFS needs to send requests and responses that exceed the Max Transmission Unit (MTU) of the network media used between the NFS client and server. TCP does this by breaking the NFS message into segments which will fit into the MTU. UDP does this by breaking the NFS message into IP fragments that each fit into the MTU. With TCP, each segment has a unique sequence number. With UDP, each fragment of a datagram shares a per-datagram 16 bit identifier, but has a unique fragment offset to indicate the fragment's place in the datagram. Let's say we are using NFS/UDP, and an NFS WRITE request is sent at time T, with datagram identifier X. The request is broken into N fragments. The first fragment is lost in the transmit somewhere, but the server receives the last N-1 fragments and holds them until it gets the first fragment, or the time to live (TTL) timer on each of the fragments expires.

Meanwhile, the client is busy doing other NFS/UDP things, and the datagram identifier gets re-used. The identifier is just 16 bits; assuming 32 kilo byte writes, giga bit/sec transmission speeds, then 2^16 * 32 * 1024 * 8 / 1000^3 is just 17.2 seconds. If the TTL is greater than 17 seconds, then the re-use of the identifier for another 32 Kbyte NFS WRITE will result in the first fragment of the new NFS request being used as the first fragment of the old NFS request. That first fragment has some interesting stuff in it, such as the file handle, and the offset into the file. If the file handles are different, then we are writing data for one file into another file. That's a security hole and a data corruption. If the file handles are the same, and the offsets are different, then we get data corruption. If the file handles and offsets are the same, we can still get data corruption, because 17 seconds ago, a retry of the first NFS WRITE might have succeeded with no transmission loss, and this new NFS WRITE request is an intentional over write (say a database record update).

I admit to never encountering the above myself, but I'd long since given up on NFS/UDP back when ethernet was a lot slower.

Slide 4 says that in Linux, the NFS/UDP total timeout is about a minute. Someone, who shall go nameless to protect the guilty, challenged that. After the presentation, I did a quick experiment with a Linux client that is running:

2.6.11-1.27_FC3 #1 Tue May 17 20:27:37 EDT 2005 i686 i686 i386 GNU/Linux

At 16:07:39 I tried to do an ls of an NFSv3/UDP mount point to a dead NFS server, and collected a packet trace. The packet trace showed retransmissions at relative time (in seconds) offsets of 9.9, 19.8, 39.7, 1.1, 2.1, 4.3, 8.8, 16.6, 35.1, 1.1, 2.2 4.4, 8.8, etc. At 16:07:48 , the messages log wrote "server not responding".

The inital timeout appears to be 10 seconds [not 100 milliseconds as I claimed in the slide], the overall "call" timeout is about a minute (9.9+19.8+39.7 = 69.4 secs ~= 16:07:48 - 16:07:39 = 21+48 = 69 secs) and then the algorithm looks extremely Solaris-like, with the exception that instead of 35.1 seconds, Solaris would use 20 seconds on the 5th retransmit.

In slide 9, I advocate using NULL RPC pings to probe whether a TCP connection is alive or not. This is to allow the client to quickly deal with the situation where a server crashed or failed over without sending a TCP disconnect indication. I didn't go into details about an algorithm, but here is what I had in mind:

When a request is sent over the connection, start (or reset if one is already started) a server crash timer that will be less than the timeout specified in the timeo= mount option. Each time a response from the NFS server is received, cancel the timer. Also, cancel the timer if the connection is ever disconnected.

When the server crash timer fires, send a NULL RPC (procedure zero) request. Reset the server crash timer. When a response to the NULL RPC request is received, cancel the server crash timer.

I was asked about a value for the server crash timer. I suggested 10 seconds. Someone objected that over low speed and/or high latency links, 10 seconds might not be enough. Even at 14.4 kbits/sec, 10 seconds is plenty of time to transmit a NULL RPC request (98 bytes over ethernet) and receive its response (86 bytes):

( 98 + 86 ) * 8 / (14.4 * 1024) = 0.0998 seconds = 99.8 milliseconds

As for high latency, Brent Welch pointed out that long distance WAN links aren't going to exceed a few hundred milliseconds.

Another objection was the additional traffic these NULL pings will introduce. This objection misses the context for why I suggested pings. At least one NFS client out there uses 10 second RPC timeouts over TCP with a justification being that the client needs to quickly detected server crash or failover. With 10 second RPC timeouts, the traffic is going to be much higher than with my proposed 10 second server crash timeout. And from the algorithm I've presented here, the NULL pings won't happen any more frequently then once per 10 seconds between any client and server pair.

Slide 10 states,

RFC3530 requires NFSv4 server to disconnect any
time it detects an NFSv4 client sending a retry over the
same connection

Rick Macklem pointed out that the RFC doesn't explicitly say that. He is correct. But it does say the server MUST not drop a request without disconnecting. NFS servers usually have a work avoidance cache whereby when an impatient client re-sends a request for a request that is in progress, the server drops the re-sent request rather than re-process the request. When I wrote that slide, I was not anticipating that a server implementation would not support work avoidance. However, in his own Connectathon presentation, Ric k made some pretty interesting arguments for an NFSv4 server not supporting work avoidance.

Slide 16 gives advice for NFS users. Which is to verify your NFS client's default timeout, and if it is under 60 seconds, increase it. A note on verifying the timeout. As an example method to break the network path from client to server, I listed disconnecting the client from the network switch. Depending on the client, that might not work because if the client's network interface adapter does not detect the presence of the local area network, it might indicate that to the IP layer, and the IP layer in turn might report a network path problem immediately. Another way to do this is to disconnect the server from the switch. It may be that your server is in production, and you don't want to do that. So a third way would be to interpose a switch between your client and the main switch, and break the connection between the interposed switch and the main switch. (The way I do this is to put the server in a break point so it stops responding.)

Enough about my presentation, but feel free to post questions in the comments.

Here are some comments on some of the other presentations.

Sam Falkner and Lisa Week gave an NFSv4 ACl talk, discussing some work they are doing to integrate the POSIX and NFSv4 ACL models for authorization. Slide 3 mentions that the new ZFS file system implements pure NFSv4 ACLs (just like Data ONTAP's WAFL does; great ideas transcend companies it appears :-). Slide 7 has a interesting idea for integrating ACLs with UNIX mode bits.
Alok Aggarwal of Sun presented his ideas for adding checksums to the NFSv4 protocol. While checksums being part of the NFSv4 protocol is a long time from now, I think Alok makes a strong case for investing in good networking and storage hardware that will hopefully be less susceptible to corrupting data.
Tom Talpey of NetApp gave an NFS/RDMA update. The news on this effort includes: (1) The LINUX NFS/RDMA server work has moved from CITI (University of Michigan) to Open Grid; (2) Sun and NetApp are funding an OpenSolaris client and server implementation at Ohio State University.
Garth Goodson of NetApp gave a useful summary of what Parallel NFS (pNFS) is, and its current status.
Tom Talpey gave an overview of the bugs in the Network Status Monitor protocol, which to me makes yet another case for using NFSv4.
Bryce Harrington of Open Source Development Labs (OSDL) discussed OSDL's efforts to test the Linux NFSv4 client and server.
Lisa Week presented the current state of the NFSv4.1 protocol. The current "what's in list" has: pNFS, directory delegations and notifications, SECINFO changes, exactly once semantics (aka sessions), implementation IDs, and clarifications and corrections motivated by NFSv4.0 implementation experience.
Tom Haynes of Sun discussed issues around scaling NFS server export s. The big takeaway is that in order to do client-based access control on each NFS request, servers need to consider vast scales. A grid of say 25,000 NFS clients X say 1300 exports combined with some really horrendous automounters translates into real challenges for NAS vendors playing in the high end. Slide 20 summarizes some excellent advice for server vendors.
Rick Macklem of the University of Guelph presented his ideas on a Recent Request Cache for NFSv4. After he took audience abuse for *gasp* using an overhead projector (which I suspect some of the age-20-something attendees had never seen before), he delivered his talk (on handwritten transparencies of course :-). I really did like his idea for computing a checksum of some of the NFS arguments and using that as an additional key. As I mentioned in my talk, RPC transaction identifier (aka XID) re-use, due to bad XID generation algorithms, causes lots of pain for some users of some NFS clients (which will go nameless to protect the guilty). Another very cool idea from Rick was using the TCP-level acknowledgement from the client to the server as an indication to the server that the client received the NFS response. The server can then delete the response from the request cache entry. Or at least, the server can move that response nearer to the front of the might-be-or-to-be-deleted list of responses. Tom Talpey asked Rick about the layering violation this would cause. Rick suggested a socket option be created for allowing the server to receive a callback when the client's TCP receiver acknowledge receipt.
Jeremy Allison of the Samba Team gave an update on Samba. It is always stimulating to listen to Jeremy predict the impending death of NFS and its takeover by CIFS. Meanwhile the number NFSv4 implementations grows (more on that in a bit).
Andy Adamson of CITI at University of the Michigan discussed the work he is doing on SPKM-3 (Simple Public Key Mechanism). SPKM-3 is a GSS-API security mechanism I wrote an RFC for several years back. SPKM-3 was in turn based on the SPKM-2 specification that Carlise Adams of what was then Bell Northern Research wrote years before. Andy noted that the current SPKM RFCs use outdated crypto algorithms, and old X.509 public key certificate specifications and so the document needs updating. The consumers of SPKM will be people who want to use NFSv4 on transcontinental links, yet are in different organizations making Kerberos V5 not feasible.
I didn't get to Sam Falkner's NFSv4/DTRACE talk, nor any of the NDMP talks.

In addition to the talks, there was of course interoperability testing of NFS, CIFS, SSH, and NDMP. NetApp was there testing CIFS, NFS, NDMP, and demonstrated pNFS. There were two new NFSv4 implementations from companies I'm not allowed to mention (due to Connectathon non-disclosure rules). Without naming more companies, I learned that two other companies are planning on releasing NFSv4 features. So these four newcomers, plus BSD, EMC, Hummingbird, IBM (AIX) Linux, NetApp, and Sun will bring us to 11 NFSv4 implementations.

Real Authentication in NFS

2006-02-24T07:44:00.000-08:00

I get asked a lot about what can be done to prevent the following:

nfs client% cd /home/jim
/home/jim: Permission denied
nfs client% ls -ld /home/jim
drwx------ 2 jim grp1 117 Feb 24 07:48 /home/jim/
nfs client% su
Password:
nfs client# su jim
nfs client% cd /home/jim
/home/jim

What is happening is that user jim has set the permissions on his data to 0700 meaning only he, the owner, should get access. But someone on the NFS client with knowledge of the super-user password can become root (user id 0), and then become jim and circumvent jim's protections. The reason why this works is that the NFS server is accepting AUTH_SYS credentials, which are basically, a user id, and 1 to 17 group ids. Simply su'ing to jim causes the NFS client in the kernel to pick up jim's user id and group ids.

Some people have suggested if a more secure directory service like LDAP is used, especially if its configured to use Kerberos V5 authentication, that this is providing Kerberos authentication and so will defeat the attack. No, that is not the case. All that does is make sure the user using LDAP is authenticated via Kerberos (and the LDAP server is authenticated to the user via Kerberos). While this is a good thing, it does absolutely nothing to prevent the scenario above.

The only thing today that prevents the scenario is to use Kerberos V5 (or some other strong authentication system, but Kerberos V5 is what most vendors have) authentication in the NFS traffic itself. This means exporting the volume with option sec=krb5 (or krb5i, or krb5p), and without anon=0 and without root=.

What happens is that even if the attacker su'es to jim, unless he knows jim's Kerberos password, he cannot become user jim over the NFS connection. Even attempting to access /home/jim as super-user, even with Kerberos credentials for super-user, is defeated, because super-user, uid 0, will be mapped to user nobody (since anon=0 and root= are absent in the export options).

Restricting access knowledge of the super-user password, while an excellent practice, is no panacea either. This is because synthetic, user-level NFS clients aren't rocket science to write, and they can be written so that any uid can be specified in the AUTH_SYS credential. There are "nfs shell" programs out there for anyone to download. While the one I've tried isn't written to allow arbitrary user ids to be inserted into the credentials of the NFS requests, it wouldn't be hard to change it.

You might find the following links interesting:

Connectathon is next week Feb 27-March 3, 2006

2006-02-24T07:35:00.000-08:00

I'll be presenting on "NFS over TCP, again" (the "again" is because this will be the third time I'll have presented on this topic) on Wednesday. Unfortunately, the talks aren't open to non-registrants as they were in past years. However, the Connectathon folks are usually very good about getting slides posted within days of the event ending. After Connectathon, I plan to blog more about it, my presentation and other presentations.

I gave a talk at LISA '05

2005-12-14T17:51:00.000-08:00

I was in San Diego last week to give a talk at the Hit the Ground Running Session at the LISA '05 conference. USENIX has now posted them, or read the slides here:

As an interesting aside, you'll note the disposable email address in the first slide. This email address shows up in in exactly two places on the web (as least according to Google), and Google only finds the PDF version at usenix.org. And yet this address has yet to be spammed. Interesting. I can believe spammers aren't adept at parsing text in images, but PDF?

Changes to comments settings

2005-09-21T20:38:00.000-07:00

I've changed the comments settings for this blog so that non-blogger members can post. However, comment posts require the verification thing. We'll see if this promotes spam or content.

opensolaris.org: the Future of Open Source Communities?

2005-09-21T18:28:00.000-07:00

Three months ago I blogged about Sun posting its NFS client and server code to opensolaris.org. At the time I was thinking that from the perspective of NFS implementers this was a way to enhance interoperability since source code of one the major and certainly the most mature NFS implementations was online for anyone to browse and used as a debugging aid, if not as a basis for implementing a competing NFS client or server.

A few weeks back I visited opensolaris.org to look for the source code to their oh so cool source code browser. After I didn't find it (yet: the author says it is on the TODO list to post it), I visited some of "communities" (basically interest groups for particular aspect of OpenSolaris) and was astonished by the high degree of activity from non-Sun employees. Curious, I went to the NFS community, and noticed that the blogs of the NFSers were linked from there (very nice). In particular, Eric Kustarz posted a blog article about a change to how the Solaris client dealt with privileged ports.

Reading Eric's article, and the comments, aside from some concerns I had about the change, I was struck by several thoughts. One, like conventional open source, the world outside of Sun now has direct, and early (i.e. before first customer ship) visibility into what is going on. Two, unlike many large open source projects, the information for getting that visibility is well organized. I don't have to subscribe to an alias with thousands of messages per day in order get that visibility into the particular parts of the operating system I'm interested in. In a sense, sourceforge with its numerous project pages, has this already and probably was a source of inspiration. But sourceforge is a collection of projects, whereas opensolaris.org has a common "look and feel" to all the communities that are part of the greater whole: OpenSolaris. Three, since it this easy to track what is going on in Solaris NFS land, maybe I could influence the outcome?

I tested the latter hypothesis by posting a comment to the NFS discussion forum. I suggested a slightly alternative approach after presenting some of the pitfalls of change. Within minutes, Noel Dellofano responded, and agreed to consider my comments.

This is revolutionary: as an employee of another NFS server vendor I could influence the design and implementation of an important NFS client without having to wait for our mutual customers to file a trouble ticket. And as we all know, filing trouble tickets is not always the fastest way to get a resolution, because we are talking about code that has already been released. Vendors understandably have heavy processes for vetting and limiting change to released products. So that what's in it for the customer, Sun, and other vendors: fewer interoperability bugs out of the chute.

But I think the bigger point is that because Sun has made opensolaris.org so easy to navigate, so easy participate in, and so open to "outsiders", (not to mention flame free), those "outsiders" are going to find that they get much more leverage with OpenSolaris than with other open operating systems. By "leverage" I mean:

leverage = productive outcomes / time spent

Here's another example illustrating leverage. Without naming names, or naming operating systems, I once spent several hours of debating a programmer on an issue with an open source operating system's NFS implementation. After noting that the open source operating system's file system design didn't lend itself well to supporting NFS semantics, one of the retorts I got back was: "I think the future of file access for [this operating system] is CIFS". That's low leverage.

High leverage attracts participation, and the path from participant to contributor can be a slippery slope.

Whether this higher leverage translates into increased market share for OpenSolaris versus other open source kernels remains to be seen. But the design and execution of opensolaris.org may represent the future of open source communities.

My Presentation at the Recent SNIA Conference

2005-08-17T09:57:00.000-07:00

I gave a presentation at the 2005 SNIA Developer Solutions Conference, entitled the Future of NFS. You can read it now.

A great day for NFS interoperability and proliferation!

2005-06-14T09:43:00.000-07:00

The opensolaris.org folks have released Solaris 10 source code. This is only about 6 months after the Solaris 10 FCS, which considering the legal issues is a tremendous accomplishment.
For NFSers this gives us access to source code of the Solaris 10 NFS client and server, especially the Solaris NFSv4 client and server. Make sure you understand the terms and conditions of the licenses for this code before using it.

It does not appear that most of the GSS-API code made it, including the RPCSEC_GSS source code. You'll have to scour the net for the tirpc-99 wad Sun released in 1999. One of these days,
I'll have to post the source code for that.

Using Active Directory as your KDC for NFS

2005-06-08T10:54:00.000-07:00

Recently I've been asked how to use Active Directory as the Key Distribution Center (KDC) for NFS, especially when used with NetApp filers and Linux 2.6 clients.

At the theoretical level, I've always know this was possible. I've used Solaris 10 NFSv[234] clients with filers configured to use Active Directory. I've used CITI's early access NFSv3 w/
Kerberos V5 authentication stuff for Linux 2.4 with filers using Active Directory. And of course, back in my Sun days, I led the team that proved NFS clients and servers could authenticate via Active Directory, work which to this day is the best documented example of how to do so.

But now that Linux 2.6 with NFSv4 and NFS/Kerberos V5 authentication is getting more real, does this still work, and if so, with all 3 NFS versions? It is a reasonable question, since Linux 2.6 continues to change.

I'm happy to report that with Windows 2000 (and 2003!) as the KDC, Fedora Core 3 (Linux 2.6.11-1.27_FC3) as the NFS client, and Data ONTAP 7.0.0.1 as the NFSv4 server, the answer is yes, at least as measured by this trivial sanity checking script:

#!/bin/sh
# NFS/Kerberos sanity.sh for Linux 2.6

if [ $# -lt 3 ]
then
echo Usage: $0 server_name server_export mount_point
echo example:
echo "  " $0 mre1.sim /vol/vol0/home /mnt
exit 1
fi

size=1m
file=$size.$$.`uname -n`
echo file = $file
serv=$1
fs=$2
mnt=$3

cd /
sudo umount -f $mnt

for proto in tcp udp ;
do
case $proto in
udp )
moreopts=",rsize=4096,wsize=4096"
;;
* )
moreopts=""
;;
esac

for vers in  2 3 4  ;
do
if [ $proto = udp ] && [ $vers = 4 ]
then
echo NFSv4 is not supported over udp
else
for sec in sys krb5 krb5i ; # krb5p ;
do
  echo ----------------------------------------
  case $vers in
  4 )
          opts="-t nfs4 -o proto=$proto,sec=${sec}$moreopts"
          ;;
  * )
          opts="-o vers=$vers,proto=$proto,sec=${sec}$moreopts"
          ;;
  esac


  if sudo mount $opts $serv:$fs $mnt ;
  then
          cd $mnt
          mount | grep -w $mnt
          rm -f $file
          if time dd if=/dev/zero of=$file bs=1024 count=1024 ;
          then
                  echo $opts PASS
                  rm -f $file
          else
                  echo $opts FAIL
                  exit 1
          fi
  else
          echo sudo mount $opts failed. FAIL
          exit 1
  fi

  cd /
  sudo umount -f $mnt
done
fi
done
done

But before one runs this script, some configuration on the KDC, the Linux client, and the ONTAP filer are necessary.

Let's look at the KDC.

I am assuming that an Active Directory realm has been created. My example uses ADNFSV4.LAB.NETAPP.COM as the Kerberos realm.

The first thing we need to create are users. Let's walk through an example for creating a user named jsmith. First thing we do is highlight the Users folder in Active Directory:

Highlight Users Folder
Windows Server 2000 Screenshot

Having done that, right click in the folder to pop up the action menu for the folder:

Pop Up Action Menu for Users
Windows Server 2000 Screenshot

Pick the New --> User option. Now we fill in the information. I find that the First name, Full name, and User login name have to agree with each other, but you may have a different experience:

Fill in Information for New User
Windows Server 2000 Screenshot

Now click next to get to the password setting window:

Password for New User
Windows Server 2000 Screenshot

Finally, we get to the confirmation window. Click finish to complete adding the user:

Confirmation Window for New User
Windows Server 2000 Screenshot

Now we see that the user, jsmith, is in the Users folder of the Active Directory realm:

Active Directory listing for ADNFSV4.LAB.NETAPP.COM realm
Windows Server 2000 Screenshot

Now we need to create a "machine" credential for the Linux NFS client. Currently, Linux 2.6 requires a credential of form:

nfs/hostname@REALM-NAME

Our host name will be scully.lab.netapp.com. The realm name is ADNFSV4.LAB.NETAPP.COM.

We start by creating yet another User principal.

You must create this principal as type User. Do NOT create the principal as type Computer. There is some dispute about this. Mario Wurzl says that he has no problem creating machine credential principals as type Computer. However, Microsoft's Kerberos Interoperability document says otherwise:

Use the Active Directory Management tool to create a new user account for the UNIX host:

Select the Users folder, right-click and select New, then choose user.

Type the name of the UNIX host.

The above passage is taken from a series steps for creating a principal of form host/hostname@REALM. We are ultimately going to create a principal of form nfs/hostname@REALM, so I contend the above excerpt from Microsoft applies. It may be the case that principals of type Computer work fine for machine credentials. I have never tried that, and absent a compelling reason, won't try it.

As we will see, this principal can be any name, but let's use a convention:

servicenameNot-fully-qualified-hostname

E.g. concatenate the service name "nfs" with the capitalized base hostname "Scully". So, our new principal will be:

nfsScully

You might be asking: "Whoa, where did this weird convention come from? Why not just call the principal ``scully''"? The issue is that you may find you need multiple machined credentials for various services. You might need host/hostname@REALM, nfs/hostname@REALM and root/hostname@REALM. You can't call the user principal for all three of these hostname. Credit goes to my old Kerberos project team at Sunfor coming up with this convention.

OK. Repeat the steps used to create principal jsmith in the Users folder for principal nfsScully.

The next step requires opening a Command Prompt window on the Windows 2000 server, and mapping nfsScully to its real machine principal,

nfs/scully.lab.netapp.com@ADNFSV4.LAB.NETAPP.COM

The command to do is ktpass, and it is invoked as:

ktpass -princ nfs/scully.lab.netapp.com@ADNFSV4.LAB.NETAPP.COM -mapuser nfsScully -pass XXXXXXXX -out UNIXscully.keytab

I have deliberately italicized the XXXXXXXX in the above to indicate that a real password needs to be provided (This password does not have to be the same as that used when user principal nfsScully was created in the Active Directory GUI. In fact, I've never used the same password for the GUI and the ktpass command. I cannot claim if this will work if the passwords are the same). You should generate password XXXXXXXX randomly, lest an attacker tries to impersonate Linux client scully. And you should be doing all this on a secure connection to the Windows 2000 server, lest an attacker packet sniff your session and grab the password. Here is a screen shot of the above example:

ktpass example

Windows Server 2000 Screenshot

You would then securely copy UNIXscully.keytab to

scully.lab.netapp.com:/etc/krb5.keytab

using a tool like scp (SSH for file copy). Note that it is possible on the Linux client to kinit to nfsScully via password XXXXXXXX. I think this is unfortunate. Machined credential passwords should use randomly generated keys that even you the system administrator don't know the password for. Randomly generate XXXXXXXX blind if possible, such via a .bat script under the Windows 2000 command shell.

Now it is time to focus attention on the Linux client.

Log onto the Linux client, and create an /etc/krb5.conf file. Here is an example:

[libdefaults]
default_realm = ADNFSV4.LAB.NETAPP.COM
default_tkt_enctypes = des-cbc-md5 ; or des-cbc-crc
default_tgs_enctypes = des-cbc-md5 ; or des-cbc-crc

[realms]
ADNFSV4.LAB.NETAPP.COM = {
kdc=ant-c0.lab.netapp.com:88
default_domain=lab.netapp.com
}

[domain_realm]
.netapp.com = ADNFSV4.LAB.NETAPP.COM
.lab.netapp.com = ADNFSV4.LAB.NETAPP.COM
.sim.netapp.com = ADNFSV4.LAB.NETAPP.COM
.adnfsv4.lab.netapp.com = ADNFSV4.LAB.NETAPP.COM

[logging]
FILE=/var/krb5/kdc.log

It is important to realize that:

The encryption type specifiers ( and default_tkt_enctypes = des-cbc-md5 ; or des-cbc-crc and default_tgs_enctypes = des-cbc-md5 ; or des-cbc-crc) cannot be omitted. Microsoft states:
Only DES-CBC-MD5 and DES-CBC-CRC encryption types are available for MIT interoperability.
The [domain_realm] section that maps DNS domain names to the Active Directory realm is critical.
Active Directory only supports upper case realms. This is the case even though the screen shots of the Windows 2000 Active Directory tree should a lower case domain name.

You want to make sure gssd is running on the Linux client:

$ ps -eaf | grep gssd
root 2587 1 0 15:37 ? 00:00:00 rpc.gssd -m

If it is not, then you will need to start gssd:

# cd /
# /etc/init.d/rpcgssd stop
# /etc/init.d/rpcgssd start

You may have to set the /etc/sysconfig/nfs file to enable Kerberized NFS. Do:

# echo "SECURE_NFS=yes" > /etc/sysconfig/nfs

That takes care of the KDC and NFS client. What of the filer?

ONTAP supports the capability of the filer to directly join an Active Directory realm without having to use the ktpass command to produce a keytab. Indeed, if you are running CIFS as well as NFS, you have joined the Active Directory realm directly as a consequence of running "cifs setup" at the filer's command line.

Prior to joining the Active Directory realm, we need to set the dns server in the filer's resolv.conf file (in the etc subdirectory of the root volume [often /vol/vol0]) to refer to the IP address of the Active Directory server. If you do not do this, the filer will be unable to resolve the Active Directory realm to the Active Directory server. This does not mean the file has to have its DNS domain name be the same as the Active Directory realm it belongs to. The example we've been working through assumes the DNS domain name and the Active Directory realm are different.

Invoke nfs setup on the filer's command line interface:

mre1> nfs setup
Enable Kerberos for NFS? y

The filer supports these types of Kerberos Key Distribution Centers (KDCs):

1 - UNIX KDC
2 - Microsoft Active Directory KDC

Enter the type of your KDC (1-2):  2
The default name of this filer will be 'MRE1'.

Do you want to modify this name? [no]:

The filer will use Windows Domain authentication.

Enter the Windows Domain for the filer []:ADNFSV4.LAB.NETAPP.COM
ADNFSV4.LAB.NETAPP.COM is a Windows 2000(tm) domain.

In order to create this filer's domain account, you must supply the name
and password of an administrator account with sufficient privilege to
add the filer to the ADNFSV4.LAB.NETAPP.COM domain.

Please enter the Windows 2000 user [Administrator@ADNFSV4.LAB.NETAPP.COM]:

Password for Administrator:

CIFS - Logged in as administrator@ADNFSV4.LAB.NETAPP.COM.
CIFS - Updating existing filer account
'cn=mre1,cn=computers,dc=adnfsv4,dc=lab,dc=netapp,dc=com'
CIFS - Connecting to domain controller.

Welcome to the ADNFSV4 (ADNFSV4.LAB.NETAPP.COM) Windows 2000(tm) domain.

Kerberos now enabled for NFS.

NFS setup complete.

If you have previously done a "cifs setup", then you won't be prompted for the realm, host name, and administrator login, because CIFS does that. Both "nfs setup" and "cifs setup" create the "nfs/mre1.sim.netapp.com" principal on the Active Directory KDC. If you go back to the Windows 2000 server, you will see an entry for MRE1 in the Computer folder under the adnfsv4.lab.netapp.com tree.

(Note that if the Active Directory KDC is running Windows 2003, "nfs setup" will ask an additional question:

Active Directory container for filer account? [cn=computers]:

Simply push the enter key).

When using Active Directory as the KDC, no krb5.keytab is created. Instead, when the mahcine account MRE1 is created in the Active Directory database, the password (randomly generated by Data ONTAP) for MRE1 is recorded in stable storage on a file in on the filer. The password for MRE1 is used to obtain service keys for CIFS and NFS, and potentially other Kerberized network services. Even if the password for administrator changes, the filer will be able to obtain service keys for CIFS and NFS.

You also need to export the volumes with the sec=krb5 or sec=krb5i (Linux currently does not support sec=krb5p.). krb5 is plain authentication, krb5i is authentication with integrity protection on the requests and responses, and krb5p is like krb5i but also encrypts the requests and responses. If using NFSv4, it is critical to note if an ancestor and descendent directory are both exported, and the descendent is exported with sec=flavorX then the ancestor must include flavorX in its list of flavors. So for example:

/vol/vol0 -sec=sys
/vol/vol0/home -sec=krb5

will break most NFSv4 clients. You will need to change this to:

/vol/vol0 -sec=sys:krb5
/vol/vol0/home -sec=krb5

At this point you should be ready to try some NFS mounts. I suggest trying the sanity test shell script listed earlier in this article and put the Linux NFS client through its paces. First you want to kinit to a user:

$ kinit jsmith
Password for jsmith@ADNFSV4.LAB.NETAPP.COM:

Then run the shell script:

$ sh sanity.sh mre1.sim /vol/vol0/home /mnt

Retries in NFSv4 Considered Harmful

2005-04-19T10:57:00.000-07:00

I've been asked lately about this text in RFC3530, which I'm responsible for:

When processing a request received over a reliable transport such as TCP, the NFS version 4 server MUST NOT silently drop the request, except if the transport connection has been broken. Given such a contract between NFS version 4 clients and servers, clients MUST NOT retry a request unless one or both of the following are true:

The transport connection has been broken

The procedure being retried is the NULL procedure

Rather than resurrect the original discussion from the NFSv4 working alias (I think it was in 2002, since that's the time when I joined Network Appliance, and rejoined the NFSv4 standards effort) and resubmitting it to the NFSv4 working alias, and then being asked about it several years from now, and doing it all over again, etc., let this blog entry serve as my words on the issue.

But is there an exception for clients that retry a request that is already in progress on the NFSv4 server? Apparently some NFSv4 servers implement this exception (which is a holdover from code inherited from NFSv3).

No there is no exception. If there was an exception, then this would contradict, or at least be inconsistent with:

clients MUST NOT retry a request

So really the question is, why shouldn't be clients allowed to retry a request over the same, unbroken connection? The answer is that TCP guarantees delivery of data. Having an application level retry above TCP defeats much of the purpose of TCP. NFSv4 supports unlimited transfer sizes. Typical NFSv4 servers allow 64 Kbyte transfers, but some support much more, including one megabyte or more. Allowing clients to retry a one megabyte, or even a 64KByte transfer over the same connection is an awful waste of resources on the client and server and on the network. And it is unnecessary because the NFSv4 server MUST never drop a request, unless the connection is broken. If you know your NFSv4 server is doing that, then you need never retry a request, unless the connection is broken.

A broken connection is the one out allowed by the NFSv4 specification to handle the case of an NFSv4 server rebooting, or to handle the case of the network partition causing a connection to timeout. The latter is necessary if the client's API to the TCP connection doesn't have any feedback telling the sender that that receiver has acknowledged receipt of data sent.

The other out allowed is retrying a NULL procedure. This is there to handle the case of an NFSv4 server crashing without sending a disconnect (in TCP, this is a FIN message) indication to the client. The client is then unaware that the connection no longer exists on the client. If the client sends a request, the server crashes, then the client will wait forever. Clients should wait a reasonable amount of time (personally, I think 60-180 seconds is reasonable, and I wrote the Solaris 2.6 NFS/TCP client with such timeouts) for a response. Then they can either break the connection on their side, and then retry, or they can send a NULL procedure "ping" to the NFSv4 server. I prefer the latter, because NULL pings don't use much resources, and if an NFSv4 server is live, it saves the cost of unneccesarily re-sending a request that results in a big transfer.

Of course, some NFSv4 client implementors might be worried that there will still be some NFSv4 servers that drop requests without a disconnected indication, or a nice NFSv4 error code like NFS4ERR_DELAY or NFS4ERR_RESOURCE. Given that there are NFSv4 servers out there that don't follow the specification with respection to retries of requests that are in progress (and I thought it was clear that servers MUST NOT do that), I guess I have to accept that some NFSv4 servers might mistakenly drop requests. So clearly, if the NULL ping does not force a disconnect, and some number of seconds later, the response to the original request has not been received, NFSv4 clients have no recourse but to disconnect, and retry.

Note that the reason NFSv4 introduces strict rules about retries over TCP is because there were no rules at all for NFSv3 over TCP. As a result some of the initial clients (and even some modern clients) had timeouts that were too short, servers would (and still do) drop requests, and client implementors really had no clear guidelines for when to retry and when to timeout. With NFSv4, we never retry on the same connection, and for the most part, rely on the timers built in to the connection-oriented transport. This seems like a better way, but that's just my opinion. :-)

retry= option to automount maps revisited

2005-03-11T09:03:00.000-08:00

My colleague, Tom Haynes, points out that there can be issues with setting retry= to high values. I've updated the blog entry on automounter tuning appropriately. Bottom line: beware of doing anything higher than retry=2 on Solaris 10 (for now anyway), or force the NFS version to vers=3.

Automounter Tuning

2005-03-10T08:26:00.000-08:00

There are two aspects to the automounter that you should pay attention to as a system administrator:

The retry count on mount attempts.
The duration of a mount.

Let's look at the retry count. When you do a manual mount, (i.e. when you use the mount command), the mount command will make a remote procedure call to the NFS server's mount daemon. If this call times out, it will try again, and again. The number of times it will try varies with each NFS client, but it is a big number. When I last looked at this, in Solaris 8, the number of retries was 10000. But the 10000 is the number of times the mount command calls the API to send a remote procedure call, usually a macro called CLNT_CALL(). Internally, CLNT_CALL() will retransmit multiple times, typically 5 times. So we are talking on the order of 50,000 attempts over the network. What this means is that by default, a mount command can take days to timeout to a dead NFS server. This is why the bg option was added to the mount command, so that the mounting NFS file systems automatically at boot time didn't prevent the system from coming up; bg stands for background and after the first mount attempt, the bg option forces the mount operation to work in the background.

There is an NFS mount option, retry, which is used to change the default number of retries. You can do:

# mount -o retry=0 filer:/vol/vol0 /mnt

In which case a single CLNT_CALL() attempt will be made to access the NFS server, and the mount will fail if the call times out. Why would you want to do such a thing? You probably wouldn't. But if you use the automounter, most likely that's exactly what you are doing. Most automounters will make one or two attempts to mount an NFS file system. That's not a very nice thing if the file system that doesn't get mounted is your home directory as you log in, or your database as your DBMS starts running. The good news is that you can override the automounter's default. Just add the option retry=1000 to your automounter maps, and you'll get much more robust automounting. The simplest is to add retry=1000 to the entries in your master automounter map file or table in NIS or LDAP. Note that the retrans option has nothing to do with mount retries. Page 98 of my book talks about the retry and retrans options.

Beware though of retry= for values higher than 2 on some versions of Solaris. My co-author for Managing NFS and NIS, Second Edition Ricardo Labiaga, who overhauled the automounter in Solaris 2.6, says that before Solaris 2.6 this was a problem, as http://www.sunhelp.org/faq/autofs.html (thanks to my colleague, Tom Haynes for the link) points out:

CAUTION:  this can "hold up" other automount requests
for 15 seconds per retry specified, on some versions of
Solaris.  Do not make this value much larger than 2!!

I found with Solaris 8, that Ricado is correct; retry=1000 works great. However, I had problems with Solaris 10. I set my master map, /etc/auto_master to:

/net -hosts -nosuid,nobrowse,retry=10000

I then put one of my NFS servers (mre1.sim) into a break point so that it would not respond. Then I did:

% ls /net/mre1.sim &

as expected, the above hanged.

Unfortunately, so did:

% ls /net/server2 &
% ls /net/server3 &

and server2 and server3 are live. Setting retry=5 wasn't very satisfying either; it took about a minute for above to complete. As a workaround, I added "vers=3" to the map options, and things work correctly.

Let's look at the duration. The automounter is also an auto-unmounter. The idea is that when NFS filesystems are no longer used, the automounter should unmount them. This a good thing, because from time to time, automounter maps are changed. If the automounter never unmounted anything, then the map updates would never be seen by the client. Ideally, the automounter would wait to attempt an unmount when it knew the file system hadn't been used for some amount of time. However, automounters don't have an interface to know if there are any processes currently with open files in the NFS file systems. As a result, the automounter has a simple minded approach: it waits some number (N) of seconds, and then attempts to unmount a file system, and does this every N seconds. If the file system is in use (busy), the unmount fails.

It turns out that an unmount attempt of a busy file system can be really bad performance-wise. An unmount attempt will flush all cached data, force all modified but unwritten blocks to be written to the NFS server, and flush all cached metadata (attributes, directories, and name cache). At the end of that, if there are still references to the filesystem, the unmount fails. This means that the processes benefiting from caching will now take latency hits as their working sets of cached data are rebuilt.

Thus, you will want to consider tuning your automount duration higher. For example, the automount command in Solaris has a -t option to set the duration to override the default of 600 seconds. You want to strike a balance between good performance and the benefits of re-synchronization with automounter map updates. If you change the location of an NFS file system no more than once a month, then setting the timeout to 86,400 seconds (24 hours) is reasonable. If you are changing things once every few days, you might find 3600 seconds is short enough; I have many years of experience with -t 3600 and vouch for it. Chapter 9 of my book goes into deep discussion of the automounter, including the -t option.

What's the deal on the 16 group id limitation in NFS?

2005-03-09T14:52:00.000-08:00

So the executive summary here is:

Use NFSv4
Use RPCSEC_GSS authentication
Use ACLs

Now I'll provide the deeper explanation for why.

NFS is built on ONC RPC (Sun RPC). NFS depends on RPC for authentication and identification of users. Most NFS deployments use an RPC authentication flavor called AUTH_SYS (originally called AUTH_UNIX, but renamed to AUTH_SYS).

AUTH_SYS sends 3 important things:

A 32 bit numeric user identifier (what you'd see in the UNIX /etc/passwd file)
A 32 bit primary numeric group identifier (ditto)
A variable length list of up to 16 32-bit numeric supplemental group identifiers (what'd you see in the /etc/group file)

So the 16 group id limit actually refers to the supplemental group identifiers, and it is specific to AUTH_SYS, not NFS. It is just that NFS (i.e. Not For Security :) has historically been deployed with AUTH_SYS. It doesn't help either that most, if not all NFS clients and servers use AUTH_SYS by default, even if they support better forms of authentication like AUTH_DH (AUTH_DES) or RPCSEC_GSS (both AUTH_DH and RPCSEC_GSS rely on cryptography to authenticate users).

It turns out that with 800 (someday I'll talk about why that limit is there) available bytes of authentication stuff in the variable length ONC RPC header for credentials and verifier, we could actually support nearly 200 supplemental group identifiers. So why don't NFS clients and servers do that?

The standard (yes, AUTH_SYS is part of an IETF standard) says 16. An NFS client that sends more is breaking the standard, and if it did send more, and the server rejected it (per the standard), what would the client do? It would have to truncate the number of supplemental group identifiers. Which 16 would it pick?
An NFS server could be forgiving and accept more than 16 supplement group identifiers, but that then begs the question as to which client is going send more given the first bullet item.

So why does the standard limit us to 16 group identifiers? The value 16 is a reflection of what UNIX operating systems supported at the time (the 1980s). Indeed, when Sun owned and controlled ONC RPC (before graciously giving IETF control), my foggy recollection (and I'm really dating myself here) is that AUTH_SYS started off with 8, then went to 12, and finally settled on 16 supplemental group identifiers. Since then, most AUTH_SYS clients and servers live in operating environments and file systems that support at least 32 supplemental group identifiers. Which is great if you don't have to use NFS to access data. Even an NFS client's operating environment supports more than 16 supplemental groups, in every case I know of, the NFS client will refuse to violate the AUTH_SYS standard and so it will not send more 16 supplemental groups. Some clients will truncate the number of supplemental groups to 16, and others will simply refuse to issue the NFS/AUTH_SYS request. So even if an NFS server wanted to be forgiving and accept AUTH_SYS requests that had more than 16 supplemental groups, this would be in vain.

So how do we get out of this?

One possible answer is to create an RPC authentication flavor like AUTH_SYS but with no limit on the number of group identifiers. The trouble is, AUTH_SYS is really bad. It isn't rocket science to exploit it. The 'Net is a much more dangerous place today than in 1980s, and so it would be unethical if IETF published an AUTH_SYS_PLUS standard. In theory, nothing prevents someone from asking IANA for a new ONC RPC flavor number, and building their own authentication flavor that does just that, and publishing it. But I think it would be unethical for vendors of NFS software to support it. But the free market often trumps ethics so we'll see if any vendor cracks first. And gee, why stop at ~200 group identifiers? Just ignore the 800 byte limitations in the ONC RPC header, and send as many as the client wants. But as we will see later, supporting nearly 200 supplemental group identifiers as other issues beyond ONC RPC and NFS.
Another way is to use a flavor like RPCSEC_GSS which doesn't send group identifiers. Instead, it lets the NFS server decide what groups the user is in (server determining access controls; what a novel concept!) based on the local /etc/group file or group tables in NIS or LDAP. Since there is no group id array in the RPC message, only internal NFS server limitations get in the way. NetApp's ONTAP server for example supports 32 supplemental group identifiers. Last I checked Solaris was either unlimited or up to 64, but it was subject to a tunable parameter. A side benefit of RPCSEC_GSS, if used over something like Kerberos V5 or public key certificates, gives you true authentication.

Does RPCSEC_GSS completely get you out of the 16 group id tangle? Not quite. As my colleague Chuck Lever pointed out to me recently, there is this side band protocol called NLM used for advisory byte range locking. I've seen just one NLM client use RPCSEC_GSS, and it wasn't Linux or Solaris. And not all NLM servers support RPCSEC_GSS. Practically speaking, this means that you have to either not use locking (for example use the llock mount option in Solaris, or use the nolock option in Linux), or you'll have to use NFS version 4.

NFS version 4 combines locking and filing (and mounting) in one single protocol. So use NFS version 4 with RPCSEC_GSS to blast past the 16 group identifier limitation.

Some caveats:

Most people use a directory service like NIS or LDAP to store their supplemental group identifier information. If you establish more than 16 supplemental groups in NIS or LDAP for your users, you'll want to make sure that all your other NFS clients support NFSv4 and support RPCSEC_GSS, and of course are configured to use Kerberos V5.
For a similar reason, make sure your NFS clients can support more than 16 group identifiers per user. When a user logs into his desktop system, the operating system will establish his credentials. If the user is in more than 16 groups, he may well be denied login access if his home directory is NFS mounted.

So you might ask, this is great but why am I limited to 32 or 64 group identifiers? The reason relates to how operating systems set up their in-kernel credentials. Usually the supplemental group identifiers are a simple array of integers. This means that each access attempt can require searching the entire array of integers. This is one thing if the array is 16-64 group identifiers, but get into 100s to 1000s or more, and the performance impact of that many group identifiers might start to get in the way. An answer might be to organize in-kernels as hash tables or trees, but this has costs too. Not to mention that as each in-kernel credential gets bigger the impact on kernel memory usage, which takes away from applications, becomes important.

Another approach to consider is ACLs. NFSv4 has them. An ACL (Access Control List) is a list of ACEs (Access Control Entries). In NFSv4 an ACE is basically:

user name or group name
permission bits
whether the named user or group is being denied or allowed access

How does this solve the problem that lots of groups solves? For a given file, you can list a bunch of users that are allowed access, and there is no over the network specification that limits how many user ACEs you can have in an ACL. The limits are purely on the server. So for a given set of files, you can let lots of users and lots of different sets of users access each file. Compare that to what lots of supplemental groups do for you. Each file has a single group id assigned to it, and you can then assign a lot of users to the group id in /etc/group or the group table in NIS or LDAP. You can assign a different group id to each file. So for a set of files, you can grant access to lots of users, and lots of different sets of users. Semantically the same.

So what ACLs do for the NFS community is make extended access purely a server problem in terms of flexibility and performance. Of course, there needs to be away to edit the ACLs on a given file, which is what NFSv4 does for you.