Wednesday, June 08, 2005

Using Active Directory as your KDC for NFS

Recently I've been asked how to use Active Directory as the Key Distribution Center (KDC) for NFS, especially when used with NetApp filers and Linux 2.6 clients.

At the theoretical level, I've always know this was possible. I've used Solaris 10 NFSv[234] clients with filers configured to use Active Directory. I've used CITI's early access NFSv3 w/
Kerberos V5 authentication stuff for Linux 2.4
with filers using Active Directory. And of course, back in my Sun days, I led the team that proved NFS clients and servers could authenticate via Active Directory, work which to this day is the best documented example of how to do so.

But now that Linux 2.6 with NFSv4 and NFS/Kerberos V5 authentication is getting more real, does this still work, and if so, with all 3 NFS versions? It is a reasonable question, since Linux 2.6 continues to change.

I'm happy to report that with Windows 2000 (and 2003!) as the KDC, Fedora Core 3 (Linux 2.6.11-1.27_FC3) as the NFS client, and Data ONTAP 7.0.0.1 as the NFSv4 server, the answer is yes, at least as measured by this trivial sanity checking script:
#!/bin/sh
# NFS/Kerberos sanity.sh for Linux 2.6

if [ $# -lt 3 ]
then
echo Usage: $0 server_name server_export mount_point
echo example:
echo " " $0 mre1.sim /vol/vol0/home /mnt
exit 1
fi

size=1m
file=$size.$$.`uname -n`
echo file = $file
serv=$1
fs=$2
mnt=$3

cd /
sudo umount -f $mnt

for proto in tcp udp ;
do
case $proto in
udp )
moreopts=",rsize=4096,wsize=4096"
;;
* )
moreopts=""
;;
esac

for vers in 2 3 4 ;
do
if [ $proto = udp ] && [ $vers = 4 ]
then
echo NFSv4 is not supported over udp
else
for sec in sys krb5 krb5i ; # krb5p ;
do
echo ----------------------------------------
case $vers in
4 )
opts="-t nfs4 -o proto=$proto,sec=${sec}$moreopts"
;;
* )
opts="-o vers=$vers,proto=$proto,sec=${sec}$moreopts"
;;
esac


if sudo mount $opts $serv:$fs $mnt ;
then
cd $mnt
mount | grep -w $mnt
rm -f $file
if time dd if=/dev/zero of=$file bs=1024 count=1024 ;
then
echo $opts PASS
rm -f $file
else
echo $opts FAIL
exit 1
fi
else
echo sudo mount $opts failed. FAIL
exit 1
fi

cd /
sudo umount -f $mnt
done
fi
done
done

But before one runs this script, some configuration on the KDC, the Linux client, and the ONTAP filer are necessary.

Let's look at the KDC.

I am assuming that an Active Directory realm has been created. My example uses ADNFSV4.LAB.NETAPP.COM as the Kerberos realm.

The first thing we need to create are users. Let's walk through an example for creating a user named jsmith. First thing we do is highlight the Users folder in Active Directory:

Highlight Users Folder
Windows Server 2000 Screenshot




Having done that, right click in the folder to pop up the action menu for the folder:


Pop Up Action Menu for Users
Windows Server 2000 Screenshot




Pick the New --> User option. Now we fill in the information. I find that the First name, Full name, and User login name have to agree with each other, but you may have a different experience:


Fill in Information for New User
Windows Server 2000 Screenshot



Now click next to get to the password setting window:


Password for New User
Windows Server 2000 Screenshot




Finally, we get to the confirmation window. Click finish to complete adding the user:



Confirmation Window for New User
Windows Server 2000 Screenshot




Now we see that the user, jsmith, is in the Users folder of the Active Directory realm:


Active Directory listing for ADNFSV4.LAB.NETAPP.COM realm
Windows Server 2000 Screenshot



Now we need to create a "machine" credential for the Linux NFS client. Currently, Linux 2.6 requires a credential of form:
nfs/hostname@REALM-NAME
Our host name will be scully.lab.netapp.com. The realm name is ADNFSV4.LAB.NETAPP.COM.

We start by creating yet another User principal.

You must create this principal as type User. Do NOT create the principal as type Computer. There is some dispute about this. Mario Wurzl says that he has no problem creating machine credential principals as type Computer. However, Microsoft's Kerberos Interoperability document says otherwise:
Use the Active Directory Management tool to create a new user account for the UNIX host:
  • Select the Users folder, right-click and select New, then choose user.
  • Type the name of the UNIX host.
The above passage is taken from a series steps for creating a principal of form host/hostname@REALM. We are ultimately going to create a principal of form nfs/hostname@REALM, so I contend the above excerpt from Microsoft applies. It may be the case that principals of type Computer work fine for machine credentials. I have never tried that, and absent a compelling reason, won't try it.

As we will see, this principal can be any name, but let's use a convention:
servicenameNot-fully-qualified-hostname
E.g. concatenate the service name "nfs" with the capitalized base hostname "Scully". So, our new principal will be:
nfsScully
You might be asking: "Whoa, where did this weird convention come from? Why not just call the principal ``scully''"? The issue is that you may find you need multiple machined credentials for various services. You might need host/hostname@REALM, nfs/hostname@REALM and root/hostname@REALM. You can't call the user principal for all three of these hostname. Credit goes to my old Kerberos project team at Sunfor coming up with this convention.

OK. Repeat the steps used to create principal jsmith in the Users folder for principal nfsScully.

The next step requires opening a Command Prompt window on the Windows 2000 server, and mapping nfsScully to its real machine principal,
nfs/scully.lab.netapp.com@ADNFSV4.LAB.NETAPP.COM
The command to do is ktpass, and it is invoked as:
ktpass -princ nfs/scully.lab.netapp.com@ADNFSV4.LAB.NETAPP.COM -mapuser nfsScully -pass XXXXXXXX -out UNIXscully.keytab
I have deliberately italicized the XXXXXXXX in the above to indicate that a real password needs to be provided (This password does not have to be the same as that used when user principal nfsScully was created in the Active Directory GUI. In fact, I've never used the same password for the GUI and the ktpass command. I cannot claim if this will work if the passwords are the same). You should generate password XXXXXXXX randomly, lest an attacker tries to impersonate Linux client scully. And you should be doing all this on a secure connection to the Windows 2000 server, lest an attacker packet sniff your session and grab the password. Here is a screen shot of the above example:


ktpass example
Windows Server 2000 Screenshot

You would then securely copy UNIXscully.keytab to
scully.lab.netapp.com:/etc/krb5.keytab
using a tool like scp (SSH for file copy). Note that it is possible on the Linux client to kinit to nfsScully via password XXXXXXXX. I think this is unfortunate. Machined credential passwords should use randomly generated keys that even you the system administrator don't know the password for. Randomly generate XXXXXXXX blind if possible, such via a .bat script under the Windows 2000 command shell.

Now it is time to focus attention on the Linux client.

Log onto the Linux client, and create an /etc/krb5.conf file. Here is an example:
[libdefaults]
default_realm = ADNFSV4.LAB.NETAPP.COM
default_tkt_enctypes = des-cbc-md5 ; or des-cbc-crc
default_tgs_enctypes = des-cbc-md5 ; or des-cbc-crc

[realms]
ADNFSV4.LAB.NETAPP.COM = {
kdc=ant-c0.lab.netapp.com:88
default_domain=lab.netapp.com
}

[domain_realm]
.netapp.com = ADNFSV4.LAB.NETAPP.COM
.lab.netapp.com = ADNFSV4.LAB.NETAPP.COM
.sim.netapp.com = ADNFSV4.LAB.NETAPP.COM
.adnfsv4.lab.netapp.com = ADNFSV4.LAB.NETAPP.COM

[logging]
FILE=/var/krb5/kdc.log
It is important to realize that:
  • The encryption type specifiers ( and default_tkt_enctypes = des-cbc-md5 ; or des-cbc-crc and default_tgs_enctypes = des-cbc-md5 ; or des-cbc-crc) cannot be omitted. Microsoft states:
    Only DES-CBC-MD5 and DES-CBC-CRC encryption types are available for MIT interoperability.
  • The [domain_realm] section that maps DNS domain names to the Active Directory realm is critical.
  • Active Directory only supports upper case realms. This is the case even though the screen shots of the Windows 2000 Active Directory tree should a lower case domain name.
You want to make sure gssd is running on the Linux client:
$ ps -eaf | grep gssd
root 2587 1 0 15:37 ? 00:00:00 rpc.gssd -m
If it is not, then you will need to start gssd:
# cd /
# /etc/init.d/rpcgssd stop

# /etc/init.d/rpcgssd start
You may have to set the /etc/sysconfig/nfs file to enable Kerberized NFS. Do:
# echo "SECURE_NFS=yes" > /etc/sysconfig/nfs
That takes care of the KDC and NFS client. What of the filer?

ONTAP supports the capability of the filer to directly join an Active Directory realm without having to use the ktpass command to produce a keytab. Indeed, if you are running CIFS as well as NFS, you have joined the Active Directory realm directly as a consequence of running "cifs setup" at the filer's command line.

Prior to joining the Active Directory realm, we need to set the dns server in the filer's resolv.conf file (in the etc subdirectory of the root volume [often /vol/vol0]) to refer to the IP address of the Active Directory server. If you do not do this, the filer will be unable to resolve the Active Directory realm to the Active Directory server. This does not mean the file has to have its DNS domain name be the same as the Active Directory realm it belongs to. The example we've been working through assumes the DNS domain name and the Active Directory realm are different.

Invoke nfs setup on the filer's command line interface:
mre1> nfs setup
Enable Kerberos for NFS?
y

The filer supports these types of Kerberos Key Distribution Centers (KDCs):

1 - UNIX KDC
2 - Microsoft Active Directory KDC

Enter the type of your KDC (1-2):
2
The default name of this filer will be 'MRE1'.

Do you want to modify this name? [no]:

The filer will use Windows Domain authentication.

Enter the Windows Domain for the filer []:
ADNFSV4.LAB.NETAPP.COM
ADNFSV4.LAB.NETAPP.COM is a Windows 2000(tm) domain.

In order to create this filer's domain account, you must supply the name
and password of an administrator account with sufficient privilege to
add the filer to the ADNFSV4.LAB.NETAPP.COM domain.

Please enter the Windows 2000 user [Administrator@ADNFSV4.LAB.NETAPP.COM]:

Password for Administrator:

CIFS - Logged in as administrator@ADNFSV4.LAB.NETAPP.COM.
CIFS - Updating existing filer account
'cn=mre1,cn=computers,dc=adnfsv4,dc=lab,dc=netapp,dc=com'
CIFS - Connecting to domain controller.

Welcome to the ADNFSV4 (ADNFSV4.LAB.NETAPP.COM) Windows 2000(tm) domain.

Kerberos now enabled for NFS.

NFS setup complete.

If you have previously done a "cifs setup", then you won't be prompted for the realm, host name, and administrator login, because CIFS does that. Both "nfs setup" and "cifs setup" create the "nfs/mre1.sim.netapp.com" principal on the Active Directory KDC. If you go back to the Windows 2000 server, you will see an entry for MRE1 in the Computer folder under the adnfsv4.lab.netapp.com tree.

(Note that if the Active Directory KDC is running Windows 2003, "nfs setup" will ask an additional question:
Active Directory container for filer account? [cn=computers]:
Simply push the enter key).

When using Active Directory as the KDC, no krb5.keytab is created. Instead, when the mahcine account MRE1 is created in the Active Directory database, the password (randomly generated by Data ONTAP) for MRE1 is recorded in stable storage on a file in on the filer. The password for MRE1 is used to obtain service keys for CIFS and NFS, and potentially other Kerberized network services. Even if the password for administrator changes, the filer will be able to obtain service keys for CIFS and NFS.

You also need to export the volumes with the sec=krb5 or sec=krb5i (Linux currently does not support sec=krb5p.). krb5 is plain authentication, krb5i is authentication with integrity protection on the requests and responses, and krb5p is like krb5i but also encrypts the requests and responses. If using NFSv4, it is critical to note if an ancestor and descendent directory are both exported, and the descendent is exported with sec=flavorX then the ancestor must include flavorX in its list of flavors. So for example:
/vol/vol0 -sec=sys
/vol/vol0/home -sec=krb5
will break most NFSv4 clients. You will need to change this to:
/vol/vol0 -sec=sys:krb5
/vol/vol0/home -sec=krb5
At this point you should be ready to try some NFS mounts. I suggest trying the sanity test shell script listed earlier in this article and put the Linux NFS client through its paces. First you want to kinit to a user:
$ kinit jsmith
Password for jsmith@ADNFSV4.LAB.NETAPP.COM:
Then run the shell script:
$ sh sanity.sh mre1.sim /vol/vol0/home /mnt

14 Comments:

Anonymous Anonymous said...

Could you please make available the referenced pages which were previously reachable under http://www.connectathon.org/seam1.0/ -- they are not there any more, yet look very promising according to how you refer to them!

thx, dietmar.berg

Saturday, July 01, 2006 6:15:00 AM  
Blogger Mike Eisler said...

Dietmar,

Thanks for pointing that out.
I've asked the owners of the
connectathon.org web site to
look into that, and they've
put the seam1.0 interoperability
material back online.

Friday, July 14, 2006 11:07:00 PM  
Anonymous Anonymous said...

Appreciate if you could provide insight into real life deployement of kerb5 with v3. It is my understanding the majority use basic kerb5, eschewing kerb5i/5p (for obvious reasons!).

What's your take on market transition to v4 versus extending v3?

Wednesday, September 13, 2006 4:17:00 PM  
Blogger Mike Eisler said...

Yes the majority of NFS/Kerberos V5
use does not use krb5i or krb5p.
Line rate (10Giga bit/sec) I/O
with integrity or encryption is
a hard problem, and solving it
economically is harder. Four
years ago, I was assured by
various TOE (TCP Offload Engine)
vendors, that inline IPsec
acceleration would be ubiquitous by
now, and so I proposed the
CCM security mechanism that would
link NFS/RPC-level Kerberos
authentication with IPsec protection. But I find that contrary
to promises, my brand new laptop
doesn't have an IPsec offload engine
on it, and alas, my Wifi is still
going in the clear except when using
NetApp's VPN.

Regarding market transition; the
transition will happen when the
Linux NFSv4 client is fully supported in the enterprise
distributions of Linux, and the
compelling features of NFSv4
(referals and delegations) are available
on the client and enough servers
to make the switch. Interest
is picking up. NFSv4.1, which
adds parallel NFS, should be
a no brainer for those with
very high performance requirements
for single file I/O.

Thursday, September 14, 2006 5:57:00 AM  
Blogger Mike Eisler said...

This comment has been removed by a blog administrator.

Thursday, September 21, 2006 7:52:00 AM  
Anonymous Anonymous said...

Hi Mike,

According to you, we have to get the name and password of the administrator of the domain - 'ADNFSV4.LAB.NETAPP'. Is it right?

However, what if we don't know it? Can we force NetApp to populate a krb5.keytab file into disk?

Thanks.

Thursday, September 21, 2006 7:48:00 PM  
Blogger Mike Eisler said...

we have to get the name and password of the administrator of the domain - 'ADNFSV4.LAB.NETAPP'.

Either "administrator" or a principal with rights to access the AD controller's KDC and rights to create service principals. Note this is the same requirement as when you set up CIFS on the filer.

However, what if we don't know it? Can we force NetApp to populate a krb5.keytab file into disk?

If you don't know it, then you need to have a discussion with your KDC administrator. Note that you need to know the password of the domain controller in order to create keytabs for the Linux NFS client.

Regardless, if you don't need to run CIFS, you can use file based keytabs (select the UNIX KDC when you run nfs setup). (If CIFS is running you must use AD as you KDC. If CIFS is not running you may use UNIX as your KDC.) But again you need access to the domain controller just as you had access to the domain controller to create the Linux keytab files.

Thursday, September 21, 2006 8:44:00 PM  
Blogger Mike Eisler said...

I was asked another question about what happens if the administrator password changes; how will the filer obtain CIFS and NFS service keys? An earlier comment I made, which I've deleted because it added confusion, implied that the administator password would be recorded in stable storage on the filer, and this administrator account was used to obtain the service keys.

Instead, what happens is that a machine account ("MRE1" in my example) is created on the Active Directory KDC. The password (randomly generated) for "MRE1" is recorded on the filer's stable storage. This password is then used to obtain CIFS and NFS service keys, instead of creating a krb5.keytab file.

I've updated the blog article to clarify this. I regret the confusion, but I thank "anonymous" for driving toward absolute clarity.

Monday, September 25, 2006 10:36:00 AM  
Anonymous Anonymous said...

I've used Solaris 10 NFSv[234] clients with filers configured to use Active Directory.

Do you happen to have a procedure written up somewhere for this? I've been working on it and am specifically having issues with NFSv4 and krb5 auth in combo. NFSv4 or krb5 work fine individually, but both together just get "Permission denied" errors sans log entry on the filer. Any help you could offer would be appreciated.

Tuesday, December 05, 2006 9:32:00 AM  
Anonymous Anonymous said...

I just wonder how to create keytab for NFS server side wiutout use "nfs seup" utility.

Could I generated keytab for NFS server using same procedures as NFS client?

Thanks!

Jeff.

Thursday, May 22, 2008 10:28:00 AM  
Blogger Mike Eisler said...

When you select a UNIX KDC via NFS setup, you then copy the krb5.conf file you are using for your UNIX/Linux client to the FAS server's /etc directory. Then create an
"nfs/fas-server-name" principal on the KDC. Then extract the principal into a keytab and copy it over to the the FAS server's /etc/krb5.keytab file.

Thursday, May 22, 2008 10:50:00 PM  
Anonymous Anonymous said...

Just wondering if you have had any luck with allowing users in a child domain (or parent, either way) being able to authenticate with Solaris or Linux?

In other words, I can login with a user from the same domain that is my default, but if I try to use a child domain (even with the realms setup and whatnot in the krb5.conf) I cannot login. I can kinit user@SUB.DOMAIN.COM but cannot login with uer from that domain, and if I try login with user@SUB.DOMAIN.COM the user doesn't exist in the kerberos database.

If you have, or have any ideas, they would be greatly appreciated.

Thanks!

Tuesday, October 07, 2008 9:00:00 AM  
Blogger Unknown said...

I am trying to "Using Active Directory as your KDC for NFS".

I use 2 RHEL-5 box as my nfs server / client. With samba, I have added both of the two into my AD domain. And ssh with GSS ticket passing is working well.

The problem is that rpc.gssd rejects my keytab:
#] tail -f /var/log/message
Nov 1 00:17:02 xen-60562d rpc.gssd[9638]: Processing keytab entry for principal 'XEN-60562D$@MYREALM.COM'
Nov 1 00:17:02 xen-60562d rpc.gssd[9638]: We will NOT use this entry (XEN-60562D$@MYREALM.COM)
Nov 1 00:17:02 xen-60562d rpc.gssd[9638]: Processing keytab entry for principal 'nfs/xen-60562d.MYREALM.COM@MYREALM.COM'
Nov 1 00:17:02 xen-60562d rpc.gssd[9638]: We will use this entry (nfs/xen-60562d.MYREALM.COM@MYREALM.COM)
Nov 1 00:17:02 xen-60562d rpc.gssd[9638]: WARNING: Key table entry not found while getting initial ticket for principal 'nfs/xen-60562d.MYREALM.COM@MYREALM.COM' from keytab 'FILE:/etc/krb5.keytab'
Nov 1 00:17:02 xen-60562d rpc.gssd[9638]: ERROR: No usable machine credentials obtained

I have no idea just now.

Friday, October 31, 2008 9:32:00 AM  
Anonymous Anonymous said...

Actually we found a way to make cross domain authentication work. If you use the upn as the uid mapping instead of the sAMAccountName then you can login to any domain using user@DOMAIN.

It's not perfect but at least it works.

Tuesday, November 11, 2008 9:15:00 AM  

Post a Comment

<< Home