Solving the Vault secure introduction problem with Kerberos

Why?

Vault is a great product, however, with any service, thought needs to go into how it securely integrates with your environment. One aspect of Hashicorp Vault is how do you ensure that hosts attempting logon are trusted? This is subtly different to interactive login to Vault which is performed via authentication methods like LDAP or Token. Think of it as single sign on for hosts, applications and services.

If you are working in a cloud environment, you will likely have the ability to integrate with the cloud's built in trust mechanism, for example AWS IAM. AWS EC2 hosts and Lambdas integrate with IAM automatically. Therefore using the aws authentication plugin for Vault, allows AWS service to assert their identity to Vault in a trustworthy manner.

However, there is at least one exception - AWS Workspaces. In AWS's wisdom, Workspaces do not integrate with IAM. That means that they cannot out of the box connect to AWS services at an instance level, be it for CloudWatch logging or access to S3. The ask is, how can we issue temporary IAM keys to Workspaces, providing functionality similar to the instance keys issued via the EC2 metadata service?

The solution? AWS Workspaces are always joined to a Kerberos based authentication system (be it Managed AD, Simple AD or Directory Connector). Kerberos has the concept of authenticating hosts aswell as users - this is what underpins the majority of single sign on that you see in Active Directory environments. As of Vault 1.4, Kerberos authentication has been a supported mechanism to authenticate to Vault. Once a host is logged into Vault, it can then use the AWS IAM secrets engine to issue a temporary keypair. Simple! (ish)

What is Kerberos

Kerberos is an open protocol that allows networked resources and users to authenticate with eachother. Kerberos uses a trusted intermediatary between principals. Principals can be human users, hosts and services. All sounds pretty abstract, but you're likely using this technology day-to-day. Microsoft Active Directory uses Kerberos along with LDAP to underpin domain authentication. The trusted intermediatary (KDC) in this scenario between clients and servers are Domain Controllers.

Being an open protocol, that pre-dates Active Directory, non-Windows hosts and non-Microsoft services can integrate with Active Directory with relative ease.

By using a Kerberos service account for Vault, clients (in this scenario, Linux Hosts) can authenticate to Vault using Kerberos. The process of joining an Active Directory domain in the background ensures that the KDC and the prinncipal both have a record of their password (yes hosts have their own password). The fundamental flow of Kerberos is as follows;

1. A client creates a TGT request, encrypting a timestamp with it's key.
2. The KDC validates the request - as it know's the client's password, it can decrypt the message. The KDC responds with a TGT response
3. This token can then be presented to the KDC to request acccess to services on the Kerberos network.
4. The KDC responds with a payload that is encrypted using the target service's key,
5. The client can't open this payload, but when presented to the target service, it proves that the client is authenticated to do so,
6. Upon validation, the service responds with a token that can then be used for authenticated communication between client and server.

This process once understood is your best way to troubleshoot issues down the line. If you have access to wireshark this would be a great time to dust it off - the exchange above can clearly be seen.

In the context of this setup, Workspaces will attempt to authenticate with Vault, with AD authenticating the transaction.

Vault Setup

Pre-requsites

All that is required is an Active Directory user to act as a service account for Vault, creation of an SPN and the generation of a Kerberos keytab on the Vault server. Vault's Kerberos functionality depends on using a keytab to generate a valid TGT, even if running on a Windows host. This is down to the use of golang's kerberos5 library, instead of depending on Operating System specific libraries, for example kerberos.dll. If on Windows you still need to create a keytab.

Configuration

I have covered setting up Vault end-to-end in great detail before - Issuing RDP Certificates using Vault, so won't cover it in massive detail here.

Deploying this codebase will incur costs, it is not entirely free-tier elegible.

In summary, the deployment shall comprise;

• a Managed AD Deployment for User Authentication,
• a Windows EC2 instance for managing Active Directory,
• a Workspace,
• a Vault Server instance.

Managed AD and Workspaces are required to form the Workspaces solution. You may not have a need for Active Directory, and Simple AD may be a better fit.

Incase you haven't used Workspaces, it is a managed desktop from AWS. When you create a workspace it is automatically joined to your directory using Samba, to Active Directory. This joining process (like a Windows host), causes the host itself to have a password, aswell as any user who logs on to that host.

The server side comprises of a standard Vault Server installation, with the Kerberos Authentication plugin enabled and configured. This comes as part of the standard Vault installation, no extra downloads required! The Vault Server host doesn't need to be joined to the domain itself, all that is required is the creation of a valid keytab based on the service account created in Active Directory.

Terraform Deployment

Copycopy code to clipboard
1DEMO_HOME=~/facl-demos/
2mkdir -p ${DEMO_HOME}
3cd ${DEMO_HOME}
4git clone https://gitlab.com/fluffy-clouds-and-lines/aws-workspaces-kerberos-vault-login.git
5cd ${DEMO_HOME}/aws-workspaces-kerberos-vault-login/demo_environment
6export TF_VAR_remote_source_ip="$(dig +short myip.opendns.com @resolver1.opendns.com)/32"
7terraform init
8terraform plan
9terraform apply

Service Account Creation

AWS Managed AD, although a Managed Service, you cannot perform any administration (except password resets) via the AWS Console. For this you will need to deploy a Windows EC2 instance, join it to the domain and install the AD Management Tools. AWS has good documentation on how to join your EC2 instance to AD (use seamless join if you can!) and install the AD Management Tools https://docs.aws.amazon.com/directoryservice/latest/admin-guide/ms_ad_manage_users_groups.html.

Once logged into your AD Management Host, use PowerShell to create your use and set the SPN;

Copycopy code to clipboard
1Import-Module ActiveDirectory
2# Create the service account
3New-ADUser svc-vault -Path '<AD Distinguished Name>' -AccountPassword(Read-Host -AsSecureString "Provide service account password") -Enabled $true
4Set-ADUser svc-vault -Replace @{"msDS-SupportedEncryptionTypes"=($encTypes -bor $AES256)}
5# Get the KVNO for Vault Server config step. Each time the password changes, the KVNO will change and the Vault keytab will need to be regenerated
6Get-ADUser svc-vault -property msDS-KeyVersionNumber
7Set-ADComputer -Identity svc-vault -ServicePrincipalNames -ServicePrincipalNames @{Add='HTTP/myvault.example.com:8200'}

Once this is done, note your service account username, password and KVNO.

Vault Server Configuration

Vault is configured using a script run as part of the userdata for the single node Vault instance in the demo. Your Vault environment may be managed differently, i.e using the Vault Terraform Provider.

This configuration configures the Kerberos auth engine and the AWS secrets engine, mapping them together based on the computer's AD group membership.

Workspace Configuration

Running Vault Agent

Vault Agent may be new to some, it is a method to run the Vault client as a daemon to automatically manage the lifecycle of secrets and certificates managed by Vault. In this scenario, Vault Agent is used to logon to Vault using the host's kerberos account, request an IAM keypair and save them to a file.

Configuring Workspaces

Unfortunately Workspaces doesn't support configuration via userdata or automation, so take a look at demo_environment/workspaces/ for the configuration to be applied. This comprises;

• A systemctl service unit,
• A secrets template - this is used by Vault agent to request a secret and save it to a file,
• A post render script which is used to read the secret that has been saved to file, process it and trigger any system changes i.e restarting a service.

Wrapup

There is a lot of moving pieces here, but it is pretty elegant solution to this problem. Documentation of Kerberos with Vault is pretty patchy, hopefully this fills in some of the gaps!