Using Vault as an SSH certificate authority

Install Vault

wget https://releases.hashicorp.com/vault/1.6.2/vault_1.6.2_linux_amd64.zipunzip vault_1.6.2_linux_amd64.zipmv vault /usr/local/sbin/vaultchown 0:0 /usr/local/sbin/vault
useradd -r -s /bin/false -d /var/lib/vault -M vaultmkdir /var/lib/vaultchown vault:vault /var/lib/vaultchmod 700 /var/lib/vaultmkdir /etc/vault
storage "file" {
path = "/var/lib/vault"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = "true"
#tls_cert_file = ""
#tls_key_file = ""
}
cluster_addr = "http://10.12.255.56:8201"
api_addr = "http://10.12.255.56:8200"
ui = "true"
# The following setting is not recommended, but you may need
# it when running in an unprivileged lxd container
disable_mlock="true"
[Unit]
Description=Vault secret store
Documentation=https://vaultproject.io/docs/
After=network.target
ConditionFileNotEmpty=/etc/vault/vault-conf.hcl
[Service]
User=vault
ExecStart=/usr/local/sbin/vault server -config=/etc/vault/vault-conf.hcl
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl enable --now vault
systemctl status vault

Initialize Vault

vault operator init
Error initializing: Put "https://127.0.0.1:8200/v1/sys/init": http: server gave HTTP response to HTTPS client
VAULT_ADDR=http://127.0.0.1:8200
export VAULT_ADDR
source /etc/profile.d/vault.sh
vault operator init
Unseal Key 1: XJLDnI09GSN34rea8etz+naMVq4oqWLrCIGasgRI9fNV
Unseal Key 2: 4/qwf+opE4s3arYgqhSedAt1DMLUeMyJlwZUHQiUlvOP
Unseal Key 3: +gijI0wzWbTngMVKAWEtwKb+VyUk5eXsMnPjiUQG9TSf
Unseal Key 4: T/tI+AyRpNIVr1N4SEljwSq5PXgqVkntYL4I2vQB4szm
Unseal Key 5: Gpv8QdsNBOMTMaxDEz4zGdzO7E8fuWlK9bWAuR0Qt9Re
Initial Root Token: s.zHYKc0I5p2kSGBaLlmOVtrgYVault initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above. When the Vault is re-sealed, restarted, or stopped, you must supply at least 3 of these keys to unseal it before it can start servicing requests.Vault does not store the generated master key. Without at least 3 key to reconstruct the master key, Vault will remain permanently sealed!It is possible to generate new unseal keys, provided you have a quorum of existing unseal keys shares. See "vault operator rekey" for more information.
#

Unseal and login

vault operator unseal
Unseal Key (will be hidden): <enter one share>
vault operator unseal
Unseal Key (will be hidden): <enter another share>
vault operator unseal
Unseal Key (will be hidden): <enter another share>
...
Sealed false
...
vault login
Token (will be hidden): <enter the root token>
Success! You are now authenticated.

Create the SSH CA

vault secrets enable -path=ssh-user-ca ssh
Success! Enabled the ssh secrets engine at: ssh-user-ca/
vault write ssh-user-ca/config/ca generate_signing_key=true
Key Value
--- -----
public_key ssh-rsa AAAAB3N.....
vault read -field=public_key ssh-user-ca/config/ca

Create signing role

vault write ssh-user-ca/roles/ssh-test - <<"EOH"
{
"algorithm_signer": "rsa-sha2-256",
"allow_user_certificates": true,
"allowed_users": "test",
"default_extensions": {
"permit-pty": ""
},
"key_type": "ca",
"max_ttl": "12h",
"ttl": "12h"
}
EOH
vault write -field=signed_key ssh-user-ca/sign/ssh-test public_key=@$HOME/.ssh/id_rsa.pub >empty.certcat empty.cert
ssh-rsa-cert-v01@openssh.com AAAAHHNza....
ssh-keygen -Lf empty.cert
empty.cert:
Type: ssh-rsa-cert-v01@openssh.com user certificate
Public key: RSA-CERT SHA256:mVV81....
Signing CA: RSA SHA256:nqMqs.... (using rsa-sha2-256)
Key ID: "vault-root-99557c...."
Serial: 2810952009944311352
Valid: from 2021-02-22T14:46:06 to 2021-02-23T02:46:36
Principals: (none)
Critical Options: (none)
Extensions:
permit-pty
vault write -field=signed_key ssh-user-ca/sign/ssh-test public_key=@$HOME/.ssh/id_rsa.pub valid_principals="test" >test.certssh-keygen -Lf test.cert
test.cert:
Type: ssh-rsa-cert-v01@openssh.com user certificate
Public key: RSA-CERT SHA256:mVV81....
Signing CA: RSA SHA256:nqMqs.... (using rsa-sha2-256)
Key ID: "vault-root-99557c...."
Serial: 10087169145372651617
Valid: from 2021-02-22T14:47:42 to 2021-02-23T02:48:12
Principals:
test
Critical Options: (none)
Extensions:
permit-pty
vault write -field=signed_key ssh-user-ca/sign/ssh-test public_key=@$HOME/.ssh/authorized_keys valid_principals="foo"
Error writing data to ssh-user-ca/sign/ssh-test: Error making API request.
URL: PUT http://127.0.0.1:8200/v1/ssh-user-ca/sign/ssh-test
Code: 400. Errors:
* foo is not a valid value for valid_principals

Signing role subtleties

CASignatureAlgorithms ^ssh-rsa

Login with the certificate

useradd -m test -s /bin/bash
TrustedUserCAKeys /etc/ssh/ssh_ca.pub
vault read -field=public_key ssh-user-ca/config/ca >/etc/ssh/ssh_ca.pub
ssh test@vault1
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 4.15.0-128-generic x86_64)
...
test@vault1:~$

Debugging login problems

/usr/sbin/sshd -p 99 -d
ssh -p 99 -vvv test@vault1
check_host_cert: certificate signature algorithm ssh-rsa: signature algorithm not supported

Avoiding the Confused Deputy

  "allowed_users": "*",

SSH roles

vault write ssh-user-ca/roles/ssh-user - <<"EOH"
{
"algorithm_signer": "rsa-sha2-256",
"allow_user_certificates": true,
"allowed_users": "{{identity.entity.metadata.ssh_username}}",
"allowed_users_template": true,
"allowed_extensions": "permit-pty,permit-agent-forwarding,permit-X11-forwarding",
"default_extensions": {
"permit-pty": "",
"permit-agent-forwarding": ""
},
"key_type": "ca",
"max_ttl": "12h",
"ttl": "12h"
}
EOH
vault write ssh-user-ca/roles/ssh-admin - <<"EOH"
{
"algorithm_signer": "rsa-sha2-256",
"allow_user_certificates": true,
"allowed_users": "root,{{identity.entity.metadata.ssh_username}}",
"allowed_users_template": true,
"allowed_extensions": "permit-pty,permit-agent-forwarding,permit-X11-forwarding,permit-port-forwarding",
"default_extensions": {
"permit-pty": "",
"permit-agent-forwarding": ""
},
"key_type": "ca",
"max_ttl": "12h",
"ttl": "12h"
}
EOH

Policies

vault policy write ssh-user - <<"EOH"
path "ssh-user-ca/sign/ssh-user" {
capabilities = ["update"]
denied_parameters = {
"key_id" = []
}
EOH
vault policy write ssh-admin - <<"EOH"
path "ssh-user-ca/sign/ssh-admin" {
capabilities = ["update"]
denied_parameters = {
"key_id" = []
}
}
EOH

Identity management part 1: userpass

vault auth enable userpassvault write auth/userpass/users/alice password=tcpip123vault write auth/userpass/users/bob password=xyzzyvault list auth/userpass/users
Keys
----
alice
bob
#
vault login -method=userpass username=alice password=tcpip123
Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token.
Key Value
--- -----
token s.3EU0uqq5CbVdP1nGPmlssj8M
token_accessor ZZt5EpWzxlqPA9OVSUXBCysU
token_duration 768h
token_renewable true
token_policies ["default"]
identity_policies []
policies ["default"]
token_meta_username alice
vault token lookup
accessor ZZt5EpWzxlqPA9OVSUXBCysU
creation_time 1614016636
creation_ttl 768h
display_name userpass-alice
entity_id 11238cf6-074d-680a-3920-0f25d4c72670
...
vault read identity/entity/id/11238cf6-074d-680a-3920-0f25d4c72670
Key Value
--- -----
aliases [map[canonical_id:11238cf6-074d-680a-3920-0f25d4c72670 creation_time:2021-02-22T17:57:16.318812652Z id:83874876-ded0-00f7-7971-ee63b7495bfa last_update_time:2021-02-22T17:57:16.318812652Z merged_from_canonical_ids:<nil> metadata:<nil> mount_accessor:auth_userpass_bbcef7b5 mount_path:auth/userpass/ mount_type:userpass name:alice]]
creation_time 2021-02-22T17:57:16.318801957Z
direct_group_ids []
disabled false
group_ids []
id 11238cf6-074d-680a-3920-0f25d4c72670
inherited_group_ids []
last_update_time 2021-02-22T17:57:16.318801957Z
merged_entity_ids <nil>
metadata <nil>
name entity_a5aeeb8b
namespace_id root
policies <nil>
vault write identity/entity/id/11238cf6-074d-680a-3920-0f25d4c72670 - <<"EOH"
{
"metadata": {
"ssh_username": "alice1"
},
"policies": ["ssh-admin"]
}
EOH
Error writing data to identity/entity/id/11238cf6-074d-680a-3920-0f25d4c72670: Error making API request.
URL: PUT http://127.0.0.1:8200/v1/identity/entity/id/11238cf6-074d-680a-3920-0f25d4c72670
Code: 403. Errors:
* 1 error occurred:
* permission denied
vault login - <root_token.txt
Success! You are now authenticated.
...
vault write identity/entity/id/11238cf6-074d-680a-3920-0f25d4c72670 - <<"EOH"
{
"metadata": {
"ssh_username": "alice1"
},
"policies": ["ssh-admin"]
}
EOH
Success! Data written to: identity/entity/id/11238cf6-074d-680a-3920-0f25d4c72670
vault login -method=userpass username=alice password=tcpip123
Success! You are now authenticated.
...
policies ["default" "ssh-admin"]
...
vault write -field=signed_key ssh-user-ca/sign/ssh-admin public_key=@$HOME/.ssh/id_rsa.pub valid_principals="alice1" >alice1.certssh-keygen -Lf alice1.cert
alice1.cert:
Type: ssh-rsa-cert-v01@openssh.com user certificate
Public key: RSA-CERT SHA256:mVV81....
Signing CA: RSA SHA256:nqMqs.... (using rsa-sha2-256)
Key ID: "vault-userpass-alice-99557...."
Serial: 10773352969806096173
Valid: from 2021-02-22T18:05:55 to 2021-02-23T06:06:25
Principals:
alice1
Critical Options: (none)
Extensions:
permit-agent-forwarding
permit-pty
vault write -field=signed_key ssh-user-ca/sign/ssh-admin public_key=@$HOME/.ssh/id_rsa.pub valid_principals="bob"
...
Code: 400. Errors:
* bob is not a valid value for valid_principalsvault write -field=signed_key ssh-user-ca/sign/ssh-test public_key=@$HOME/.ssh/id_rsa.pub
...
Code: 403. Errors:
* 1 error occurred:
* permission denied

Management UI

Looking at the entity alias which connects userpass name “alice” to this entity

Identity management part 2: OpenID Connect

Enable OIDC authentication

vault auth enable -path=google oidc
Success! Enabled oidc auth method at: google/
  1. Visit the Google API Console.
  2. Create or a select a project.
  3. Create a new credential via Credentials > Create Credentials > OAuth Client ID.
  4. Configure the OAuth Consent Screen. Select type “external” unless you have a Google Workspace account. Add scopes “email”, “profile”, “openid”. Application Name is required. Save.
  5. Back to Create Credentials > OAuth Client Id. Select application type: “Web Application”.
  6. Configure Authorized Redirect URIs. (For now, add http://localhost:8250/oidc/callback and http://vault.example.com:8200/ui/vault/auth/google/oidc/callback — where “vault.example.com” is some name that resolves to your container’s IP address, in the local /etc/hosts file if necessary)
  7. Save client ID and secret.
vault write auth/google/config - <<EOF
{
"oidc_discovery_url": "https://accounts.google.com",
"oidc_client_id": "
your_client_id",
"oidc_client_secret": "
your_client_secret",
"default_role": "standard"
}
EOF
vault write auth/google/role/standard - <<EOF
{
"allowed_redirect_uris": ["http://
vault.example.com:8200/ui/vault/auth/google/oidc/callback","http://localhost:8250/oidc/callback"],
"user_claim": "sub",
"oidc_scopes": ["profile","email"],
"claim_mappings": {
"name": "name",
"nickname": "nickname",
"email": "email"
}

}
EOF
vault login -method=oidc -path=google
Complete the login via your OIDC provider. Launching browser to:
https://accounts.google.com/o/oauth2/v2/auth?client_id=....&nonce=....&redirect_uri=http%3A%2F%2Flocalhost%3A8250%2Foidc%2Fcallback&response_type=code&scope=openid+profile+email&state=....
Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token.Key                  Value
--- -----
...
policies ["default"]
token_meta_email xxxx@gmail.com
token_meta_name Alice Sample
token_meta_role standard
vault token lookup
...
entity_id 4254a4d7-1e74-fdbf-b44b-697c3383ff7a
...
meta map[email:xxxx@gmail.com name:Alice Sample role:standard]
vault read identity/entity/id/4254a4d7-1e74-fdbf-b44b-697c3383ff7a
...
metadata <nil>
# Login with the root token again
vault list identity/entity/id
Keys
----
11238cf6-074d-680a-3920-0f25d4c72670
4254a4d7-1e74-fdbf-b44b-697c3383ff7a
vault write identity/entity/merge from_entity_ids="4254a4d7-1e74-fdbf-b44b-697c3383ff7a" to_entity_id="11238cf6-074d-680a-3920-0f25d4c72670"
Success! Data written to: identity/entity/merge
vault list identity/entity/id
Keys
----
11238cf6-074d-680a-3920-0f25d4c72670
vault write -field=signed_key ssh-user-ca/sign/ssh-admin public_key=@$HOME/.ssh/id_rsa.pub valid_principals="alice1" >alice1.cert
Error writing data to ssh-user-ca/sign/ssh-admin: Error making API request.
URL: PUT http://10.12.255.56:8200/v1/ssh-user-ca/sign/ssh-admin
Code: 400. Errors:
* template '{{identity.entity.metadata.ssh_username}}' could not be rendered -> no entity found
vault login -method=oidc -path=google
Complete the login via your OIDC provider. Launching browser to:
https://accounts.google.com/o/oauth2/v2/auth?client_id=....&nonce=....&redirect_uri=http%3A%2F%2Flocalhost%3A8250%2Foidc%2Fcallback&response_type=code&scope=openid+profile+email&state=....Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token.Key Value
--- -----
token s.ul7UzNBCDUcXEvMvniPUAC3N
token_accessor vw5pq6RNwcl3PFQDpQaEE18v
token_duration 768h
token_renewable true
token_policies ["default"]
identity_policies ["ssh-admin"]
policies ["default" "ssh-admin"]
token_meta_role standard
vault write -field=signed_key ssh-user-ca/sign/ssh-admin public_key=@$HOME/.ssh/id_rsa.pub valid_principals="alice1" >alice1.certssh-keygen -Lf alice1.cert
alice1.cert:
Type: ssh-rsa-cert-v01@openssh.com user certificate
Public key: RSA-CERT SHA256:mVV81....
Signing CA: RSA SHA256:nqMqs.... (using rsa-sha2-256)
Key ID: "vault-google-11368...."
Serial: 16887243150350464400
Valid: from 2021-02-23T16:42:45 to 2021-02-24T04:43:15
Principals:
alice1
Critical Options: (none)
Extensions:
permit-agent-forwarding
permit-pty

The user experience (CLI)

vault login -method=oidc -path=google
vault write
...as above...
# After which they can do:
#
ssh user@somehost.example.com
vault ssh -mount-point=ssh-user-ca -role=ssh-admin -mode=ca user@somehost.example.com
failed to sign public key ~/.ssh/id_rsa.pub: Error making API request.URL: PUT http://10.12.255.56:8200/v1/ssh-user-ca/sign/ssh-admin
Code: 400. Errors:
* extensions [permit-user-rc] are not on allowed list

The user experience (web UI)

vault auth tune -listing-visibility=unauth userpass/vault auth tune -listing-visibility=unauth -description="Google Account" google/
vault policy write ssh-admin - <<"EOH"
path "ssh-user-ca/roles" {
capabilities = ["list"]
}
path "ssh-user-ca/config/zeroaddress" {
capabilities = ["read"]
}
path "ssh-user-ca/sign/ssh-admin" {
capabilities = ["update"]
}
EOH
Vault UI login page
Vault UI: sign key

Tying up the security loose ends

Secure communication with HTTPS

vault secrets enable pkivault secrets tune -max-lease-ttl=175200h pkivault write -field=certificate pki/root/generate/internal common_name="ca.vault.local" ttl=175200h >/etc/vault/vault-cacert.pem
openssl x509 -in /etc/vault/vault-cacert.pem -noout -text
...
Issuer: CN = ca.vault.local
Validity
Not Before: Feb 24 09:53:52 2021 GMT
Not After : Feb 19 09:54:21 2041 GMT
...
vault write pki/roles/anycert allowed_domains="*" allow_subdomains=true allow_glob_domains=true max-ttl=8760h
vault write pki/issue/anycert common_name="vault.example.com" alt_names="vault.example.com" ip_sans="10.12.255.56,127.0.0.1" ttl=8760hKey                 Value
--- -----
certificate -----BEGIN CERTIFICATE-----
MIIDW...
-----END CERTIFICATE-----
expiration 1645696653
issuing_ca -----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
private_key -----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
private_key_type rsa
serial_number 66:23:c7:....
  • Certificate: copy everything from -----BEGIN CERTIFICATE------ to -----END CERTIFICATE----- inclusive to /etc/vault/vault-cert.pem
  • Issuing CA certificate: this is the same as vault-cacert.pem you already generated
  • Private key: copy this to /etc/vault/vault-key.pem and set permissions so it’s only readable by the vault user:
    chmod 400 /etc/vault/vault-key.pem
    chown vault:vault /etc/vault/vault-key.pem
storage "file" {
path = "/var/lib/vault"
}
listener "tcp" {
address = "0.0.0.0:8200"
#tls_disable = "true"
tls_cert_file = "/etc/vault/vault-cert.pem"
tls_key_file = "/etc/vault/vault-key.pem"

}
cluster_addr = "https://10.12.255.56:8201"
api_addr = "https://10.12.255.56:8200"
ui = "true"
# The following setting is not recommended, but you may need
# it when running in an unprivileged lxd container
disable_mlock="true"
VAULT_ADDR=https://127.0.0.1:8200
export VAULT_ADDR
VAULT_CACERT=/etc/vault/vault-cacert.pem
export VAULT_CACERT
systemctl restart vaultsource /etc/profile.d/vault.sh
vault statusvault operator unseal
vault operator unseal
vault operator unseal

Vault hardening

Conclusion

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store