Backstory (Skip if you don’t like my usual rambling)

One of the fun aspects I get to handle at work is when a new user is onboarded on the web team I get to generate some SSH keys for said user and authorize it across our fleet of internal servers and customer servers

While it wasn’t impossible, I had to keep a decent list of every server we had access to and ensure I kept a good list of who was authorized for where, and hope I don’t accidentally break the ~/.ssh/authorized_keys file with a bad copy/paste

Eventually manual key management wasn’t all that great, and I had heard great things about Ansible from the likes of Alex Kretzschmar and Jeff Geerling

So for my first ever ansible project I made, I did an ssh key manager that adds/removes keys based on the folder structure (If the key is under keys/root_allowed the key gets added to the root account, if under keys/allowed the key would be added into the authorized key list for the user, if under keys/revoked they get scanned for and removed from the user account, etc)

Since it was my first true ansible project, it was very mashed together with zero concept of best practices for ansible,

It even takes 20-ish minutes to execute since it’s a looping mess (While there are about 20-ish servers it has to go through, on a couple of the cPanel servers I have a custom task that goes through and adds the regular user keys to every user on there for ease of SSH-ing into accounts with VSCode Remote SSH for example.)

So when an article floated through /r/linux that a friend forwarded to me talking about SSH Certificates for host/client authorization via SmallStep CA I was intrigued to say the least

Unfortunately as the comments on the Reddit post point out, the article isn’t without biases, since SmallStep is a company that has some open source products for managing a small private x509/SSH CA and some commercial software that would allow you to use a hosted highly-available version of their open source offering

While I do agree that the article isn’t the greatest since it glosses over a few problems (Like the problem of if your CA server is down when you try to get your key signed, or atleast note the chicken and the egg problem of using SSO and potentially having it run on a server that requires said SSO to be up to log in, thus if your server fails to start the processes for the SSO service, your kind of screwed from logging in to fix it),
Some of the comments seem to suggest that the article as a whole is bad, certainly with a title of “If You’re Not Using SSH Certificates You’re Doing SSH Wrong” I can see where they are coming from, but there is some very useful information included, such as going over the problem of TOFU or Trust on First Use problem default SSH has, not to mention this article showed me that OpenSSH certificate authorization is a thing and showed me the basic concepts of the technology.

The other thing is that as long as your not removing other config lines from your SSH configs, your server will still be fine working with default OpenSSH keys, so if you want to implement this, I would recommend having a regular SSH backup key in an encrypted vault or printed and kept safe incase something in here goes horribly wrong.

Also while I am typing out this long winded foreword, I would like to thank iBug and Peter of Framkant.org for their fantastic blog entries on the subject that helped me get the core setup down for OpenSSH CAs

The start of the actual post

There are two avenues for implementing an OpenSSH CA, Host Key Signing and User Key Signing

Both ways are nearly identical in the way you sign them, it’s just down to configuration differences and what end is trusting/authorizing the other.

So for Host Key Signing your using an OpenSSH CA to sign the host keys on a server providing an easy chain for Users/Clients to trust against

While User Key Signing is the reverse of the above where your using an OpenSSH CA to sign your personal ssh keys providing an easy chain for the server to authorize your key access to certain ssh resources.

Setting up an OpenSSH CA

This is the easiest part of this whole experience, you just generate a normal SSH key as you normally would.

The only asterisks to consider is using separate CAs for Host Key Signing and User Key Signing. and the use of RSA keys vs ED25519 or other newer keys

Same or Seperate Keys for User/Host Key Signing

For simplicity you could use the same CA keys for host signing and user signing, but ultimately if the CA key is compromised in some way your whole chain of trust is shattered, with them being split that gives you atleast some wiggle room to use one to replace the other and rebuild a chain of trust.

RSA vs newer keys

One thing to consider is what key types your servers will support, If you still have older servers running older versions of OpenSSH, Like CentOS 6 I would highly recommend working on an upgrade path for those, however if you do still need to support servers like this (For example the extended support CloudLinux 6 Servers) then RSA will “work” in those cases

But if your servers have OpenSSH 6.5 or newer then you should be good to generate ed25519 keys for example.

Generating Keys

Navigate to a safe spot you want to generate your keys into and use ssh-keygen like normal

rsa sample:

ssh-keygen -b 4096 -t rsa -f ./user_ca -C "Example.com SSH User CA"
ssh-keygen -b 4096 -t rsa -f ./host_ca -C "Example.com SSH Host CA"

ed25519 sample:

ssh-keygen -t ed25519 -f ./user_ca -C "Example.com SSH User CA"
ssh-keygen -t ed25519 -f ./host_ca -C "Example.com SSH Host CA"

I would highly recommend entering a password when prompted to encrypt the private key, also ensure your using a decent password that is unique for each CA, You will need the password each time you want to sign a host/user key.

However if you wish to use Ansible with this for example through the openssh_cert_module then your going to have to not use passwords on your keys unfortunately

(There was a PR to address this a while back, but it was closed to allow the coder to prioritize other idempotency problems with the openssh module sets as a whole, I’ll keep an eye out if Ajpantuso does end up returning to it and completes it, if so I’ll update this piece of the article)

Setting up Hosting Key Signing

Pull Public Keys from Server

First step is to pull the server’s host public keys from the server
example:

mkdir -p ./public_keys/server.example.com
scp [email protected]:/etc/ssh/*.pub ./public_keys/server.example.com/

Signing the Server Keys

Next step is to sign the keys you got using this format
ssh-keygen -s ca_private_key -I signature_line -h host_public_key To break this down

example:

ssh-keygen -s ./host_ca -I server.example.com,8.8.8.8 -h ./public_keys/server.example.com/ssh_host_ecdsa_key.pub
ssh-keygen -s ./host_ca -I server.example.com,8.8.8.8 -h ./public_keys/server.example.com/ssh_host_rsa_key.pub
ssh-keygen -s ./host_ca -I server.example.com,8.8.8.8 -h ./public_keys/server.example.com/ssh_host_ed25519_key.pub

Copying back the keys, and editing sshd_config to use the keys

This will generate the signed pub files that you will need to copy back to the server

scp ./public_keys/server.example.com/ssh_host_*_key-cert.pub [email protected]:/etc/ssh/

You will need to edit the sshd config on the server to make it use the new signed keys we made. Edit /etc/ssh/sshd_config and add in HostCertificate lines

example:

HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

Finally reload the ssh daemon on your server

Setting your known_hosts file to authorize the CA

Just a quick sidenote on ensuring your computer now trusts your ca

Edit your ~/.ssh/known_hosts and we are going to add in a special line to let your machine know about the CA Public key

The format is as follows
@cert-authority limiter ssh-public-key

The first part tells SSH that this line pertains to a CA key,
limiter refers to what DNS names apply to this key (You can use just a * to say trust the key everywhere, but I would recommend limiting it to a root domain name such as *.example.com, that way server.example.com is included along with web01.example.com or any other servers listed as subdomains under example.com)
ssh-public-key is the CA’s public key string

example:

@cert-authority *.example.com ssh-rsa AAAAB3Nz.....

With this if you SSH into a server with it’s cert chain setup properly you won’t be asked to verify the host key and add it into known_hosts

Setting Up User Key Signing

sshd_config edits

First step will be to copy the user_ca.pub file to the server at /etc/ssh/user_ca.pub for example

Then add in the following lines to the /etc/ssh/sshd_config file (Don’t include the comments)

# Note that this says CAKeys, meaning if you have multiple 
# user CA's you would like to authorize for this server you can put multiple keys inside the file
# And the server will trust each CA string inside
TrustedUserCAKeys /etc/ssh/user_ca.pub

TrustedUserCAKeys authorizes CA keys to allow any login as long as their principal matches the login,

And then restart the ssh daemon

User Key Signing

Still simple, but we have a few things to go over

Here is a sample ssh-keygen line ssh-keygen -s ca_private_key -I key_id -n signature_line -V -1w:+52w1d user_public_key And to break it down:

Authorized Principals

Earlier principals were mentioned for the user keys, so for keys this is where you can put usernames the user is allowed to sign into

However you can change OpenSSH from taking Principals as usernames to be group assignments instead

Adding the following to your sshd_config

AuthorizedPrincipalsFile /etc/ssh/authorized_principals/%u

And creating a folder for the new principal files to live in

mkdir -p /etc/ssh/authorized_principals

After a reloading of the ssh daemon we can now do shared groups per key signing.

# cat /etc/ssh/authorized_principals/root
root
sysadmins

# cat /etc/ssh/authorized_principals/prod
prod
webservers

So with this only keys with principals for root and sysadmin will be able to login as root, and users with prod or webservers can access the prod user.

Do note that this will disable normal user login usage for principal declarations, so if you want users to be able to use their own accounts again, you will have to assign bob’s account the bob principal for example

Here is an example using user account and principals together. So lets say we have 4 users on a server, two being shared or generic:

• root
• prod

and two that are regular users:

• bob
• alice

bob’s key just has the bob and webmaster principals, while alice has both alice and sysadmin
and we have our authorized principals setup as the following:

# cat /etc/ssh/authorized_principals/root
sysadmin

# cat /etc/ssh/authorized_principals/prod
webmaster
sysadmin

# cat /etc/ssh/authorized_principals/bob
bob

# cat /etc/ssh/authorized_principals/alice
alice

Then our table of access would look like this:

As we can see Bob and Alice have access to their own accounts and prod, but they don’t have access to each others accounts nor does Bob have access to root while Alice does

Revoking User Keys

One problem that isn’t well covered is what do you do when a key is compromised or when a user leaves and you need to disable their key from logging in on all of your servers.

Unfortunately there isn’t an easy way to have one Key Revocation List on one server that the rest of your servers reference, The only real simple way to manage a list is to make a single KRL and copy it to all servers and reference them in the sshd_config

I would highly recommend keeping a copy of each signed cert (or base pub file before signing) stored safely locally on one machine, so if you need to revoke them you can.

Going back to the previous example, let’s say Bob has left example.com and has moved on, now Alice has to revoke his key on all servers

If no KRL existed before we would generate a new KRL with the following command ssh-keygen -kf ./revoked_keys -z 1 ./[email protected]

Or you can make a blank one using the following command
ssh-keygen -kf ./revoked_keys

Then if you want to update an existing KRL or populate a new blank one, you can just do the following: ssh-keygen -kuf ./revoked_keys ./[email protected]

Finally if you want to verify a pub file has been revoked, you can use the -Q option

[alice@server ~]$ ssh-keygen -Qf ./revoked_keys ./[email protected]
/home/alice/[email protected] ([email protected]): REVOKED

Now your going to want to distribute this revoked list to all of your servers (I would recommend Ansible for this or another configuration management framework), in /etc/ssh/revoked_keys for example

Finally make sure you add RevokedKeys /etc/ssh/revoked_keys into your sshd config and reload the ssh daemon

Do note that this only revokes keys related to CA authorized logins, if the public key is manually added into the authorized_keys file of a user, that will still allow them to login

One way to combat this is to set AuthorizedKeysFile in the sshd_config to /dev/null. This will disable the authorized_keys file for all user accounts including root, but it will secure you against any key based logins outside of CA signed ones.

You could also login as root and try to find the keystring in all files on the server

# May take a LONG time, and may not work
grep -r 'ssh-rsa AAAAB3Nx..... [email protected]' /

Then remove it from any files it shows up in
Again I wouldn’t recommend this method since it’s a bit messy to do on every server in a fleet.

Another way is to do short validity periods for certs (Although good luck not making that a pain in the ass to do manually)