Why would you bother?
It’s reasonable (but maybe not the best) practice to want to run a sensu backend as a publicly addressable service with secure api and dashboard access. Let’s Encrypt is a great option for generating and renewing certs for your public facing sensu-backend process, because your browser and your OS most likely already has Let’s Encrypt as a trusted root CA. By using Let’s Encrypt generated certs it really reduces the amount of friction you’ll have accessing your tls secured backend from a web browser or sensuctl (even when the backend is only accessible while on a vpn)
But Let’s Encrypt is a poor choice for generating your agent client certs used in the mutual TLS connections between agents and the backend. You generally don’t want your agents client certs to be trusted widely by many services by using a commonly trusted root CA like Let’s Encrypt. And the Let’s Encrypt verification process can be very cumbersome for agent client certs anyways. For agent client certs its much better to have a private CA that you control, and issue client certs to individual clients.
So here’s what I did to get this setup for a backend I have running in the cloud.
Preparing the /etc/sensu directory structure
To make it easier for me to keep my private CA certs and my letsenycrpt certs separate I like to structure my /etc/sensu directory like this
/etc/sensu/
└── tls/
├── letsencrypt/
└── private/
Generate your Let’s Encrypt backend certificates
I used the recommended certbot process, which generates certificates and setup the recommended renewal cronjob running on the backend host.
Ref: https://letsencrypt.org/getting-started/
If this were a k8s cluster I’d have setup a k8s job controller to mimic the behavior of the renwal cronjob.
Make sure the generated certs are readable by the sensu group
certbot will generate certs in /etc/letsencrypt/live/<domainname>/ which are just pointers into
/etc/letsencrypt/archive/<domainname>/
Because the sensu-backend runs as the sensu user, you’ll need to adjust the file permissions and group ownership so that the sensu user is able to read the files.
You can test if the permissions on the private key are sufficient with:
sudo -u sensu cat /etc/letsencrypt/live/<domainname>/privkey.pem
and with the signed certificate with:
sudo -u sensu cat /etc/letsencrypt/live/<domainname>/fullchain.pem
I think add symlinks for the live privkey.pem and fullchain.pem into my ‘/etc/sensu/tls/letsencrypt/’ directory. I’ll reference these symlinks in the backend configuration.
Configure your backend to use the letsencrypt certs to secure all the things
##
# tls configuration:
# unless overridden by other config options will establish tls connections for:
# api, dashboard and agent websocket endpoint
##
cert-file: "/etc/sensu/tls/letsencrypt/fullchain.pem"
key-file: "/etc/sensu/tls/letsencrypt/key.pem"
Restarting the backend service after adding these 2 lines, should now give you tls secured access to the api (https://) the dashboard (https://) and the agent websocket (wss://), but agent mutual tls not yet active.
Create your private CA and generate your first agent cert
We’ve got a pretty good guide for how to generate certs using a private CA. Because we are using the Lets Encrypt generated certs for pretty much everything associated with the backend server, we only need to use the private CA to generate and verify the agent cert.
So start off by generating the private CA:
https://docs.sensu.io/sensu-go/latest/guides/generate-certificates/#create-a-certificate-authority-ca
Then you can skip ahead to generating the agent cert:
https://docs.sensu.io/sensu-go/latest/guides/generate-certificates/#generate-agent-certificate
There’s one important thing to stress about the agent cert, the CN you encode in the cert needs to map to a sensu user with RBAC rights necessary for correct agent operation. The default user ‘agent’ works great for your first cert. In fact once you create this cert you can use it for all your agents, or you can create different certs for each agent (so that you can later revoke each agent’s access seperately). You can go real deep here and generate certs for different users other than the default ‘agent’ if you want to start restricting agents to specific namespaces using RBAC. By the power of RBAC you have the power!!!
Anyways… so you now have several files associated with your private CA. You’ll want to archive the private CA files somewhere safe, so you can use them again later to generate or revoke additional client certs. But there’s only 3 files you need to drop into your running environments.
On the backend server place the ca.pem you created in
/etc/sensu/tls/private/
make sure its readable by the sensu user. The ca.pem (or equivalently named file) is the only file the backend host needs
On the agent system place the agent.pem and agent-key.pem certificates in
/etc/sensu/tls/private/
Make sure these files are readable by the sensu user
The agent does not need the ca.pem as the server will be using the Lets Encrypt generated certs, and the agent will be able to rely on the OS’s root certs to validate the server~
Configuring sensu-backend to enable mTLS
##
# Enable agent mTLS
# once enabled all agents will require client certs
# you cannot mix user/pass auth agents with mTLS
##
agent-auth-cert-file: "/etc/sensu/tls/letsencrypt/fullchain.pem"
agent-auth-key-file: "/etc/sensu/tls/letsencrypt/key.pem"
agent-auth-trusted-ca-file: "/etc/sensu/tls/private/ca.pem"
Let me break this down.
The agent-auth-cert-file and agent-auth-key-file options override the cert-file and key-file options and instruct the backend server to enable mutual TLS. However I’m still using the same Lets Encrypt certificates as before to identify the backend to any websocket client that connects.
The agent-auth-trusted-ca-file is now pointing to the private CA that was used to sign the agent client certificate. When agents connect, they will be offering the backend the client cert, and this is the CA the backend will use to verify it.
Restart the backend service, and now all agents connecting to the agent websocket will need to provide a client cert signed by the private CA.
Configuring an agent to use mTLS
Now the last bit, configuring the agents. With mTLS enabled now on the backend, agents won’t be able to authenticate using the user/password. They will need to use client certs instead.
##
# agent mTLS authentication
#
##
cert-file: "/etc/sensu/tls/private/agent.pem"
key-file: "/etc/sensu/tls/private//agent-key.pem"
Restarting the agent after making these changes should result in agent using mTLS based authentication with the backend server.
Note, I didn’t need to set the trusted-ca-file attribute in the agent configuration because the backend server is identifying itself with the Lets Encrypted generated certs, that my agent’s host OS already trusts as a root CA. The only privately signed certs in use are by the agents, so only the backend needs to be configured to point to the special private CA.
So there it is, mixing Let’s Encrypt signed backend server certificates with private CA signed client certs to get mutual TLS up and running. And once mTLS agent connections are established, you’ll be able to get access to sensu-backend defined secrets and make use of them inside your checks.