Secrets Management on Sensu OSS

Sensu Go
1

Hello All,

So it looks like the secrets management including the env provider is only on the commercial version of Sensu Go. What would be the best way to handler secrets on the sensu backends on the Sensu GO OSS?

I’m running the sensu backends on a container environment and was wondering if the handler configs could pick up container environment variables set by me?

---
api_version: core/v2
type: Handler
metadata:
  name: slack

spec:
  type: pipe
  command: sensu-slack-handler --channel '#channel' --username 'sensu'
  env_vars:
  - SLACK_WEBHOOK_URL=$SLACK_WEBHOOK_URL
  runtime_assets:
    - sensu-slack-handler
  filters:
    - is_incident
  timeout: 10

$SLACK_WEBHOOK_URL being an environment variable on the container.

1 Like
2

Hey,

Tl;dr

It’s always possible to set and use environment variables in the running environment for the Sensu backend and agent both the OSS and commercial binaries. In addition, many of the Sensu pipeline components (handlers, mutatators, checks, etc…) support configuration of env_vars as a resource attribute.

In fact, what you try as an example in your handler is just a little off. You can’t reference a the value of an env_var to itself like that. I’ll show you why in an example below.

Setting up environment variables for Sensu to use

If you don’t want to put secrets in your monitoring configuration(because your hosting it in a git repo for example), then you can inject then into the running environment at backend start up. If using k8s, podman or docker you should be able to inject envvars into the running environment of the Sensu backend as per the normal process for those technologies. Or if you are using Sensu on a modernish linux distribution using systemd for managing the running backend you can instruct systemd to read in a file of envvars into the service environment. The service scripts in the commercial packages set this up already.

Example time

Let me share with you an example from my workstation using systemd.

Here’s my handler, all it does right now is report back the environment variables that the handler command has access to.

type: Handler
api_version: core/v2
metadata:
  created_by: admin
  labels:
    sensu.io/managed_by: sensuctl
  name: echo_slack_env
  namespace: default
spec:
  command: set |grep SLACK_ > /var/log/sensu/slack_environment.txt
  timeout: 0
  type: pipe

Initially that’s nothing. But after I populate the file /etc/sysconfig/sensu-backend

cat /etc/sysconfig/sensu-backend 

PERSONAL_SLACK_WEBHOOK_URL="http://this.is.redacted/because/I/care"
SLACK_WEBHOOK_URL="http://seriously.not./going/to/leak/this"

and restart the sensu-backend service running under systemd… the handler output is different.

sudo cat /var/log/sensu/slack_environment.txt

PERSONAL_SLACK_WEBHOOK_URL=http://this.is.redacted/because/I/care
SLACK_WEBHOOK_URL=http://seriously.not./going/to/leak/this

Cool.
Similar things are possible with k8s and docker. k8s in particular exposes some very useful secrets as env_vars in the running environment.For example k8s lets you set up a service account and exposes the necessary auth token that help you interact with the k8s api from inside a managed container. Any envars your container manager sets up at container startup are available just like what I setup in my systemd example above.

Sensu secrets at scale

Okay so you are probably asking yourself now, if I can do all this, why bother with the secrets management at all as a commercial feature?

The commercial Sensu secrets management feature is intended primarily to support Enterprise-scale secrets management using Vault (and other external secrets providers in the future). This is especially valuable in organizations that require strict secrets auditing, where operators aren’t allow to put sensitive information into the configs are even in group restricted config files on disk.

Sensu’s secrets are implemented to function like envvars to make it easier to transition from commercial version to OSS and back. Sensu commercial releases provide both an env and Vault secrets provider, but the env secrets provider is there primarily to help transition to something like Vault and to help prototype. You can use the env provider in a qa/testing for example and transition to Vault in production.

As an aside, the Sensu secret management implementation also provides a first class solution for secure distribution to agents, when mutual TLS is also enabled. But that’s a topic for a different day. If you are in an org that requires mtls you’re probably going to be requiring something like Vault anyways and won’t be allow to encode secrets directly.

4 Likes
3

@jspaleta Thanks a lot for the quick response on this. Really helps. I guess I was trying to make it work using the env_vars using the file option in a containerized environment which is why it didn’t work. Didn’t realize that the injected environment variables would be accessible via the command block itself. It works now when I try to access the environment variable from the command block.

However, just curious to know whether I can use an injected environment variable to set env_vars? This use case is if we would need to set multiple handlers of the same type for different teams. Tried the below example but doesn’t seem to work. Any thoughts?

---
api_version: core/v2
type: Handler
metadata:
  name: slack

spec:
  type: pipe
  command: sensu-slack-handler --channel '#channel' --username 'sensu'
  env_vars:
  - SLACK_WEBHOOK_URL=$SLACK_ENV
  runtime_assets:
    - sensu-slack-handler
  filters:
    - is_incident
  timeout: 10

Where $SLACK_ENV is an injected environment variable.

4

Hey,
the env_vars array doesn’t get interpreted by the shell, What happens is the env_var value is passed verbatim as a string literal.
But what you can do is edit the command to look something like this:

 command: SLACK_WEBHOOK_URL=$SLACK_ENV; sensu-slack-handler --channel '#channel' --username 'sensu'

Since the command runs inside the shell, you can override the systemwide SLACK_WEBHOOK_URL referencing another env_var value using normal rules for the shell substitution.

1 Like
5

@jspaleta Cool! Thanks a lot for helping out with this. This totally works for our use case. This also clears up some confusion on working with secrets on OSS versions.

Thanks a lot again! You guys and Sensu GO community are the best! :slight_smile:

1 Like
6

I’d add, that this sort of preloading of envvars like that also helps readability for future members of your team. Instead of using SLACK_WEBHOOK_URL silently, this sort of command construction gives anyone reading the sensu resource definitions notice that the env_var is in use.

1 Like