Secrets Management on Kubernetes with ENV Provider

seizste · June 2, 2020, 4:10pm

We used the Sensu Go K8s Quickstart Template to spin up a 3 Node Sensu Go Cluster hosted on Kubernetes - GitHub - sensu/sensu-k8s-quick-start - as we have a lot of Checks that require Username and Password we need to get Secrets Management up and running.

We followed the Guide to get certificates for mTLS installed and Agents are communicating, but we’re not really getting Secrets Management Up and Running.

Creating a /etc/default/sensu-backend (k8s pods are using Alpine Linux 3.8.5) file does not seem to work, we also tried to directly create Environment Variables on Backend Pods but still no success. Only thing that seems to work is, if we do both but is this really required ? btw. Hashicorp Vault is not an option for us … anyone else running Sensu Go on k8s and having experience with Secrets Management ?

calebhailey · June 2, 2020, 9:07pm

Hi there, @seizste great question!

The /etc/default/sensu-backend approach relies on process management (e.g. systemd or sysvinit) which would not normally be present in a containerized environment, so I wouldn’t recommend that approach.

The Sensu Go env Secrets Provider should work directly with Kubernetes own built-in secrets management. In practice, Kubernetes secrets are discrete K8s resources that make secrets available to reference from various pod controllers (e.g. StatefulSets); in other words, you have to create a K8s Secret, and then also reference it to actually use it somewhere.

Here’s a few example K8s secrets:

---
apiVersion: v1
kind: Secret
metadata:
  name: influxdb
type: Opaque
stringData:
  addr: http://influxdb-0.influxdb.sensu-system.svc.cluster.local:8086
  db: sensu 
  user: admin 
  password: password

---
apiVersion: v1
kind: Secret
metadata:
  name: servicenow
type: Opaque
stringData:
  host: xxxxxxxx.service-now.com/
  username: admin 
  password: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  cmdb_ci_table: cmdb_ci 
  event_table: event 
  incident_table: incident

And here’s an example excerpt from a StatefulSet resource, which fetches the value of a K8s Secret and exposes it as an environment variable:

containers:
- name: sensu-backend
  image: sensu/sensu:5.20.1
  command: ...
  env:
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: INFLUXDB_ADDR
    valueFrom:
      secretKeyRef:
        name: influxdb
        key: addr
  - name: INFLUXDB_DB
    valueFrom:
      secretKeyRef:
        name: influxdb
        key: db
  - name: INFLUXDB_USER
    valueFrom:
      secretKeyRef:
        name: influxdb
        key: user
  - name: INFLUXDB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: influxdb
        key: password

In this example, I’m fetching the values of my influxdb secret and mapping them as environment variables (INFLUXDB_ADDR, INFLUXDB_DB, INFLUXDB_USER, and INFLUXDB_PASSWORD) which will be accessible from the sensu-backend container in this StatefulSet pod.

I hope this helps!

jspaleta · June 2, 2020, 9:13pm

Hmm maybe for completeness we need a simple example for k8s sidecar using a k8s secret in the k8s quick start example.

calebhailey · June 2, 2020, 9:14pm

One more thing… you should be able to troubleshoot/verify that these environment variables are being correctly exposed by using kubectl exec.

https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/

Here’s a quick example/demo:

seizste · June 3, 2020, 7:16am

Hello @calebhailey,

thanks for the detailed help. I finally got it working using the following steps / configs in k8s and sensu.

k8s-secret

apiVersion: v1
data:
  es-pwd-logging-and-metric: P@ssw0rd!
kind: Secret
metadata:
  name: sensu-env-secrets
  namespace: sensu-system
type: Opaque

k8s-deployment
...
env:
  - name: SENSU_ES_PWD_LOGGING_AND_METRIC
    valueFrom:
      secretKeyRef:
        name: sensu-env-secrets
        key: es-pwd-logging-and-metric
...

sensu-secret

type: Secret
api_version: secrets/v1
metadata:
  name: es_pwd_logging_and_metric
  namespace: default
spec:
  id: SENSU_ES_PWD_LOGGING_AND_METRIC
  provider: env

sensu-check

type: CheckConfig
api_version: core/v2
metadata:
  name: es-mgmt-and-monitoring-health
  namespace: default
spec:
  command: check-es-cluster-health.rb -h server.test.cloud -p 9243 -l cluster 
    -u elastic -P $ES_PWD
interval: 300
publish: true
runtime_assets:
  - sensu-plugins-elasticsearch
  - sensu-ruby-runtime
  secrets:
  - name: ES_PWD
    secret: es_pwd_logging_and_metric
subscriptions:
  - linux

raulgs · June 3, 2020, 2:59pm

Just for my understanding Do these env variables need to be available on the backend or the agents?
I currently do have a check where I have exposed secrets using env variables on the agent.
If backend only is fine I will move them there.

calebhailey · June 3, 2020, 4:10pm

@raulgs great question! The real value-add of Sensu’s built-in Secrets Management capabilities is that you only need to define the secrets on the backends. However, in order for Sensu to transmit those secrets to agents for use in checks, you must have secured agent-to-sever communication (TLS encrypted transport) and be using mTLS agent authentication. This is explained in the Secrets reference documentation, here:

Secrets are only transmitted over a transport layer security (TLS) websocket connection. Unencrypted connections must not transmit privileged information. For checks, hooks, and assets, you must enable mutual TLS (mTLS). Sensu will not transmit secrets to agents that do not use mTLS.

Sensu only exposes secrets to Sensu services like environment variables and automatically redacts secrets from all logs, the API, and the dashboard.

This guide covers both topics:

I hope this helps!

seizste · June 3, 2020, 4:27pm

@calebhailey … a follow up question, where can i use secrets to replace something in the config of a check or handler … or asked differently, is the following code valid ?

type: Handler
api_version: core/v2
metadata:
  name: dnscheck-1486439-prod
spec:
  command: sensu-go-elasticsearch --index metrics-sensu-dnscheck-1486439-prod --full_event_logging
  env_vars:
    - $ES_SENSU_CLOUD_ID
  runtime_assets:
    - sensu-go-elasticsearch
  secrets:
    - name: ES_SENSU_CLOUD_ID
      secret: es_pwd_1486439_prod
  timeout: 0
  type: pipe

calebhailey · June 3, 2020, 9:32pm

Hey @seizste

You’re on the right track! A few observations/comments:

Sensu Secrets are directly exposed as environment variables, so you do not need to set an environment variable in addition to the secret in the handler definition (in your example handler definition you have an env_var and a secret called ES_SENSU_CLOUD_ID, which is not necessarily an invalid configuration, but it’s most likely going to cause some conflict).
Because Sensu Secrets read secret values and expose them to the check or handler as environment variables, the plugin/command being executed by Sensu either needs to implicitly read those environment variables, or they must be passed into the command via a command line argument; for example:
```
myplugin.rb --url=${MY_SECRET}
```
It looks like the Elasticsearch plugin you’re using supports a single environment variable (ELASTICSEARCH_URL ) with a value like https://user:pass@hostname:port. It appears that your Kubernetes secret only contains the Elasticsearch “password”, so you’ll either need to construct the ELASTICSEARCH_URL in the handler command, or modify the contents of your secret and expose the secret as ELASTICSEARCH_URL instead of ES_SENSU_CLOUD_ID

Here’s a few example Sensu secret + handler definitions to help explain what I mean by this last comment:

Pass secret an explicit environment variable in the command

---
type: Secret
api_version: secrets/v1
metadata:
  name: es_pwd_1486439_prod
spec:
  provider: env
  id: SENSU_ES_PWD_LOGGING_AND_METRIC
---
type: Handler
api_version: core/v2
metadata:
  name: dnscheck-1486439-prod
spec:
  command: >-
    ELASTICSEARCH_URL="https://admin:${ES_SENSU_CLOUD_ID}@elasticsearch.yourdomain.com:9243" &&
    sensu-go-elasticsearch --index metrics-sensu-dnscheck-1486439-prod --full_event_logging
  runtime_assets:
    - sensu-go-elasticsearch
  secrets:
    - name: ES_SENSU_CLOUD_ID
      secret: es_pwd_1486439_prod
  timeout: 0
  type: pipe

Pass secret as implicit environment variable

---
type: Secret
api_version: secrets/v1
metadata:
  name: sensu_es_url
spec:
  provider: env
  id: SENSU_ES_URL
---
type: Handler
api_version: core/v2
metadata:
  name: dnscheck-1486439-prod
spec:
  command: sensu-go-elasticsearch --index metrics-sensu-dnscheck-1486439-prod --full_event_logging
  runtime_assets:
    - sensu-go-elasticsearch
  secrets:
    - name: ELASTICSEARCH_URL
      secret: sensu_es_url
  timeout: 0
  type: pipe

NOTE: In this latter example, you’d need to modify your Kubernetes secret so that the value of the secret is in the format of: https://user:pass@hostname:port.

I hope this helps!

seizste · June 4, 2020, 7:10am

Thanks a lot @calebhailey … i went for the second option and it works like a charm !

calebhailey · June 11, 2020, 7:13pm

Awesome! Glad to hear it!

Topic		Replies	Views
Secrets Management on Sensu OSS Sensu Go	5	767	July 21, 2020
Install Install Sensu docker image in K8S Getting Started sensu-go	2	515	April 8, 2021
How can we deploy sensu-go server cluster on Kubernetes? Sensu Go	2	913	November 25, 2019
Sensu \| Monitoring Kubernetes, part 4: the Sensu-native approach The Sensu Blog	4	450	July 9, 2021
Creating Global ENV Secrets Sensu Go sensu-go	7	311	February 24, 2021

Secrets Management on Kubernetes with ENV Provider

Pass secret an explicit environment variable in the command

Pass secret as implicit environment variable

Related topics