# Secrets Management on Kubernetes with ENV Provider

**URL:** <https://discourse.sensu.io/t/secrets-management-on-kubernetes-with-env-provider/1841>\
**Category:** Sensu Go\
**Created:** [June 2, 2020, 4:10pm UTC](https://discourse.sensu.io/t/secrets-management-on-kubernetes-with-env-provider/1841 "2020-06-02T16:10:53Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![seizste](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/seizste/32/343_2.png) [@seizste](https://discourse.sensu.io/u/seizste)\
**Post date:** [June 2, 2020, 4:10pm UTC](https://discourse.sensu.io/t/secrets-management-on-kubernetes-with-env-provider/1841/1 "2020-06-02T16:10:53Z")

</div>

We used the Sensu Go K8s Quickstart Template to spin up a 3 Node Sensu Go Cluster hosted on Kubernetes - [GitHub - sensu/sensu-k8s-quick-start](https://github.com/sensu/sensu-k8s-quick-start) - as we have a lot of Checks that require Username and Password we need to get Secrets Management up and running.

We followed the Guide to get certificates for mTLS installed and Agents are communicating, but we’re not really getting Secrets Management Up and Running.

Creating a /etc/default/sensu-backend (k8s pods are using Alpine Linux 3.8.5) file does not seem to work, we also tried to directly create Environment Variables on Backend Pods but still no success. Only thing that seems to work is, if we do both but is this really required ? btw. Hashicorp Vault is not an option for us … anyone else running Sensu Go on k8s and having experience with Secrets Management ?

---

<div class="post-metadata">

**Author:** ![calebhailey](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/calebhailey/32/46_2.png) [@calebhailey](https://discourse.sensu.io/u/calebhailey)\
**Post date:** [June 2, 2020, 9:07pm UTC](https://discourse.sensu.io/t/secrets-management-on-kubernetes-with-env-provider/1841/2 "2020-06-02T21:07:03Z")

</div>

Hi there, @seizste 👋 great question!

The `/etc/default/sensu-backend` approach relies on process management (e.g. systemd or sysvinit) which would not normally be present in a containerized environment, so I wouldn’t recommend that approach.

The Sensu Go `env` Secrets Provider should work directly with Kubernetes own built-in secrets management. In practice, Kubernetes secrets are discrete K8s resources that make secrets available to reference _from various pod controllers_ (e.g. StatefulSets); in other words, you have to _create_ a K8s Secret, and then also _reference it_ to actually use it somewhere.

Here’s a few example K8s secrets:

```auto
---
apiVersion: v1
kind: Secret
metadata:
  name: influxdb
type: Opaque
stringData:
  addr: http://influxdb-0.influxdb.sensu-system.svc.cluster.local:8086
  db: sensu 
  user: admin 
  password: password

---
apiVersion: v1
kind: Secret
metadata:
  name: servicenow
type: Opaque
stringData:
  host: xxxxxxxx.service-now.com/
  username: admin 
  password: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  cmdb_ci_table: cmdb_ci 
  event_table: event 
  incident_table: incident  

```

And here’s an example excerpt from a StatefulSet resource, which fetches the value of a K8s Secret and exposes it as an environment variable:

```auto
containers:
- name: sensu-backend
  image: sensu/sensu:5.20.1
  command: ...
  env:
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: INFLUXDB_ADDR
    valueFrom:
      secretKeyRef:
        name: influxdb
        key: addr
  - name: INFLUXDB_DB
    valueFrom:
      secretKeyRef:
        name: influxdb
        key: db
  - name: INFLUXDB_USER
    valueFrom:
      secretKeyRef:
        name: influxdb
        key: user
  - name: INFLUXDB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: influxdb
        key: password

```

In this example, I’m fetching the values of my `influxdb` secret and mapping them as environment variables (`INFLUXDB_ADDR`, `INFLUXDB_DB`, `INFLUXDB_USER`, and `INFLUXDB_PASSWORD`) which will be accessible from the `sensu-backend` container in this StatefulSet pod.

I hope this helps!

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [June 2, 2020, 9:13pm UTC](https://discourse.sensu.io/t/secrets-management-on-kubernetes-with-env-provider/1841/3 "2020-06-02T21:13:37Z")

</div>

Hmm maybe for completeness we need a simple example for k8s sidecar using a k8s secret in the k8s quick start example.

---

<div class="post-metadata">

**Author:** ![calebhailey](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/calebhailey/32/46_2.png) [@calebhailey](https://discourse.sensu.io/u/calebhailey)\
**Post date:** [June 2, 2020, 9:14pm UTC](https://discourse.sensu.io/t/secrets-management-on-kubernetes-with-env-provider/1841/4 "2020-06-02T21:14:34Z")

</div>

One more thing… you should be able to troubleshoot/verify that these environment variables are being correctly exposed by using `kubectl exec`.

[https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/](https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/)

Here’s a quick example/demo:

 ![2020-06-02 14.11.53](https://us1.discourse-cdn.com/flex016/uploads/sensu/original/1X/163d85d1edae8481f06e2cb7f67ae4d5f8c6f4ba.gif)

---

<div class="post-metadata">

**Author:** ![seizste](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/seizste/32/343_2.png) [@seizste](https://discourse.sensu.io/u/seizste)\
**Post date:** [June 3, 2020, 7:16am UTC](https://discourse.sensu.io/t/secrets-management-on-kubernetes-with-env-provider/1841/5 "2020-06-03T07:16:18Z")

</div>

Hello @calebhailey,

thanks for the detailed help. I finally got it working using the following steps / configs in k8s and sensu.

```
k8s-secret

apiVersion: v1
data:
  es-pwd-logging-and-metric: P@ssw0rd!
kind: Secret
metadata:
  name: sensu-env-secrets
  namespace: sensu-system
type: Opaque

k8s-deployment
...
env:
  - name: SENSU_ES_PWD_LOGGING_AND_METRIC
    valueFrom:
      secretKeyRef:
        name: sensu-env-secrets
        key: es-pwd-logging-and-metric
...

sensu-secret

type: Secret
api_version: secrets/v1
metadata:
  name: es_pwd_logging_and_metric
  namespace: default
spec:
  id: SENSU_ES_PWD_LOGGING_AND_METRIC
  provider: env

sensu-check

type: CheckConfig
api_version: core/v2
metadata:
  name: es-mgmt-and-monitoring-health
  namespace: default
spec:
  command: check-es-cluster-health.rb -h server.test.cloud -p 9243 -l cluster 
    -u elastic -P $ES_PWD
interval: 300
publish: true
runtime_assets:
  - sensu-plugins-elasticsearch
  - sensu-ruby-runtime
  secrets:
  - name: ES_PWD
    secret: es_pwd_logging_and_metric
subscriptions:
  - linux
```

---

<div class="post-metadata">

**Author:** ![raulgs](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/raulgs/32/323_2.png) [@raulgs](https://discourse.sensu.io/u/raulgs)\
**Post date:** [June 3, 2020, 2:59pm UTC](https://discourse.sensu.io/t/secrets-management-on-kubernetes-with-env-provider/1841/6 "2020-06-03T14:59:34Z")

</div>

Just for my understanding Do these env variables need to be available on the backend or the agents?  
I currently do have a check where I have exposed secrets using env variables on the agent.  
If backend only is fine I will move them there.

---

<div class="post-metadata">

**Author:** ![calebhailey](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/calebhailey/32/46_2.png) [@calebhailey](https://discourse.sensu.io/u/calebhailey)\
**Post date:** [June 3, 2020, 4:10pm UTC](https://discourse.sensu.io/t/secrets-management-on-kubernetes-with-env-provider/1841/7 "2020-06-03T16:10:46Z")

</div>

@raulgs great question! The real value-add of Sensu’s built-in Secrets Management capabilities is that you only need to define the secrets on the backends. However, in order for Sensu to transmit those secrets to agents for use in checks, you must have secured agent-to-sever communication (TLS encrypted transport) _and_ be using mTLS agent authentication. This is explained in the Secrets reference documentation, [here](https://docs.sensu.io/sensu-go/latest/reference/secrets/):

> Secrets are only transmitted over a transport layer security (TLS) websocket connection. Unencrypted connections must not transmit privileged information. For checks, hooks, and assets, you must [enable mutual TLS (mTLS)](https://docs.sensu.io/sensu-go/latest/guides/securing-sensu/#sensu-agent-mtls-authentication). Sensu will not transmit secrets to agents that do not use mTLS.
> 
> Sensu only exposes secrets to Sensu services like environment variables and automatically redacts secrets from all logs, the API, and the dashboard.

This guide covers both topics:

> **[Secure Sensu | Sensu Go 6.10](https://docs.sensu.io/sensu-go/latest/operations/deploy-sensu/secure-sensu/)**
>
> Secure components like the Sensu API and web UI and agent-to-server communication and configure agent mTLS authentication to make Sensu production-ready.

I hope this helps!

---

<div class="post-metadata">

**Author:** ![seizste](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/seizste/32/343_2.png) [@seizste](https://discourse.sensu.io/u/seizste)\
**Post date:** [June 3, 2020, 4:27pm UTC](https://discourse.sensu.io/t/secrets-management-on-kubernetes-with-env-provider/1841/8 "2020-06-03T16:27:03Z")

</div>

@calebhailey … a follow up question, where can i use secrets to replace something in the config of a check or handler … or asked differently, is the following code valid ?

```
type: Handler
api_version: core/v2
metadata:
  name: dnscheck-1486439-prod
spec:
  command: sensu-go-elasticsearch --index metrics-sensu-dnscheck-1486439-prod --full_event_logging
  env_vars:
    - $ES_SENSU_CLOUD_ID
  runtime_assets:
    - sensu-go-elasticsearch
  secrets:
    - name: ES_SENSU_CLOUD_ID
      secret: es_pwd_1486439_prod
  timeout: 0
  type: pipe
```

---

<div class="post-metadata">

**Author:** ![calebhailey](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/calebhailey/32/46_2.png) [@calebhailey](https://discourse.sensu.io/u/calebhailey)\
**Post date:** [June 3, 2020, 9:32pm UTC](https://discourse.sensu.io/t/secrets-management-on-kubernetes-with-env-provider/1841/9 "2020-06-03T21:32:42Z")

</div>

Hey @seizste 👋

You’re on the right track! A few observations/comments:

- Sensu Secrets are directly exposed as environment variables, so you **do not** need to set an environment variable _in addition to_ the secret in the handler definition (in your example handler definition you have an `env_var` _and_ a `secret` called `ES_SENSU_CLOUD_ID`, which is not necessarily an invalid configuration, but it’s most likely going to cause some conflict).

- Because Sensu Secrets read secret values and expose them to the check or handler as environment variables, the plugin/command being executed by Sensu either needs to implicitly read those environment variables, or they must be passed into the command via a command line argument; for example:

- It looks like the [Elasticsearch plugin you’re using](https://bonsai.sensu.io/assets/jkerry/sensu-go-elasticsearch) supports a single environment variable (`ELASTICSEARCH_URL `) with a value like `https://user:pass@hostname:port`. It appears that your Kubernetes secret only contains the Elasticsearch “password”, so you’ll either need to construct the `ELASTICSEARCH_URL` in the handler command, or modify the contents of your secret and expose the secret as `ELASTICSEARCH_URL` instead of `ES_SENSU_CLOUD_ID`

Here’s a few example Sensu secret + handler definitions to help explain what I mean by this last comment:

### Pass secret an explicit environment variable in the command

```auto
---
type: Secret
api_version: secrets/v1
metadata:
  name: es_pwd_1486439_prod
spec:
  provider: env
  id: SENSU_ES_PWD_LOGGING_AND_METRIC
---
type: Handler
api_version: core/v2
metadata:
  name: dnscheck-1486439-prod
spec:
  command: >-
    ELASTICSEARCH_URL="https://admin:${ES_SENSU_CLOUD_ID}@elasticsearch.yourdomain.com:9243" &&
    sensu-go-elasticsearch --index metrics-sensu-dnscheck-1486439-prod --full_event_logging
  runtime_assets:
    - sensu-go-elasticsearch
  secrets:
    - name: ES_SENSU_CLOUD_ID
      secret: es_pwd_1486439_prod
  timeout: 0
  type: pipe

```

### Pass secret as implicit environment variable

```auto
---
type: Secret
api_version: secrets/v1
metadata:
  name: sensu_es_url
spec:
  provider: env
  id: SENSU_ES_URL
---
type: Handler
api_version: core/v2
metadata:
  name: dnscheck-1486439-prod
spec:
  command: sensu-go-elasticsearch --index metrics-sensu-dnscheck-1486439-prod --full_event_logging
  runtime_assets:
    - sensu-go-elasticsearch
  secrets:
    - name: ELASTICSEARCH_URL
      secret: sensu_es_url
  timeout: 0
  type: pipe

```

> _ **NOTE:** In this latter example, you’d need to modify your Kubernetes secret so that the value of the secret is in the format of: `https://user:pass@hostname:port`._

I hope this helps!

---

<div class="post-metadata">

**Author:** ![seizste](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/seizste/32/343_2.png) [@seizste](https://discourse.sensu.io/u/seizste)\
**Post date:** [June 4, 2020, 7:10am UTC](https://discourse.sensu.io/t/secrets-management-on-kubernetes-with-env-provider/1841/10 "2020-06-04T07:10:03Z")

</div>

Thanks a lot @calebhailey … i went for the second option and it works like a charm !

---

<div class="post-metadata">

**Author:** ![calebhailey](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/calebhailey/32/46_2.png) [@calebhailey](https://discourse.sensu.io/u/calebhailey)\
**Post date:** [June 11, 2020, 7:13pm UTC](https://discourse.sensu.io/t/secrets-management-on-kubernetes-with-env-provider/1841/11 "2020-06-11T19:13:34Z")

</div>

Awesome! Glad to hear it!