For Discussion: Strategies for provisioning Sensu resources in freshly generated backend container

Sensu Go
1

There was a very interesting question from a user on slack. I thought it would be good to archive the question here and more importantly capture some ideas for strategies for how to accomplish the task.

Here’s the user’s original comment:
"I’m trying to simplify my docker entrypoint since my current one involves having a simultaneous script wait for the backend to start and then run sensuctl configure/create commands

The fundamental question:
Is it possible to initialize sensu-backend with resources when the backend container(s) is(are) started?

My answer:
Unfortunately, no. Because the backend has to wait for the etcd store to initialize, it’s not possible to add resources in one step. And when using a clustered backend with embedded etcd, all the backend’s have to start and negotiate the etcd quorum process before any resources can be injected into the etcd store.

In fact as of Sensu Go 5.16, even the official Docker image uses an entry point script with that waits for the backend’s etcd store to become available before issuing sensu-backend init to set the initial admin user and password.

Mitigation Strategies:
Just the top of my head, here’s my ideas on how to mitigate this and achieve resource creation as simply as possible.

  1. Use docker compose and dockerize to run the sensuctl command. The dockerize project is pretty clever, allowing you to wait to run a container command until a specific service endpoint is available. This approach has the benefit of working well with a clustered sensu-backend. The downside is, it’s still a race with backend init. Can you be sure the sensuctl will run after the needed admin password is set?

  2. Extend the entry point logic wait loop that is calling sensu-backend init to also optionally call sensuctl create -r if a particular directory is mounted into the container that contains resource files. The benefit here is the admin and password are already available to the entry script in order to run the backend init. It would be interesting to see someone make an attempt at this by contributing a Dockerfile here for people to test. I’m not how this would work in a clustered backend. If each backend is trying to create the same resources, that’s unneeded duplication of effort. The benefit is, you can be sure the sensuctl is run after the backend init.

Any other ideas?

2

Hi! I’m the user!

I think I’ve so far decided on this technique: a secondary container as the “API Caller”, which calls a script full of sensuctl commands.

version: '3'

services:
  sensuAdmin:
    container_name: sensuctl
    image: "sensu/sensu:5.14.0"
    restart: always
    volumes:
      - ./configs/sensu-backend-config/:/etc/sensu
      - ./configs/nginx-config/certs:/etc/certs
    command: /bin/sh -c /etc/sensu/sensu-backend.sh
  sensuBackend:
    extra_hosts:
      - "monitoring.site.com:127.0.0.1"
    container_name: monitoring.site.com
    image: "sensu/sensu:5.14.0"
    restart: always
    ports:
      - "4000:3000"
      - "2380:2380"
      - "443:8081"
      - "9999:8080"
    volumes:
      - ./configs/sensu-backend-config/:/etc/sensu
      - ./configs/nginx-config/certs:/etc/certs
    command: sensu-backend start

The backend starts, and a second container runs API calls, and reruns on exit (i.e. failure), giving a retry system that doesn’t depend backgrounding commands. There’s probably less silly ways to do this. Because CTL seems to be idempotent, it’s working so far…

3

Hey!

Look at dockerize,

It adds the ability to delay command execution waiting for the sensu-backend api socket to become available for connection.

Sequential container ordering still one of the harder problems to solve in the space.
Some days I sort of wish I had access to systemd declative dependency language inside docker and k8s to include support for socket activation and socket hand-off.

I realize that systemd gets a bad rap, but on my linux development workstation, its easy for me to run system and user services as lightweight containers using podman run from systemd services and express the interdependence conditions accurately.

k8s tries to give us some control with a concept of an init container…but the lack of mature socket activation and more importantly handoff…still forces us to write containers with complicated init looping to work around the problem.

1 Like