Docker image init not consistent

Sensu Go
1

Hello,

I’ve adjusted the way we’re going to be deploying sensu go, and have started writing some configuration to begin using the docker images. Given that setting this up with Docker is relatively simple, I’m at a loss for the behavior I’m seeing.

On my workstation if I pull the image and start the container the entrypoint takes over correctly and I’m able to log in with the default admin credentials. Curling the health check returns good information and the backend responds readily to configuration.

Starting up a vagrant image based on Ubuntu 18.04 the container will start and it seems the processes are running correctly, but health checks do not return and some portion of the init is not working, as once the container starts and I’m able to access the web UI it will not accept the admin credentials specified in the entrypoint.

Docker Desktop on Mac OSX

REMC02XX71UJG5J:~ mcnultt$ docker exec -ti a55a1ed35dfc ps faux
PID   USER     TIME  COMMAND
    1 root      0:00 {sensu-backend} /usr/bin/dumb-init sh /usr/local/bin/sensu
    6 root      0:00 sh /usr/local/bin/sensu-backend start --state-dir /var/lib
    9 root      0:01 /opt/sensu/bin/sensu-backend start --state-dir /var/lib/se
   45 root      0:00 ps faux

Docker Engine - Community 19.03.6 on Ubuntu 18.04

root@default-ubuntu-1804:~# docker exec -ti 38aaf30597bb ps
PID   USER     TIME  COMMAND
    1 root      0:00 {sensu-backend} /usr/bin/dumb-init sh /usr/local/bin/sensu
    6 root      0:00 sh /usr/local/bin/sensu-backend start --state-dir /var/lib
    9 root      0:01 /opt/sensu/bin/sensu-backend start --state-dir /var/lib/se
   27 root      0:00 ps

Curling the health endpoint on this machine returns a variety of errors, sometimes it does not return at all and times out.

root@default-ubuntu-1804:~# curl -X GET http://127.0.0.1:8080/health
curl: (52) Empty reply from server

root@default-ubuntu-1804:~# curl -X GET http://127.0.0.1:8080/healthd
{"code":2,"message":"not found"}

root@default-ubuntu-1804:~# curl -X GET http://127.0.0.1:8080/health
curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused
2

The Ubuntu based image you are spinning up, are you installing and starting the Sensu backend directly?

As of version 5.16.0, the Sensu backend is not initialized by default see (release notes). If you install and start up a sensu-backend process, you need to initialize it.

This is done automatically in the Sensu provided docker images.

3

I’m sorry, I should’ve specified I’m using your provided images. Sensu/sensu:latest. The ubuntu box is simply the platform on which docker is controlling the containers.

4

A little more troubleshooting.

root@default-ubuntu-1804:~# docker exec -it e502c266dd73 printenv
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=e502c266dd73
TERM=xterm
SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=admin
SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=P@ssw0rd!
HOME=/root


root@default-ubuntu-1804:~# docker exec -it e502c266dd73 /opt/sensu/bin/sensu-backend init
{"component":"sensu-enterprise","error":"error connecting to cluster: context deadline exceeded","level":"fatal","msg":"error executing sensu-backend","time":"2020-02-24T18:39:51Z"}
5

I just spun up an Ubuntu 18.04 box in Vagrant, loaded up Docker CE and I’m not able to replicate.

vagrant@ubuntu-bionic:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
Codename:	bionic

vagrant@ubuntu-bionic:~$ docker version
    Client: Docker Engine - Community
     Version:           19.03.6
     API version:       1.40
     Go version:        go1.12.16
     Git commit:        369ce74a3c
     Built:             Thu Feb 13 01:27:49 2020
     OS/Arch:           linux/amd64
     Experimental:      false

    Server: Docker Engine - Community
     Engine:
      Version:          19.03.6
      API version:      1.40 (minimum version 1.12)
      Go version:       go1.12.16
      Git commit:       369ce74a3c
      Built:            Thu Feb 13 01:26:21 2020
      OS/Arch:          linux/amd64
      Experimental:     false
     containerd:
      Version:          1.2.10
      GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
     runc:
      Version:          1.0.0-rc8+dev
      GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
     docker-init:
      Version:          0.18.0
      GitCommit:        fec3683

vagrant@ubuntu-bionic:~$ curl -s -X GET http://127.0.0.1:8080/health | jq .
{
  "Alarms": null,
  "ClusterHealth": [
    {
      "MemberID": 9882886658148555000,
      "MemberIDHex": "8927110dc66458af",
      "Name": "default",
      "Err": "",
      "Healthy": true
    }
  ],
  "Header": {
    "cluster_id": 4255616304056077000,
    "member_id": 9882886658148555000,
    "raft_term": 2
  }
}
6 
root@default-ubuntu-1804:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
Codename:	bionic
root@default-ubuntu-1804:~#
root@default-ubuntu-1804:~# docker version
Client: Docker Engine - Community
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.12.16
 Git commit:        369ce74a3c
 Built:             Thu Feb 13 01:27:49 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.16
  Git commit:       369ce74a3c
  Built:            Thu Feb 13 01:26:21 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Versions match exactly, I have no idea what could be causing this. I’m not adjusting a single thing. It must be the infrastructure surrounding it.

7

I did leave this out, but I assume it to be the same as well.

vagrant@ubuntu-bionic:~$ docker ps -a
CONTAINER ID        IMAGE                COMMAND                 CREATED             STATUS              PORTS                                                                     NAMES
9b8c9d472df0        sensu/sensu:5.17.2   "sensu-backend start"   7 minutes ago       Up 7 minutes        0.0.0.0:3000->3000/tcp, 2379-2380/tcp, 8081/tcp, 0.0.0.0:8080->8080/tcp   sensu-backend
8

Indeed, though I’m trying with pinning 5.17.2

root@default-ubuntu-1804:~# docker ps -a
CONTAINER ID        IMAGE                COMMAND                 CREATED             STATUS              PORTS                                                                     NAMES
93f78ccc96a0        sensu/sensu:latest   "sensu-backend start"   5 minutes ago       Up 5 minutes        0.0.0.0:3000->3000/tcp, 2379-2380/tcp, 0.0.0.0:8080-8081->8080-8081/tcp   sensu-backend
9

Same behavior.

Service up in the container but not returning health endpoint and can’t log in. Going to try the rhel based image.

10

RHEL image works as documented.

11

That’s definitely strange. I’ve been stopping/starting instances of that image and can’t get it to fail.

12

Just noticed a typo in one of the example /health endpoint requests (requesting an invalid /healthd endpoint). Just FYI.