Assistance requested on deploying a handler to Sensu Go backend container in Kubernetes

KendallChenoweth · July 4, 2019, 3:31pm

I have a 5 node sensu Go Backend cluster in kubernetes (described last below). I can use an asset to deploy a check script resource. That check script can invoke a handler. I can define the handler asset and upload a handler definition. I confused on how to use an asset to to deploy an actual handler to the sensu backend container.

It’s my understanding that the the asset, when deployed, should exist on the sensu-backend container in /var/cache/sensu/sensu-agent. I’m not seeing the sensu-agent directory on the sensu-backend container. (The Pod for the sensu-backend also includes another container for the sensu-agent. I assume I need this in order to deploy the asset.)

I have a 5 node sensu Go Backend cluster in kubernetes (described last below). I can use an asset to deploy a check script resource. That check script can invoke a handler. I can define the handler asset and upload a handler definition. I confused on how to use an asset to to deploy an actual handler to the sensu backend container.

It’s my understanding that the the asset, when deployed, should exist on the sensu-backend container in /var/cache/sensu/sensu-agent. I’m not seeing the sensu-agent directory on the sensu-backend container. (The Pod for the sensu-backend also includes another container for the sensu-agent. I assume I need this in order to deploy the asset.)

sensuctl entity list
        ID          Class    OS                Subscriptions                         Last Seen            
 ───────────────── ─────── ─────── ────────────────────────────────────── ─────────────────────────────── 
  kokopelli         agent   linux   system,entity:kokopelli                2019-07-04 11:29:50 -0400 EDT  
  sensu-backend-1   agent   linux   sensu-backend,entity:sensu-backend-1   2019-07-04 11:29:47 -0400 EDT  
  sensu-backend-2   agent   linux   sensu-backend,entity:sensu-backend-2   2019-07-04 11:29:45 -0400 EDT  
  sensu-backend-3   agent   linux   sensu-backend,entity:sensu-backend-3   2019-07-04 11:29:46 -0400 EDT  
  sensu-backend-4   agent   linux   sensu-backend,entity:sensu-backend-4   2019-07-04 11:29:45 -0400 EDT  
  sensu-backend-5   agent   linux   sensu-backend,entity:sensu-backend-5   2019-07-04 11:29:48 -0400 EDT

type: Asset
api_version: core/v2
metadata:
  name: sensu-email-handler
  namespace: default
  labels: {}
  annotations: {}
spec:
  url: https://github.com/sensu/sensu-email-handler/releases/download/0.1.0/sensu-email-handler_0.1.0_linux_amd64.tar.gz
  sha512: 755c7a673d94997ab9613ec5969666e808f8b4a8eec1ba998ee7071606c96946ca2947de5189b24ac34a962713d156619453ff7ea43c95dae62bf0fcb
e766f2e
  filters:
  - entity.system.os == 'linux'
  - entity.system.arch == 'amd64'



api_version: core/v2
type: Handler
metadata:
  namespace: demo
  name: email
spec:
  type: pipe
  runtime_assets:
  - sensu-email-handler
  command: sensu-email-handler -f sensu@erwin.cyberbears.net -t kendall@erwin.cyberbears.net -s erwin.cyberbear
s.net 
  timeout: 30
  filters:
  - is_incident
  - not_silenced


sensuctl check create check-cpu \
--command 'check-cpu.rb -w 5 -c 15' \
--interval 60 \
--subscriptions system \
--runtime-assets sensu-plugins-cpu-checks,sensu-ruby-runtime \
--handlers email

sensuctl asset list
            Name                                                URL                                       Hash    
 ────────────────────────── ─────────────────────────────────────────────────────────────────────────── ───────── 
  sensu-email-handler        //github.com/.../sensu-email-handler_0.1.0_linux_amd64.tar.gz               755c7a6  
  sensu-influxdb-handler     //github.com/.../sensu-influxdb-handler_3.1.2_linux_amd64.tar.gz            612c6ff  
  sensu-plugins-cpu-checks   //github.com/.../sensu-plugins-cpu-checks_4.0.0_centos_linux_amd64.tar.gz   518e7c1  
  sensu-ruby-runtime         //github.com/.../sensu-ruby-runtime_0.0.5_centos_linux_amd64.tar.gz         1c9f0af

sensuctl handler list
    Name      Type   Timeout           Filters            Mutator                                                Execute                                                                                          Environment Variables                                                    Assets          
 ─────────── ────── ───────── ────────────────────────── ───────── ──────────────────────────────────────────────────────────────────────────────────────────────────── ───────────────────────────────────────────────────────────────────────────────────────────────────────── ──────────────────────── 
  email       pipe        10   is_incident,not_silenced             RUN:  sensu-email-handler -f sensu@cyberbears.net -t kendall@cyberbears.net -s mail.cyberbears.net                                                                                                                                     
  influx-db   pipe         0                                        RUN:  sensu-influxdb-handler -d sensu                                                                INFLUXDB_ADDR=http://influxdb.default.svc.cluster.local:8086,INFLUXDB_USER=sensu,INFLUXDB_PASS=password   sensu-influxdb-handler 

sensuctl check info check-cpu
=== check-cpu
Name:              check-cpu
Interval:          60
Command:           check-cpu.rb -w 5 -c 15
Cron:              
Timeout:           0
TTL:               0
Subscriptions:     system
Handlers:          email
Runtime Assets:    sensu-plugins-cpu-checks, sensu-ruby-runtime
Hooks:             
Publish?:          true
Stdin?:            false
Proxy Entity Name: 
Namespace:         default
Metric Format:     
Metric Handlers:


apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: sensu-backend-1
spec:
  selector:
    matchLabels:
      app: sensu-backend-1
  template:
    metadata:
      labels:
        app: sensu-backend-1
    spec:
      containers:
      - name: sensu-backend-server-1
        image: sensu/sensu:5.10.2
        command: ["/opt/sensu/bin/sensu-backend","start","--log-level","info","--etcd-listen-client-urls","http://0.0.0.0:2379","--
etcd-name","backend-1","--etcd-advertise-client-urls","http://sensu-backend-1.default.svc.cluster.local:2379","--etcd-initial-clust
er","backend-1=http://sensu-backend-1.default.svc.cluster.local:2380,backend-2=http://sensu-backend-2.default.svc.cluster.local:238
0,backend-3=http://sensu-backend-3.default.svc.cluster.local:2380,backend-4=http://sensu-backend-4.default.svc.cluster.local:2380,,
backend-5=http://sensu-backend-5.default.svc.cluster.local:2380","--etcd-initial-cluster-state","new","--etcd-initial-advertise-pee
r-urls","http://sensu-backend-1.default.svc.cluster.local:2380","--state-dir","/var/lib/sensu/sensu-backend/etcd1","--etcd-listen-p
eer-urls","http://0.0.0.0:2380"]
        ports:
        - name: api
          protocol: TCP
          containerPort: 8080
        - name: agent
          protocol: TCP
          containerPort: 8081
        - name: web
          protocol: TCP
          containerPort: 3000
        - name: client
          protocol: TCP
          containerPort: 2379
        - name: peer
          protocol: TCP
          containerPort: 2380
        - name: mystery 
          protocol: TCP
          containerPort: 6060
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 20  
      - name: sensu-backend-agent-1
        image: sensu/sensu:5.10.2
        command: ["sensu-agent","start","--backend-url","ws://10.244.0.210:8081","--subscriptions","sensu-backend","--name","sensu-
backend-1"]

 
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: sensu-backend-2
spec:
  selector:
    matchLabels:
      app: sensu-backend-2
  template:
    metadata:
      labels:
        app: sensu-backend-2
    spec:
      containers:
      - name: sensu-backend-2
        image: sensu/sensu:5.10.2
        command: ["/opt/sensu/bin/sensu-backend","start","--log-level","info","--etcd-listen-client-urls","http://0.0.0.0:2379","--
etcd-name","backend-2","--etcd-advertise-client-urls","http://sensu-backend-2.default.svc.cluster.local:2379","--etcd-initial-clust
er","backend-1=http://sensu-backend-1.default.svc.cluster.local:2380,backend-2=http://sensu-backend-2.default.svc.cluster.local:238
0,backend-3=http://sensu-backend-3.default.svc.cluster.local:2380,backend-4=http://sensu-backend-4.default.svc.cluster.local:2380,,
backend-5=http://sensu-backend-5.default.svc.cluster.local:2380","--etcd-initial-cluster-state","new","--etcd-initial-advertise-pee
r-urls","http://sensu-backend-2.default.svc.cluster.local:2380","--state-dir","/var/lib/sensu/sensu-backend/etcd1","--etcd-listen-p
eer-urls","http://0.0.0.0:2380"]
        ports:
        - name: api
          protocol: TCP
          containerPort: 8080
        - name: agent
          protocol: TCP
          containerPort: 8081
        - name: web
          protocol: TCP
          containerPort: 3000
        - name: client
          protocol: TCP
          containerPort: 2379
        - name: peer
          protocol: TCP
          containerPort: 2380
        - name: mystery 
          protocol: TCP
          containerPort: 6060
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 20  
      - name: sensu-backend-agent-2
        image: sensu/sensu:5.10.2
        command: ["sensu-agent","start","--backend-url","ws://10.244.0.210:8081","--subscriptions","sensu-backend","--name","sensu-
backend-2"]

---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: sensu-backend-3
spec:
  selector:
    matchLabels:
      app: sensu-backend-3
  template:
    metadata:
      labels:
        app: sensu-backend-3
    spec:
      containers:
      - name: sensu-backend-3
        image: sensu/sensu:5.10.2
        command: ["/opt/sensu/bin/sensu-backend","start","--log-level","info","--etcd-listen-client-urls","http://0.0.0.0:2379","--
etcd-name","backend-3","--etcd-advertise-client-urls","http://sensu-backend-3.default.svc.cluster.local:2379","--etcd-initial-clust
er","backend-1=http://sensu-backend-1.default.svc.cluster.local:2380,backend-2=http://sensu-backend-2.default.svc.cluster.local:238
0,backend-3=http://sensu-backend-3.default.svc.cluster.local:2380,backend-4=http://sensu-backend-4.default.svc.cluster.local:2380,,
backend-5=http://sensu-backend-5.default.svc.cluster.local:2380","--etcd-initial-cluster-state","new","--etcd-initial-advertise-pee
r-urls","http://sensu-backend-3.default.svc.cluster.local:2380","--state-dir","/var/lib/sensu/sensu-backend/etcd1","--etcd-listen-p
eer-urls","http://0.0.0.0:2380"]
        ports:
        - name: api
          protocol: TCP
          containerPort: 8080
        - name: agent
          protocol: TCP
          containerPort: 8081
        - name: web
          protocol: TCP
          containerPort: 3000
        - name: client
          protocol: TCP
          containerPort: 2379
        - name: peer
          protocol: TCP
          containerPort: 2380
        - name: mystery 
          protocol: TCP
          containerPort: 6060
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 20  
      - name: sensu-backend-agent-3
        image: sensu/sensu:5.10.2
        command: ["sensu-agent","start","--backend-url","ws://10.244.0.210:8081","--subscriptions","sensu-backend","--name","sensu-
backend-3"]

---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: sensu-backend-4
spec:
  selector:
    matchLabels:
      app: sensu-backend-4
  template:
    metadata:
      labels:
        app: sensu-backend-4
    spec:
      containers:
      - name: sensu-backend-4
        image: sensu/sensu:5.10.2
        command: ["/opt/sensu/bin/sensu-backend","start","--log-level","info","--etcd-listen-client-urls","http://0.0.0.0:2379","--
etcd-name","backend-4","--etcd-advertise-client-urls","http://sensu-backend-4.default.svc.cluster.local:2379","--etcd-initial-clust
er","backend-1=http://sensu-backend-1.default.svc.cluster.local:2380,backend-2=http://sensu-backend-2.default.svc.cluster.local:238
0,backend-3=http://sensu-backend-3.default.svc.cluster.local:2380,backend-4=http://sensu-backend-4.default.svc.cluster.local:2380,,
backend-5=http://sensu-backend-5.default.svc.cluster.local:2380","--etcd-initial-cluster-state","new","--etcd-initial-advertise-pee
r-urls","http://sensu-backend-4.default.svc.cluster.local:2380","--state-dir","/var/lib/sensu/sensu-backend/etcd1","--etcd-listen-p
eer-urls","http://0.0.0.0:2380"]
        ports:
        - name: api
          protocol: TCP
          containerPort: 8080
        - name: agent
          protocol: TCP
          containerPort: 8081
        - name: web
          protocol: TCP
          containerPort: 3000
        - name: client
          protocol: TCP
          containerPort: 2379
        - name: peer
          protocol: TCP
          containerPort: 2380
        - name: mystery 
          protocol: TCP
          containerPort: 6060
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 20  
      - name: sensu-backend-agent-4
        image: sensu/sensu:5.10.2
        command: ["sensu-agent","start","--backend-url","ws://10.244.0.210:8081","--subscriptions","sensu-backend","--name","sensu-
backend-4"]

---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: sensu-backend-5
spec:
  selector:
    matchLabels:
      app: sensu-backend-5
  template:
    metadata:
      labels:
        app: sensu-backend-5
    spec:
      containers:
      - name: sensu-backend-5
        image: sensu/sensu:5.10.2
        command: ["/opt/sensu/bin/sensu-backend","start","--log-level","info","--etcd-listen-client-urls","http://0.0.0.0:2379","--
etcd-name","backend-5","--etcd-advertise-client-urls","http://sensu-backend-5.default.svc.cluster.local:2379","--etcd-initial-clust
er","backend-1=http://sensu-backend-1.default.svc.cluster.local:2380,backend-2=http://sensu-backend-2.default.svc.cluster.local:238
0,backend-3=http://sensu-backend-3.default.svc.cluster.local:2380,backend-4=http://sensu-backend-4.default.svc.cluster.local:2380,,
backend-5=http://sensu-backend-5.default.svc.cluster.local:2380","--etcd-initial-cluster-state","new","--etcd-initial-advertise-pee
r-urls","http://sensu-backend-5.default.svc.cluster.local:2380","--state-dir","/var/lib/sensu/sensu-backend/etcd1","--etcd-listen-p
eer-urls","http://0.0.0.0:2380"]
        ports:
        - name: api
          protocol: TCP
          containerPort: 8080
        - name: agent
          protocol: TCP
          containerPort: 8081
        - name: web
          protocol: TCP
          containerPort: 3000
        - name: client
          protocol: TCP
          containerPort: 2379
        - name: peer
          protocol: TCP
          containerPort: 2380
        - name: mystery 
          protocol: TCP
          containerPort: 6060
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 20  
      - name: sensu-backend-agent-5
        image: sensu/sensu:5.10.2
        command: ["sensu-agent","start","--backend-url","ws://10.244.0.210:8081","--subscriptions","sensu-backend","--name","sensu-
backend-5"]

---
apiVersion: v1
kind: Service
metadata:
  name: sensu-backend-1
spec:
  ports:
  - name: api
    port: 8080 
    protocol: TCP
    targetPort: 8080
  - name: agent
    port: 8081
    protocol: TCP
    targetPort: 8081
  - name: web
    port: 3000
    protocol: TCP
    targetPort: 3000
  - name: client
    port: 2379 
    protocol: TCP
    targetPort: 2379
  - name: peer 
    port: 2380 
    protocol: TCP
    targetPort: 2380
  - name: mystery
    port: 6060 
    protocol: TCP
    targetPort: 6060
  selector:
    app: sensu-backend-1

---
apiVersion: v1
kind: Service
metadata:
  name: sensu-backend-2
spec:
  ports:
  - name: api
    port: 8080 
    protocol: TCP
    targetPort: 8080
  - name: agent
    port: 8081
    protocol: TCP
    targetPort: 8081
  - name: web
    port: 3000
    protocol: TCP
    targetPort: 3000
  - name: client
    port: 2379 
    protocol: TCP
    targetPort: 2379
  - name: peer 
    port: 2380 
    protocol: TCP
    targetPort: 2380
  - name: mystery
    port: 6060 
    protocol: TCP
    targetPort: 6060
  selector:
    app: sensu-backend-2

---
apiVersion: v1
kind: Service
metadata:
  name: sensu-backend-3
spec:
  ports:
  - name: api
    port: 8080 
    protocol: TCP
    targetPort: 8080
  - name: agent
    port: 8081
    protocol: TCP
    targetPort: 8081
  - name: web
    port: 3000
    protocol: TCP
    targetPort: 3000
  - name: client
    port: 2379 
    protocol: TCP
    targetPort: 2379
  - name: peer 
    port: 2380 
    protocol: TCP
    targetPort: 2380
  - name: mystery
    port: 6060 
    protocol: TCP
    targetPort: 6060
  selector:
    app: sensu-backend-3

---
apiVersion: v1
kind: Service
metadata:
  name: sensu-backend-4
spec:
  ports:
  - name: api
    port: 8080 
    protocol: TCP
    targetPort: 8080
  - name: agent
    port: 8081
    protocol: TCP
    targetPort: 8081
  - name: web
    port: 3000
    protocol: TCP
    targetPort: 3000
  - name: client
    port: 2379 
    protocol: TCP
    targetPort: 2379
  - name: peer 
    port: 2380 
    protocol: TCP
    targetPort: 2380
  - name: mystery
    port: 6060 
    protocol: TCP
    targetPort: 6060
  selector:
    app: sensu-backend-4

---
apiVersion: v1
kind: Service
metadata:
  name: sensu-backend-5
spec:
  ports:
  - name: api
    port: 8080 
    protocol: TCP
    targetPort: 8080
  - name: agent
    port: 8081
    protocol: TCP
    targetPort: 8081
  - name: web
    port: 3000
    protocol: TCP
    targetPort: 3000
  - name: client
    port: 2379 
    protocol: TCP
    targetPort: 2379
  - name: peer 
    port: 2380 
    protocol: TCP
    targetPort: 2380
  - name: mystery
    port: 6060 
    protocol: TCP
    targetPort: 6060
  selector:
    app: sensu-backend-5

---
apiVersion: v1
kind: Service
metadata: 
  name: sensu-backend
spec:
  ports:
    - name: web
      port: 80
      protocol: TCP
      targetPort: 3000
    - name: agent
      port: 8081
      protocol: TCP
      targetPort: 8081
    - name: api
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    app: sensu-backend-1
    app: sensu-backend-2
    app: sensu-backend-3
    app: sensu-backend-4
    app: sensu-backend-5
  loadBalancerIP: 10.244.0.210
  type: LoadBalancer


---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: sensu-ingress-web
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: sensu-backend
          servicePort: 80

KendallChenoweth · July 4, 2019, 4:21pm

I setup a shared volume on the /var/cache/sensu directory and I now see both sensu-agent and sensu-backend directories in the pod. This didn’t seem to make a difference; do I need to do this to make this work?

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: sensu-backend-1
spec:
  selector:
    matchLabels:
      app: sensu-backend-1
  template:
    metadata:
      labels:
        app: sensu-backend-1
    spec:
      volumes:
      - name: shared-data
        emptyDir: {}
      containers:
      - name: sensu-backend-server-1
        volumeMounts:
        - name: shared-data
          mountPath: /var/cache/sensu
        image: sensu/sensu:5.10.2
        command: ["/opt/sensu/bin/sensu-backend","start","--log-level","info","--etcd-listen-client-urls","http://0.0.0.0:2379","--
etcd-name","backend-1","--etcd-advertise-client-urls","http://sensu-backend-1.default.svc.cluster.local:2379","--etcd-initial-clust
er","backend-1=http://sensu-backend-1.default.svc.cluster.local:2380,backend-2=http://sensu-backend-2.default.svc.cluster.local:238
0,backend-3=http://sensu-backend-3.default.svc.cluster.local:2380,backend-4=http://sensu-backend-4.default.svc.cluster.local:2380,,
backend-5=http://sensu-backend-5.default.svc.cluster.local:2380","--etcd-initial-cluster-state","new","--etcd-initial-advertise-pee
r-urls","http://sensu-backend-1.default.svc.cluster.local:2380","--state-dir","/var/lib/sensu/sensu-backend/etcd1","--etcd-listen-p
eer-urls","http://0.0.0.0:2380"]
        ports:
        - name: api
          protocol: TCP
          containerPort: 8080
        - name: agent
          protocol: TCP
          containerPort: 8081
        - name: web
          protocol: TCP
          containerPort: 3000
        - name: client
          protocol: TCP
          containerPort: 2379
        - name: peer
          protocol: TCP
          containerPort: 2380
        - name: mystery 
          protocol: TCP
          containerPort: 6060
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 20  
      - name: sensu-backend-agent-1
        volumeMounts:
        - name: shared-data
          mountPath: /var/cache/sensu
        image: sensu/sensu:5.10.2
        command: ["sensu-agent","start","--backend-url","ws://10.244.0.210:8081","--subscriptions","sensu-backend","--name","sensu-
backend-1"]

KendallChenoweth · July 6, 2019, 10:07am

I’m beginning to think that the problem is because the asset and sensu-backend filters don’t line up. I looked up the configuration on the kube demo, but the email asset there doesn’t define any filters.

If I’m on the right track, can you provide me with an sensu backend entity attributes, the handler asset attributes and the email handler definition. If I have a working set, than I can probably move past this obsticale.

Thanks!

jspaleta · July 9, 2019, 10:12pm

First is the check check actually running? do you see an event corresponding the the check in sensuctl event list?

The cache volume mount isn’t necessary it can be useful to cache asset downloads to reduce network activity…but not necessary.

In the backend containers handler assets that are used should show up
in “/var/cache/sensu/sensu-backend”

If the event exists, check the sensu backend logs with kubectl logs for asset errors. Likely reasons why an asset would not populate into the cache include:

asset download failed -> this will leave an artifact in /tmp/ or /var/tmp in the container… if the download was cut off part way through and will include a log error.
asset sha512sum did not verify -> error will be logged
asset entity filters don’t match backend entity information - > error will be logged

KendallChenoweth · July 9, 2019, 11:24pm

I found the error in the log. Thanks for the debugging information.

I really appreciate your help and will share the successful experience I had with my manager.

Have a great day!

-Kendall Chenoweth

Topic		Replies	Views
How to use the sensu-plugins-kubernetes plugin Sensu Plugins	2	1709	July 31, 2019
How can we deploy sensu-go server cluster on Kubernetes? Sensu Go	2	912	November 25, 2019
Sensu Go plugin distribution model Sensu Go	17	1397	June 15, 2020
Checks using asset not assigned, but named similarly Sensu Go sensu-go , assets	2	340	March 1, 2022
Asset not found Sensu Go	2	822	January 26, 2020

Assistance requested on deploying a handler to Sensu Go backend container in Kubernetes

Related topics