Sensu Go Remediation check not working

Sensu Go
1

Hey Guys,

I’m trying to create a remediation check. When I created it in aws ec2 instance it worked. Now I’m trying to do the same in an opsworks stack using chef. The check is created successfully. But it is not working.

I am providing my check configuration here.

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "check-nginx",
    "namespace": "default",
    "labels": {},
    "annotations": {
      "io.sensu.remediation.config.actions": "[\n  {\n    \"description\": \"Perform this action once after Nginx has been down for 30 seconds.\",\n    \"request\": \"remediate-nginx\",\n    \"occurrences\": [ 2 ],\n    \"severities\": [ 1,2,3 ]\n  }\n]\n"
    }
  },
  "spec": {
    "command": "/opt/sensu-plugins-ruby/embedded/bin/check-nginx-status.rb -u http://localhost:80",
    "handlers": [
      "remediation",
      "slack"
    ],
    "interval": 10,
    "publish": true,
    "subscriptions": [
      "system"
    ]
  }
}

Remediation handler :

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "remediation",
    "namespace": "default",
    "labels": {
      "sensu.io/managed_by": "sensuctl"
    },
    "created_by": "admin"
  },
  "spec": {
    "command": "sensu-remediation-handler",
    "env_vars": [
      "SENSU_API_URL=http://127.0.0.1:9090",
      "SENSU_API_USER=remediation-handler",
      "SENSU_API_PASS=password"
    ],
    "filters": null,
    "handlers": null,
    "runtime_assets": [
      "sensu-remediation-handler"
    ],
    "secrets": null,
    "timeout": 10,
    "type": "pipe"
  }
}

Can someone help me with this?

2

@ganeshkatakam can you provide log output please?

3

Sure @aaronsachs-sensu

Here is the log:

{"check":"check-nginx-agent-1","component":"agent","entity":"agent-1","event_uuid":"0209617c-3f8b-41b5-8f3a-e78bc4298c33","level":"info","msg":"sending event to backend","time":"2020-04-24T03:36:00Z"}

{"component":"agent","content_type":"application/octet-stream","level":"info","msg":"sending message","payload_size":1026,"time":"2020-04-24T03:36:00Z","type":"event"}

{"component":"transport","level":"debug","msg":"sending ping","time":"2020-04-24T03:36:00Z"}

{"component":"transport","level":"debug","msg":"pong received from the backend, setting the read deadline to 1587699405","time":"2020-04-24T03:36:00Z"}

{"component":"agent","content_type":"application/octet-stream","level":"info","msg":"message received","payload_size":436,"time":"2020-04-24T03:36:10Z","type":"check_request"}

{"component":"agent","level":"info","msg":"scheduling check execution: check-nginx-agent-1","time":"2020-04-24T03:36:10Z"}

{"assets":null,"check":"check-nginx-agent-1","component":"agent","level":"debug","msg":"fetching assets for check","namespace":"default","time":"2020-04-24T03:36:10Z"}

{"check":"check-nginx-agent-1","component":"agent","entity":"agent-1","event_uuid":"3f2a946a-879d-476f-8bc2-e5745c3adbdd","level":"info","msg":"sending event to backend","time":"2020-04-24T03:36:10Z"}

{"component":"agent","content_type":"application/octet-stream","level":"info","msg":"sending message","payload_size":1026,"time":"2020-04-24T03:36:10Z","type":"event"}

{"component":"agent","content_type":"application/octet-stream","level":"info","msg":"message received","payload_size":436,"time":"2020-04-24T03:36:20Z","type":"check_request"}

{"component":"agent","level":"info","msg":"scheduling check execution: check-nginx-agent-1","time":"2020-04-24T03:36:20Z"}

{"assets":null,"check":"check-nginx-agent-1","component":"agent","level":"debug","msg":"fetching assets for check","namespace":"default","time":"2020-04-24T03:36:20Z"}

{"check":"check-nginx-agent-1","component":"agent","entity":"agent-1","event_uuid":"2d19fe27-4060-4781-9a4a-e3b0dfc417bc","level":"info","msg":"sending event to backend","time":"2020-04-24T03:36:20Z"}

{"component":"agent","content_type":"application/octet-stream","level":"info","msg":"sending message","payload_size":1026,"time":"2020-04-24T03:36:20Z","type":"event"}

{"component":"agent","content_type":"application/octet-stream","level":"info","msg":"message received","payload_size":436,"time":"2020-04-24T03:36:30Z","type":"check_request"}

{"component":"agent","level":"info","msg":"scheduling check execution: check-nginx-agent-1","time":"2020-04-24T03:36:30Z"}

{"assets":null,"check":"check-nginx-agent-1","component":"agent","level":"debug","msg":"fetching assets for check","namespace":"default","time":"2020-04-24T03:36:30Z"}

{"check":"check-nginx-agent-1","component":"agent","entity":"agent-1","event_uuid":"18f3225f-303e-45de-a017-08fbb846bebb","level":"info","msg":"sending event to backend","time":"2020-04-24T03:36:30Z"}

{"component":"agent","content_type":"application/octet-stream","level":"info","msg":"sending message","payload_size":1026,"time":"2020-04-24T03:36:30Z","type":"event"}

{"component":"transport","level":"debug","msg":"sending ping","time":"2020-04-24T03:36:30Z"}

{"check":"keepalive","component":"agent","entity":"agent-1","event_uuid":"58cb7b7b-1e80-4797-90e6-94af766d16f0","level":"info","msg":"sending event to backend","time":"2020-04-24T03:36:30Z"}

{"component":"transport","level":"debug","msg":"pong received from the backend, setting the read deadline to 1587699435","time":"2020-04-24T03:36:30Z"}

{"component":"agent","content_type":"application/octet-stream","level":"info","msg":"message received","payload_size":436,"time":"2020-04-24T03:36:40Z","type":"check_request"}

{"component":"agent","level":"info","msg":"scheduling check execution: check-nginx-agent-1","time":"2020-04-24T03:36:40Z"}

{"assets":null,"check":"check-nginx-agent-1","component":"agent","level":"debug","msg":"fetching assets for check","namespace":"default","time":"2020-04-24T03:36:40Z"}

{"check":"check-nginx-agent-1","component":"agent","entity":"agent-1","event_uuid":"dfb2dda7-504b-4da2-9222-c0a73eead471","level":"info","msg":"sending event to backend","time":"2020-04-24T03:36:40Z"}

{"component":"agent","content_type":"application/octet-stream","level":"info","msg":"sending message","payload_size":1026,"time":"2020-04-24T03:36:40Z","type":"event"}

{"component":"agent","content_type":"application/octet-stream","level":"info","msg":"message received","payload_size":436,"time":"2020-04-24T03:36:50Z","type":"check_request"}

{"component":"agent","level":"info","msg":"scheduling check execution: check-nginx-agent-1","time":"2020-04-24T03:36:50Z"}

{"assets":null,"check":"check-nginx-agent-1","component":"agent","level":"debug","msg":"fetching assets for check","namespace":"default","time":"2020-04-24T03:36:50Z"}

{"check":"check-nginx-agent-1","component":"agent","entity":"agent-1","event_uuid":"7e457c13-bf18-4bdd-b63d-b195bc8c9277","level":"info","msg":"sending event to backend","time":"2020-04-24T03:36:50Z"}

{"component":"agent","content_type":"application/octet-stream","level":"info","msg":"sending message","payload_size":1026,"time":"2020-04-24T03:36:50Z","type":"event"}

{"component":"agent","content_type":"application/octet-stream","level":"info","msg":"message received","payload_size":436,"time":"2020-04-24T03:37:00Z","type":"check_request"}

{"component":"agent","level":"info","msg":"scheduling check execution: check-nginx-agent-1","time":"2020-04-24T03:37:00Z"}

{"assets":null,"check":"check-nginx-agent-1","component":"agent","level":"debug","msg":"fetching assets for check","namespace":"default","time":"2020-04-24T03:37:00Z"}
4

What’s the log level set to for the agent? This doesn’t seem to show anything that would immediately indicate to me that something’s wrong, or that a remediation check isn’t being executed. I’d drop it into debug (set this in agent.yml) and try to execute the check again.

5

The log level is debug only.

Is there any way to log the handler execution?

If so, we can check if the handler is triggering the remediation check or not.