Unable to get sensu slack handler to work

Sensu Integrations Sensu Plugins
1

I am testing sensu slack integration in sensu-go and am running into an error.
I created a slack webhook url, handler and asset. I am hitting events api to run my test.

I use the same webhook in other places and it works.
Handler Config:

sensuctl handler info slack --format yaml
type: Handler
api_version: core/v2
metadata:
  created_by: admin
  labels:
    sensu.io/managed_by: sensuctl
  name: slack
  namespace: default
spec:
  command: sensu-slack-handler --channel '#<slack_channel>'
  env_vars:
  - SLACK_WEBHOOK_URL=<hook_url>
  filters: null
  handlers: null
  runtime_assets:
  - sensu-slack-handler
  secrets: null
  timeout: 10
  type: pipe

Asset definition. Hosted assets locally

type: Asset
api_version: core/v2
metadata:
  created_by: admin
  labels:
    sensu.io/managed_by: sensuctl
  name: sensu-slack-handler
  namespace: default
spec:
  builds:
  - filters:
    - entity.system.os == 'windows'
    - entity.system.arch == 'amd64'
    headers: null
    sha512: 6e2e49a07901fc4669d257d946585e9a1f7eb2a130963a5a9c7c95d5dfbfdead1631cffb70ec4c274512b87189e36a3b009c008ebba5fbc2425ee55109dc21d8
    url: https://<local_url>/sensu-slack-handler_1.5.0_windows_amd64.tar.gz
  - filters:
    - entity.system.os == 'linux'
    - entity.system.arch == 'amd64'
    headers: null
    sha512: 00412e28608bb547ccafe7de259d7869ffecf64f1b3746ba8027b60c56f6fd13ee5c729fd8ba33fb95bc4de13eaa87be0c4bf61ca170d0650727d2e240ff93bf
    url: https://<local_url>/sensu-slack-handler_1.5.0_linux_amd64.tar.gz
  filters: null
  headers: null

Test - Ran in sensu-backed VM.

curl -X POST -H 'Content-Type: application/json' -d '{
  "check": {
    "metadata": {
      "name": "testing-slack-handler"
    },
    "status": 2,
    "output": "this is a test event to see if Slack works",
    "handlers": [
      "slack"
    ]
  }
}' http://127.0.0.1:3031/events

Error in logs:

{"assets":["sensu-slack-handler"],"check":"testing-slack-handler","component":"pipelined","entity":"<entity_fqdn>","error":"exec: \"sh\": executable file not found in $PATH","event_uuid":"ce93f2e7-fe11-4c29-b22a-f7fbad7d7a61","handler":"slack","level":"error","msg":"failed to execute event pipe handler","namespace":"default","time":"2021-08-03T07:07:24Z"}
{"check_name":"testing-slack-handler","check_namespace":"default","component":"pipelined","entity_name":"<entity_fqdn>","entity_namespace":"default","handler":"slack","level":"error","msg":"exec: \"sh\": executable file not found in $PATH","time":"2021-08-03T07:07:24Z","uuid":"ce93f2e7-fe11-4c29-b22a-f7fbad7d7a61"}

Sensu Version:

{"etcd":{"etcdserver":"3.3.22","etcdcluster":"3.3.0"},"sensu_backend":"6.1.1"}

My sensu-backed is running inside a docker container. Not sure what’s missing at this point. Please suggest.

2

whoa,
that’s a very unexpected error for sure.
let me check something real quick.

3

Okay i think i understand the error.

The sensu-backend requires access to the sh executable command in the executable path, this is the shell it will run handlers and mutators inside of.

If your sensu-backend environment does not have access to the sh executable then all pipe handler execution will fail with a similar error.

1 Like
4

Thanks, there’s no shell right now in sensu-backed container.
Is there a preferred way of doing this in Sensu? I couldn’t find a runtime asset for shell.
Should I mount extra volumes that contain shell binaries from my VM to the docker container?

5

I added volumes from VM to the container and it seems to have progressed. I am now stuck with a certificate error. I ran openssl connect check on port 443 to the slack webhook from VM and it runs fine. I don’t see any option in the handler to skip tls verification.

{"log":"{\"assets\":[\"sensu-slack-handler\"],\"check\":\"testing-slack-handler\",\"component\":\"pipelined\",\"entity\”:\”<><><>\”,\”event_uuid\":\"b61011f9-4e00-4f19-a8d0-3ebc8cffde0e\",\"handler\":\"slack\",\"level\":\"info\",\"msg\":\"event pipe handler executed\",\"namespace\":\"default\",\"output\":\"Usage:\\n  sensu-slack-handler [flags]\\n  sensu-slack-handler [command]\\n\\nAvailable Commands:\\n  help        Help about any command\\n  version     Print the version number of this plugin\\n\\nFlags:\\n  -c, --channel string                The channel to post messages to (default \\\"#general\\\")\\n  -t, --description-template string   The Slack notification output template, in Golang text/template format (default \\\"{{ .Check.Output }}\\\")\\n  -h, --help                          help for sensu-slack-handler\\n  -i, --icon-url string               A URL to an image to use as the user avatar (default \\\"https://www.sensu.io/img/sensu-logo.png\\\")\\n  -u, --username string               The username that messages will be sent as (default \\\"sensu\\\")\\n  -w, --webhook-url string            The webhook url to send messages to\\n\\nUse \\\"sensu-slack-handler [command] --help\\\" for more information about a command.\\n\\nError executing sensu-slack-handler: error executing handler: Failed to send Slack message: Post \\\"https://<><><>\\\”: x509: certificate signed by unknown authority\\n\",\"status\":1,\"time\":\"2021-08-05T13:57:48Z\"}\n","stream":"stderr","time":"2021-08-05T13:57:48.093908865Z"}
6

hmm this is the official sensu container?
Or are you rolling your own container from scratch?

7

There’s no option for skipping tls validation for slack in the handler at present. You’ll want to include the OS packaged ca certificats in your container.

As an aside, I’m generally wary about exposing tls validation skip for externally managed service handlers because of the risk of dns man-in-the-middle.

8

I am using the official container. I can hit the webhook just fine from VM. Not sure why the container doesn’t pick up the CA from VM. I will try to build an image with ca-certificates package and see how it goes.
And yes, I don’t want to skip tls :slight_smile: just interested in testing this out.

9

Hmm,
I’ll find time today to re-test the latest official container image using the sensu-go-workshop environment, which should ensure a clean test.