Http-check cannot handle "connection refused"

Sensu Integrations Sensu Plugins
1

Using Senso Go 6.5.4. Added the asset:

sensuctl asset add sensu/http-checks -r http-checks

Created the check definition:

---
type: CheckConfig
api_version: core/v2
metadata:
  name: check-user
  namespace: default
spec:
  command: http-check --url http://user.service.consul:80/metadata
  interval: 60
  proxy_entity_name: user
  publish: true
  round_robin: true
  runtime_assets:
  - http-checks
  subscriptions:
  - http
  handlers:
  - slack

Added the check to an agent. Then I simulated service failure by blocking outbound TCP port 80 with iptables, such that the check would get connection refused. I verified with curl that the connection is indeed refused.

The check did detect that there was a problem - however, the check itself crashed:

# sensuctl event info user check-user
=== user - check-user
Entity:    user
Check:     check-user
Output:    Usage:
  http-check [flags]
  http-check [command]

Available Commands:
  help        Help about any command
  version     Print the version number of this plugin

Flags:
  -H, --header strings           Additional header(s) to send in check request
  -h, --help                     help for http-check
  -i, --insecure-skip-verify     Skip TLS certificate verification (not recommended!)
  -C, --mtls-cert-file string    Certificate file for mutual TLS auth in PEM format
  -K, --mtls-key-file string     Key file for mutual TLS auth in PEM format
  -r, --redirect-ok              Allow redirects
  -s, --search-string string     String to search for, if not provided do status check only
  -T, --timeout int              Request timeout in seconds (default 15)
  -t, --trusted-ca-file string   TLS CA certificate bundle in PEM format
  -u, --url string               URL to test (default "http://localhost:80/")

Use "http-check [command] --help" for more information about a command.

Error executing http-check: error executing check: Get http://user.service.consul:80/metadata: dial tcp 10.2.95.221:80: connect: connection refused
Status:    2
History:   0,0,0,0,0,2
Silenced:  false
Timestamp: 2021-11-11 00:15:30 +0000 UTC
UUID:      5b9b85bd-1e43-4cfa-8f09-f2069048c97b

Once I cleared iptables, the check detected that the service was “restored”:

# sensuctl event info user check-user
=== user - check-user
Entity:    user
Check:     check-user
Output:    http-check OK: HTTP Status 200 for http://user.service.consul:80/metadata
Status:    0
History:   0,0,0,0,0,2,2,0,0,0,0,0,0,0
Silenced:  false
Timestamp: 2021-11-11 00:23:31 +0000 UTC
UUID:      99f76bd7-a014-47a6-8f29-080fba4435cc

How can I prevent the crash?

2

Adding more info - this is clearly an issue with the http-check executable itself. I found it in /var/cache/sensu/sensu-agent/…/bin/ and I tried to run it on a port known to be closed. It produced a generic usage error message, which is clearly wrong:

# ./http-check --url http://user.service.consul:81/metadata
Usage:
  http-check [flags]
  http-check [command]

Available Commands:
  help        Help about any command
  version     Print the version number of this plugin

Flags:
  -H, --header strings           Additional header(s) to send in check request
  -h, --help                     help for http-check
  -i, --insecure-skip-verify     Skip TLS certificate verification (not recommended!)
  -C, --mtls-cert-file string    Certificate file for mutual TLS auth in PEM format
  -K, --mtls-key-file string     Key file for mutual TLS auth in PEM format
  -r, --redirect-ok              Allow redirects
  -s, --search-string string     String to search for, if not provided do status check only
  -T, --timeout int              Request timeout in seconds (default 15)
  -t, --trusted-ca-file string   TLS CA certificate bundle in PEM format
  -u, --url string               URL to test (default "http://localhost:80/")

Use "http-check [command] --help" for more information about a command.

Error executing http-check: error executing check: Get http://user.service.consul:81/metadata: dial tcp 10.2.95.221:81: connect: connection refused
3

Same error if I tell http-check to connect to a port that’s simply not responding at all (ignoring all packets).

It throws a usage error when it should really just say “connection timeout” or something.

4

Github issue describing this problem: