Sensu Hooks Passing Parameters

So I am trying to configure my Sensu to run a remediation script to (self-heal) my systems. So for instance I have a Web Check that has a hook to run a remediation script, but I need to pass to the remediation script a variable of the hostname to remediate.

I tried to configure my remediation script like:

Name                           Command                        Timeout   Stdin?

remediate /usr/local/bin/remediation {{ .labels.hostname }} 360 false

However it seems that the variables {{ .labels.hostname }} doesn’t work as part of the command to run as remediation?

I see that it logs this when it appears to try to run the remediation:

sensugo sensu-backend: {“component”:“agentd”,“error”:“check is invalid: check name must not be empty”,“level”:“error”,“msg”:“error handling message”,“payload”:"\u0008\ufffd\ufffd\ufffd\ufffd\u0005\u0012\ufffd\u0002\n\u0005agent\u001a\ufffd\u0001\n\u0007sensugo\u0012\u0005linux\u001a\u0006centos"\u0004rhel*\u00087.7.19082r\n\u001a\n\u0002lo\u001a\u000b127.0.0.1/8\u001a\u0007::1/128\nT\n\u0004eth0\u0012\u0011fe:eb:12:03:c1:66\u001a\r10.0.3.101/20\u001a\u000c10.0.1.20/20\u001a\u001cfe80::fceb:12ff:fe03:c166/64:\u0005amd64"\u0005proxy(\ufffd\ufffd\ufffd\ufffd\u0005:\u0000Z\u0005agentj\u0008passwordj\u0006passwdj\u0004passj\u0007api_keyj\tapi_tokenj\naccess_keyj\nsecret_keyj\u000bprivate_keyj\u0006secretr\u0012\n\u0007sensugo\u0012\u0007defaultz\u000e5.14.1#b45811c\u001aq\ufffd\u0001hunmatched token: template: :1:103: executing “” at \u003c.labels.hostname\u003e: map has no entry for key “labels”\ufffd\u0001\u0003\ufffd\u0002\u0000*\u0000",“time”:“2019-10-28T19:56:21-07:00”,“type”:“event”}

Any help would be greatly appreciated.

Devin

Hi @devinacosta,

Did you configured the hostname label for your agent entity? You should be able to verify that information with a command like sensuctl entity info $entity_name --format yaml. Token substitution only works with the metadata of the entity, not the check or hook metadata.

If you could also give us the check configuration and the hook configuration, that would be greatly appreciated!

For reference, I could successfully test this scenario by first modifying my sensu-agent configuration to add the following label for an entity named whisky:

$ sensuctl entity info whisky
type: Entity
api_version: core/v2
metadata:
  labels:
    foo: bar
[...]

Then, I simply configured the following hook and assigned it to a check ran by this entity:

$ sensuctl hook info echo
type: HookConfig
api_version: core/v2
metadata:
  name: echo
  namespace: default
spec:
  command: echo {{ .labels.foo }}
[...]

Finally, I confirmed the hook output by inspecting the resulting event:

$ sensuctl event info whisky check-echo
[...]
hooks:
    - command: echo {{ .labels.foo }}
      duration: 0.00432363
      executed: 1572376524
      issued: 0
      metadata:
        name: echo
        namespace: default
      output: |
        bar
[...]

I do have the following configuration and it’s not working?

Blockquote
type: Entity
api_version: core/v2
metadata:
labels:
hostname: dev.banksite.net
ip: 63.241.146.64
proxy_type: website
name: ws_dev.banksite.net
namespace: default
spec:
deregister: false
deregistration: {}
entity_class: proxy
last_seen: 0
sensu_agent_version: “”
subscriptions:

  • web
    system:
    network:
    interfaces: null

Blockquote
$ sensuctl hook info remediate --format yaml
type: HookConfig
api_version: core/v2
metadata:
name: remediate
namespace: default
spec:
command: /usr/local/bin/remediation {{ .labels.hostname }}
runtime_assets: null
stdin: true
timeout: 360

Blockquote
$ sensuctl check info check_http_hooks --format yaml
type: CheckConfig
api_version: core/v2
metadata:
name: check_http_hooks
namespace: default
spec:
check_hooks:

  • critical:
    • remediate
  • non-zero:
    • remediate
      command: check_http -H {{ .labels.hostname }} -u /
      env_vars: null
      handlers:
  • victorops
    high_flap_threshold: 0
    interval: 60
    low_flap_threshold: 0
    output_metric_format: “”
    output_metric_handlers: null
    proxy_entity_name: “”
    proxy_requests:
    entity_attributes:
    • entity.entity_class == ‘proxy’
    • entity.labels.proxy_type == ‘website’
      splay: false
      splay_coverage: 0
      publish: true
      round_robin: false
      runtime_assets:
  • checks.tar.gz
    stdin: false
    subdue: null
    subscriptions:
  • proxy
    timeout: 0
    ttl: 0

This is what I got when it failed:

Blockquote
$ sensuctl event info ws_dev.banksite.net check_http_hooks
=== ws_dev.banksite.net - check_http_hooks
Entity: ws_dev.banksite.net
Check: check_http_hooks
Output: connect to address dev.banksite.net and port 80: Connection refused
HTTP CRITICAL - Unable to open TCP socket
Status: 2
History: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2
Silenced: false
Timestamp: 2019-10-29 13:35:21 -0700 MST

But it didn’t seem to run the remediation with hostname passed to it?

Try this:

sensuctl event info --format yaml ws_dev.banksite.net check_http_hooks
and look for a hooks attribute under check in the output

The default tabular output of sensuctl only shows a few primary fields of information. If you need to see the entire output you want json or yaml output.

Here’s an even better cmdline command using jq to pull out just the section of interest

sensuctl event info local-poc check_load --format json |jq .check.hooks

The output doesn’t make sense:

$ sensuctl event info ws_dev.banksite.net check_http_hooks --format json |jq .check.hooks
null

Is the hook somewhere not being applied right?

Okay looks like this is a bug associated with token substitution used in hooks being used with proxy requests.

The log message is being generated by the sensu-backend service. It’s a bit cryptic, but my best intuition is the token substitution logic being run against the hook for the proxy entity is expecting a ‘check’ object to be available in scope but its not.

The error is on the backend and not the agent, because the backend is doing the work of generating checks to run for each proxy that matches the proxy request conditions. So there seems to be a bug in that logic somewhere. What is likely happening is the token sub is failing for the hook, the backend then drops the hook and doesn’t attach the hook for the new generated check…so the hook is never fired in this case.