Sensu Hooks Passing Parameters

So I am trying to configure my Sensu to run a remediation script to (self-heal) my systems. So for instance I have a Web Check that has a hook to run a remediation script, but I need to pass to the remediation script a variable of the hostname to remediate.

I tried to configure my remediation script like:

Name                           Command                        Timeout   Stdin?

remediate /usr/local/bin/remediation {{ .labels.hostname }} 360 false

However it seems that the variables {{ .labels.hostname }} doesn’t work as part of the command to run as remediation?

I see that it logs this when it appears to try to run the remediation:

sensugo sensu-backend: {“component”:“agentd”,“error”:“check is invalid: check name must not be empty”,“level”:“error”,“msg”:“error handling message”,“payload”:"\u0008\ufffd\ufffd\ufffd\ufffd\u0005\u0012\ufffd\u0002\n\u0005agent\u001a\ufffd\u0001\n\u0007sensugo\u0012\u0005linux\u001a\u0006centos"\u0004rhel*\u00087.7.19082r\n\u001a\n\u0002lo\u001a\u000b127.0.0.1/8\u001a\u0007::1/128\nT\n\u0004eth0\u0012\u0011fe:eb:12:03:c1:66\u001a\r10.0.3.101/20\u001a\u000c10.0.1.20/20\u001a\u001cfe80::fceb:12ff:fe03:c166/64:\u0005amd64"\u0005proxy(\ufffd\ufffd\ufffd\ufffd\u0005:\u0000Z\u0005agentj\u0008passwordj\u0006passwdj\u0004passj\u0007api_keyj\tapi_tokenj\naccess_keyj\nsecret_keyj\u000bprivate_keyj\u0006secretr\u0012\n\u0007sensugo\u0012\u0007defaultz\u000e5.14.1#b45811c\u001aq\ufffd\u0001hunmatched token: template: :1:103: executing “” at \u003c.labels.hostname\u003e: map has no entry for key “labels”\ufffd\u0001\u0003\ufffd\u0002\u0000*\u0000",“time”:“2019-10-28T19:56:21-07:00”,“type”:“event”}

Any help would be greatly appreciated.


1 Like

Hi @devinacosta,

Did you configured the hostname label for your agent entity? You should be able to verify that information with a command like sensuctl entity info $entity_name --format yaml. Token substitution only works with the metadata of the entity, not the check or hook metadata.

If you could also give us the check configuration and the hook configuration, that would be greatly appreciated!

For reference, I could successfully test this scenario by first modifying my sensu-agent configuration to add the following label for an entity named whisky:

$ sensuctl entity info whisky
type: Entity
api_version: core/v2
    foo: bar

Then, I simply configured the following hook and assigned it to a check ran by this entity:

$ sensuctl hook info echo
type: HookConfig
api_version: core/v2
  name: echo
  namespace: default
  command: echo {{ }}

Finally, I confirmed the hook output by inspecting the resulting event:

$ sensuctl event info whisky check-echo
    - command: echo {{ }}
      duration: 0.00432363
      executed: 1572376524
      issued: 0
        name: echo
        namespace: default
      output: |
1 Like

I do have the following configuration and it’s not working?

type: Entity
api_version: core/v2
proxy_type: website
namespace: default
deregister: false
deregistration: {}
entity_class: proxy
last_seen: 0
sensu_agent_version: “”

  • web
    interfaces: null

$ sensuctl hook info remediate --format yaml
type: HookConfig
api_version: core/v2
name: remediate
namespace: default
command: /usr/local/bin/remediation {{ .labels.hostname }}
runtime_assets: null
stdin: true
timeout: 360

$ sensuctl check info check_http_hooks --format yaml
type: CheckConfig
api_version: core/v2
name: check_http_hooks
namespace: default

  • critical:
    • remediate
  • non-zero:
    • remediate
      command: check_http -H {{ .labels.hostname }} -u /
      env_vars: null
  • victorops
    high_flap_threshold: 0
    interval: 60
    low_flap_threshold: 0
    output_metric_format: “”
    output_metric_handlers: null
    proxy_entity_name: “”
    • entity.entity_class == ‘proxy’
    • entity.labels.proxy_type == ‘website’
      splay: false
      splay_coverage: 0
      publish: true
      round_robin: false
  • checks.tar.gz
    stdin: false
    subdue: null
  • proxy
    timeout: 0
    ttl: 0

This is what I got when it failed:

$ sensuctl event info check_http_hooks
=== - check_http_hooks
Check: check_http_hooks
Output: connect to address and port 80: Connection refused
HTTP CRITICAL - Unable to open TCP socket
Status: 2
History: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2
Silenced: false
Timestamp: 2019-10-29 13:35:21 -0700 MST

But it didn’t seem to run the remediation with hostname passed to it?

Try this:

sensuctl event info --format yaml check_http_hooks
and look for a hooks attribute under check in the output

The default tabular output of sensuctl only shows a few primary fields of information. If you need to see the entire output you want json or yaml output.

1 Like

Here’s an even better cmdline command using jq to pull out just the section of interest

sensuctl event info local-poc check_load --format json |jq .check.hooks

1 Like

The output doesn’t make sense:

$ sensuctl event info check_http_hooks --format json |jq .check.hooks

Is the hook somewhere not being applied right?

Okay looks like this is a bug associated with token substitution used in hooks being used with proxy requests.

The log message is being generated by the sensu-backend service. It’s a bit cryptic, but my best intuition is the token substitution logic being run against the hook for the proxy entity is expecting a ‘check’ object to be available in scope but its not.

The error is on the backend and not the agent, because the backend is doing the work of generating checks to run for each proxy that matches the proxy request conditions. So there seems to be a bug in that logic somewhere. What is likely happening is the token sub is failing for the hook, the backend then drops the hook and doesn’t attach the hook for the new generated check…so the hook is never fired in this case.

1 Like

Check hooks aren’t intended to be used for remediation; they don’t offer workflow controls such as “try once” and/or escalation tasks if the first task doesn’t restore service.

Have you seen the remediation handler plugin? It executed checks which have access to entity metadata (as tokens), which addresses your original issue.

I hope this helps!!