Sensu Hooks Passing Parameters

[devinacosta]

So I am trying to configure my Sensu to run a remediation script to (self-heal) my systems. So for instance I have a Web Check that has a hook to run a remediation script, but I need to pass to the remediation script a variable of the hostname to remediate.

I tried to configure my remediation script like:

Name                           Command                        Timeout   Stdin?

remediate /usr/local/bin/remediation {{ .labels.hostname }} 360 false

However it seems that the variables {{ .labels.hostname }} doesn’t work as part of the command to run as remediation?

I see that it logs this when it appears to try to run the remediation:

sensugo sensu-backend: {“component”:“agentd”,“error”:“check is invalid: check name must not be empty”,“level”:“error”,“msg”:“error handling message”,“payload”:"\u0008\ufffd\ufffd\ufffd\ufffd\u0005\u0012\ufffd\u0002\n\u0005agent\u001a\ufffd\u0001\n\u0007sensugo\u0012\u0005linux\u001a\u0006centos"\u0004rhel*\u00087.7.19082r\n\u001a\n\u0002lo\u001a\u000b127.0.0.1/8\u001a\u0007::1/128\nT\n\u0004eth0\u0012\u0011fe:eb:12:03:c1:66\u001a\r10.0.3.101/20\u001a\u000c10.0.1.20/20\u001a\u001cfe80::fceb:12ff:fe03:c166/64:\u0005amd64"\u0005proxy(\ufffd\ufffd\ufffd\ufffd\u0005:\u0000Z\u0005agentj\u0008passwordj\u0006passwdj\u0004passj\u0007api_keyj\tapi_tokenj\naccess_keyj\nsecret_keyj\u000bprivate_keyj\u0006secretr\u0012\n\u0007sensugo\u0012\u0007defaultz\u000e5.14.1#b45811c\u001aq\ufffd\u0001hunmatched token: template: :1:103: executing “” at \u003c.labels.hostname\u003e: map has no entry for key “labels”\ufffd\u0001\u0003\ufffd\u0002\u0000*\u0000",“time”:“2019-10-28T19:56:21-07:00”,“type”:“event”}

Any help would be greatly appreciated.

Devin

[palourde]

Hi @devinacosta,

Did you configured the hostname label for your agent entity? You should be able to verify that information with a command like sensuctl entity info $entity_name --format yaml. Token substitution only works with the metadata of the entity, not the check or hook metadata.

If you could also give us the check configuration and the hook configuration, that would be greatly appreciated!

For reference, I could successfully test this scenario by first modifying my sensu-agent configuration to add the following label for an entity named whisky:

$ sensuctl entity info whisky
type: Entity
api_version: core/v2
metadata:
  labels:
    foo: bar
[...]

Then, I simply configured the following hook and assigned it to a check ran by this entity:

$ sensuctl hook info echo
type: HookConfig
api_version: core/v2
metadata:
  name: echo
  namespace: default
spec:
  command: echo {{ .labels.foo }}
[...]

Finally, I confirmed the hook output by inspecting the resulting event:

$ sensuctl event info whisky check-echo
[...]
hooks:
    - command: echo {{ .labels.foo }}
      duration: 0.00432363
      executed: 1572376524
      issued: 0
      metadata:
        name: echo
        namespace: default
      output: |
        bar
[...]

[devinacosta]

I do have the following configuration and it’s not working?

Blockquote
type: Entity
api_version: core/v2
metadata:
labels:
hostname: dev.banksite.net
ip: 63.241.146.64
proxy_type: website
name: ws_dev.banksite.net
namespace: default
spec:
deregister: false
deregistration: {}
entity_class: proxy
last_seen: 0
sensu_agent_version: “”
subscriptions:

• web
  system:
  network:
  interfaces: null

Blockquote
$ sensuctl hook info remediate --format yaml
type: HookConfig
api_version: core/v2
metadata:
name: remediate
namespace: default
spec:
command: /usr/local/bin/remediation {{ .labels.hostname }}
runtime_assets: null
stdin: true
timeout: 360

Blockquote
$ sensuctl check info check_http_hooks --format yaml
type: CheckConfig
api_version: core/v2
metadata:
name: check_http_hooks
namespace: default
spec:
check_hooks:

• critical:
  • remediate
• non-zero:
  • remediate
    command: check_http -H {{ .labels.hostname }} -u /
    env_vars: null
    handlers:
• victorops
  high_flap_threshold: 0
  interval: 60
  low_flap_threshold: 0
  output_metric_format: “”
  output_metric_handlers: null
  proxy_entity_name: “”
  proxy_requests:
  entity_attributes:
  • entity.entity_class == ‘proxy’
  • entity.labels.proxy_type == ‘website’
    splay: false
    splay_coverage: 0
    publish: true
    round_robin: false
    runtime_assets:
• checks.tar.gz
  stdin: false
  subdue: null
  subscriptions:
• proxy
  timeout: 0
  ttl: 0

This is what I got when it failed:

Blockquote
$ sensuctl event info ws_dev.banksite.net check_http_hooks
=== ws_dev.banksite.net - check_http_hooks
Entity: ws_dev.banksite.net
Check: check_http_hooks
Output: connect to address dev.banksite.net and port 80: Connection refused
HTTP CRITICAL - Unable to open TCP socket
Status: 2
History: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2
Silenced: false
Timestamp: 2019-10-29 13:35:21 -0700 MST

But it didn’t seem to run the remediation with hostname passed to it?

[jspaleta]

@devinacosta

sensuctl event info ws_dev.banksite.net check_http_hooks

Try this:

sensuctl event info --format yaml ws_dev.banksite.net check_http_hooks
and look for a hooks attribute under check in the output

The default tabular output of sensuctl only shows a few primary fields of information. If you need to see the entire output you want json or yaml output.

[jspaleta]

Here’s an even better cmdline command using jq to pull out just the section of interest

sensuctl event info local-poc check_load --format json |jq .check.hooks

[devinacosta]

The output doesn’t make sense:

$ sensuctl event info ws_dev.banksite.net check_http_hooks --format json |jq .check.hooks
null

Is the hook somewhere not being applied right?

[jspaleta]

Okay looks like this is a bug associated with token substitution used in hooks being used with proxy requests.

The log message is being generated by the sensu-backend service. It’s a bit cryptic, but my best intuition is the token substitution logic being run against the hook for the proxy entity is expecting a ‘check’ object to be available in scope but its not.

The error is on the backend and not the agent, because the backend is doing the work of generating checks to run for each proxy that matches the proxy request conditions. So there seems to be a bug in that logic somewhere. What is likely happening is the token sub is failing for the hook, the backend then drops the hook and doesn’t attach the hook for the new generated check…so the hook is never fired in this case.

Bug: Proxy Check not firing hooks that use token subs

[calebhailey]

Check hooks aren’t intended to be used for remediation; they don’t offer workflow controls such as “try once” and/or escalation tasks if the first task doesn’t restore service.

Have you seen the remediation handler plugin? It executed checks which have access to entity metadata (as tokens), which addresses your original issue.

https://bonsai.sensu.io/assets/sensu/sensu-remediation-handler

I hope this helps!!