Status code 127

In Sensu classic (Sensu core), I have a check failing because the check script doesn’t exist on the client. I know how to fix this, but what I’m more interested in is how to handle the error on the server. The return status value isn’t 0,1,2 or 3. How I can ensure using a filter that status=127 is handled as well as critical and unknown?

Upon closer inspection my error might really be somewhere in here… Not quite sure yet what do with this error message…

{“timestamp”:“2019-09-23T14:43:07.044406-0400”,“level”:“error”,“message”:“attribute value eval error”,“object”:{“id”:“1cd89447-36fd-4e14-bc9d-91f4894c4a73”,“client”:{“name”:“lcms.mathworks.com”,“address”:“unknown”,“subscriptions”:[“client:lcms.mathworks.com”],“keepalives”:false,“version”:“1.4.2”,“timestamp”:1569249074,“type”:“proxy”},“check”:{“command”:"/opt/mathworks/sensu/checks/check_app_health_v2.py -H lcms.mathworks.com -p 443",“environment”:“production”,“handlers”:[“alerta”,“email”],“interval”:300,“mathworks”:{“application”:“SystemHealth”,“email”:“webinfra-email@localhost”,“notify”:“webinfra-pager”,“resource”:“lcms.mathworks.com”},“publish”:true,“source”:“lcms.mathworks.com”,“standalone”:false,“subscribers”:[“lcms_proxy”],“timeout”:60,“ttl”:900,“ttl_status”:2,“type”:“standard”,“name”:“lcms_frontdoor”,“issued”:1569264187,“executed”:1569264187,“duration”:0.001,“output”:“sh: 1: /opt/mathworks/sensu/checks/check_app_health_v2.py: not found\n”,“status”:127,“origin”:“sensu-server-i-05eadf03703a67875”,“history”:[“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”],“total_state_change”:0},“occurrences”:73,“occurrences_watermark”:73,“last_ok”:null,“action”:“create”,“timestamp”:1569264187,“last_state_change”:null,“silenced”:false,“silenced_by”:},“raw_eval_string”:“eval: value != :::check.occurrences|1::: && value % 6 != 0”,“raw_value”:null,“error”:“undefined method `%’ for nil:NilClass”}

Thanks!

Yes this is fairly common when you have nodes that are coming up and relying on something like chef, puppet, ansible, etc for last mile delivery. I talk about it briefly here: https://blog.sensu.io/alert-fatigue-part-5-fine-tuning-silencing, you can skip down to “Safe Mode”. This essentially tells you not to execute checks on the client unless they are defined on the server and client. As in my setups the last mile thats dropping the config on the node (chef) is the same mechanism that installs the check dependencies this mitigates it for me. Your mileage may vary based on your setup and needs.

Forgive my ignorance. Does this mean that if the check script on the client host doesn’t exist, the return code from the check will not be 0,1,2 or 3 (because it didn’t execute). Handlers only handle return codes 0-3, right? A status code of 127 is more administrative then. Is there anyway to execute a handler on that? (Are there other possibilities beside 0-3 and 127?

Is this the same behavior in Classic and Go?

Thanks for your help. I’m having great success running Sensu!

The client won’t even execute the script even if the server told it to if the definition does not exist. As sensu-go’s configuration is done via HTTP(s) rather than file based so standalone mode no longer exists. There is a partial equivalent but that sadly does not help you here. You could write a filter to ignore status codes of (126|127) so its not sending to TSDBs, notification systems, etc.