Status code 127

Sensu Classic (EOL)
1

In Sensu classic (Sensu core), I have a check failing because the check script doesn’t exist on the client. I know how to fix this, but what I’m more interested in is how to handle the error on the server. The return status value isn’t 0,1,2 or 3. How I can ensure using a filter that status=127 is handled as well as critical and unknown?

Upon closer inspection my error might really be somewhere in here… Not quite sure yet what do with this error message…

{“timestamp”:“2019-09-23T14:43:07.044406-0400”,“level”:“error”,“message”:“attribute value eval error”,“object”:{“id”:“1cd89447-36fd-4e14-bc9d-91f4894c4a73”,“client”:{“name”:“lcms.mathworks.com”,“address”:“unknown”,“subscriptions”:[“client:lcms.mathworks.com”],“keepalives”:false,“version”:“1.4.2”,“timestamp”:1569249074,“type”:“proxy”},“check”:{“command”:"/opt/mathworks/sensu/checks/check_app_health_v2.py -H lcms.mathworks.com -p 443",“environment”:“production”,“handlers”:[“alerta”,“email”],“interval”:300,“mathworks”:{“application”:“SystemHealth”,“email”:“webinfra-email@localhost”,“notify”:“webinfra-pager”,“resource”:“lcms.mathworks.com”},“publish”:true,“source”:“lcms.mathworks.com”,“standalone”:false,“subscribers”:[“lcms_proxy”],“timeout”:60,“ttl”:900,“ttl_status”:2,“type”:“standard”,“name”:“lcms_frontdoor”,“issued”:1569264187,“executed”:1569264187,“duration”:0.001,“output”:“sh: 1: /opt/mathworks/sensu/checks/check_app_health_v2.py: not found\n”,“status”:127,“origin”:“sensu-server-i-05eadf03703a67875”,“history”:[“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”,“127”],“total_state_change”:0},“occurrences”:73,“occurrences_watermark”:73,“last_ok”:null,“action”:“create”,“timestamp”:1569264187,“last_state_change”:null,“silenced”:false,“silenced_by”:},“raw_eval_string”:“eval: value != :::check.occurrences|1::: && value % 6 != 0”,“raw_value”:null,“error”:“undefined method `%’ for nil:NilClass”}

Thanks!

2

Yes this is fairly common when you have nodes that are coming up and relying on something like chef, puppet, ansible, etc for last mile delivery. I talk about it briefly here: https://blog.sensu.io/alert-fatigue-part-5-fine-tuning-silencing, you can skip down to “Safe Mode”. This essentially tells you not to execute checks on the client unless they are defined on the server and client. As in my setups the last mile thats dropping the config on the node (chef) is the same mechanism that installs the check dependencies this mitigates it for me. Your mileage may vary based on your setup and needs.

3

Forgive my ignorance. Does this mean that if the check script on the client host doesn’t exist, the return code from the check will not be 0,1,2 or 3 (because it didn’t execute). Handlers only handle return codes 0-3, right? A status code of 127 is more administrative then. Is there anyway to execute a handler on that? (Are there other possibilities beside 0-3 and 127?

Is this the same behavior in Classic and Go?

Thanks for your help. I’m having great success running Sensu!

4

The client won’t even execute the script even if the server told it to if the definition does not exist. As sensu-go’s configuration is done via HTTP(s) rather than file based so standalone mode no longer exists. There is a partial equivalent but that sadly does not help you here. You could write a filter to ignore status codes of (126|127) so its not sending to TSDBs, notification systems, etc.

5

hello guys,

i had written shell script for memory usage. and i am running the script with memory checks. provided full path . but while running the checksits frequently getting status code 127 and sometime its passing. i am not getting exactly the issue
image
image

please find the above snap its working fine 30 secsonds after that its giving status code 127…
thanks in adavance

6

This sounds like something in the snap is cleaning up tmp/memory_usage.sh after some period of time. Does it recover after a period of time? Does this happen if you do this outside of the context of a snap? I don’t know much about snaps, if you can help me isolate the problem or confirm its specific to snap usage that would be helpful in tracking down a solution for you.

7

let me explain the scenario, i had installed sensu to monitor the 2 azure vms and i have 3 checks on every vm . for 2 checks i had used the built in plugin in sensu bonsai . and for 3rd check i,e for memory usage i used sensu plugin but the output was in the bytes format. i was not able to convert to human readble format. so i thought of writing own script and executing so same thing i wrote in shell scripted and kept in /tmp/ folder in every containers. but all containers script is running properly. only in particular vm i am getting error like this… NOTE : i am running sensu backend and agents as docker containers.

8

Ok, so I saw that you opened up Status code 127 for checking memory usage. Let’s have the conversation there instead of on this thread. This one is 3 years old and is for Sensu Classic–not Sensu Go.