Help understanding Labels/Annotations in Migrations from Sensu-Core to Sensu-Go

Sensu Go
,
1

Hello,

I am in the process of migrating our sensu core servers to sensu-go. This has been a task in itself, and not very easy. I am on the creation of checks, and I am having issues with “Environments”.

I understand in the agent.yml I can use annotations or labels to say things like environment is dev. Which is great. The feature that I am trying to understand is environment variables. In our current checks we use environment variables such as:

Check

check_inode_metrics:
command: python3 /etc/sensu/plugins/dpe_inode_metrics.py -g xxxxx.xxxx.com
subscribers:
- dpe
interval: 60
occurrences: 10
plugins:
- dpe_inode_metrics.py
filters:
- occurrences
- only_business_hours
handlers: email
environment_settings:
prod:
command: python3 /etc/sensu/plugins/dpe_inode_metrics.py -g xxxx.xxxx.com
handlers: pagerduty

I cannot find a way to do this. I have tried many different ways and the only one I can come up with, is if each entity has every “Environment” variable/annotations assigned to it. Is there an easier way to do this? I feel like having 100 different annotations or labels on each entity will be a nightmare to maintain. We also have pagerduty maps, teams, etc. assigned to each check currently.

Here is a basic cpu check translated by the translator to see how we use the variables:

Translated Check

{
“api_version”:“core/v2”,
“type”:“Check”,
“metadata”:{
“namespace”:“default”,
“name”:“dpe_check_cpu”,
“labels”:{},
“annotations”:{
“sensu.io.json_attributes”:"{“email”:[“dpe_team@xxxxxx.com”],“pagerduty_name”:“low_priority”,“pagerduty_map”:{“low_priority”:“xxxxxxxx”,“high_priority”:“xxxxxxxx”},“environments”:[“test”,“dev”,“prod”],“severities”:[“critical”,“unknown”],“refresh”:1800,“occurrences”:10,“allow_client_override”:false,“disable_email”:false,“disable_webex”:false,“disable_pagerduty”:false,“environment_settings”:{“test”:{“allow_client_override”:true,“email”:[“dpe_team@xxxxxxxx.com”],“handlers”:[“email”]},“prod”:{“pagerduty_name”:“high_priority”,“handlers”:[“pagerduty”,“email”]}},“plugins”:[“dpe_check_cpu.rb”],“team”:“dpe”,“defined_in”:“dpe_default_checks.yaml”}",
“fatigue_check/occurrences”:“10”,
“fatigue_check/interval”:“1800”
}
},
“spec”:{
“command”:“sensu_ruby /etc/sensu/plugins/dpe_check_cpu.rb -c 99 -w 95”,
“subscriptions”:[
“dpe”
],
“publish”:true,
“interval”:60,
“handlers”:[
“dpe_business_hour_email”
]
}
}

2

hey @mvinzi . I am not sure I understand the problem. How are environment variables different than a label on an entity?

From what I can see it seems like you are trying to have a different handler per environment, is that correct?

If so, I think you could do one check with all the handlers you needed, and then use filters to stop the handler from activating in certain environments (i.e. filter that does not page if it is a dev environment)

Does that answer your question?

3

Hey!

Have you thought about mapping your environment types to Sensu namespace to isolate environments? That way you can maintain agents in Sensu namespaces like “dev” and “production” using a different collection of monitoring resources tailored for each environment. Then its just a matter of setting the correct namespace attribute in the agent.yml.

The trade-off here when using namespaces is you’ll need to maintain resources in each namespace, but an upside is resources in each namespace should be easier to understand, less conditional filter logic in your handler filters.

Also, I use the “default” namespace as sort of a “oops we forgot to set the agent namespace” and have sensu drive registration event alerts differently in “default” namespace so I can catch that situation and address it.