Help: Does anyone have for me a working example of proxies and a sensu.CheckDependencies

Sensu Go
1

Does anyone have for me a working example of proxies and a sensu.CheckDependencies

I have many proxys with snmp checks and a opsgenie handler.
A proxy has 8 checks with different snmp checks.
Plus one ping check.
When I currently have a failure, I get 9 messages, only from one proxy.
I want to reduce this mass of messages with CheckDependencies.
Can someone here post me an example how this works in detail?

I use sensu ee in the latest version (6.4)

Thanks

type: CheckConfig
api_version: core/v2
metadata:
  annotations:
    fatigue_check/occurrences: "3"
  labels:
    sensu.io/workflow: sensu-flow
  name: check-ping-proxy
spec:
  command: check_fping {{.labels.ping_hostname}} -n 3 -v
  handlers:
    - opsgenie
  high_flap_threshold: 0
  interval: 60
  low_flap_threshold: 0
  output_metric_format: nagios_perfdata
  output_metric_handlers:
    - influxdb
  proxy_entity_name: check-ping-proxy
  proxy_requests:
    entity_attributes:
      - entity.entity_class == 'proxy'
      - entity.labels.ping == 'true'
    splay: false
    splay_coverage: 90
  publish: true
  round_robin: true
  runtime_assets:
    - sensu-plugins-monitoring-plugins
  stdin: false
  subscriptions:
    - proxy
  timeout: 30
  ttl: 0

type: EventFilter
api_version: core/v2
metadata:
  labels:
    sensu.io/workflow: sensu-flow
  name: check-ping-proxy
spec:
  action: allow
  expressions:
    - sensu.CheckDependencies("check-ping-proxy")

---
type: Handler
api_version: core/v2
metadata:
  labels:
    sensu.io/workflow: sensu-flow
  name: opsgenie
spec:
  type: pipe
  command: 'sensu-opsgenie-handler --withAnnotations --withLabels --priority "{{.Entity.Annotations.opsgenie_priority}}" --sensuDashboard "https://sensu.local/c/~/n" --fullDetails --addHooksToDetails'
  env_vars:
  - OPSGENIE_TEAM=itops
  - OPSGENIE_REGION=us
  - OPSGENIE_AUTHTOKEN=zzzzzz
  timeout: 60
  runtime_assets:
  - sensu-opsgenie-handler
  filters:
    - is_incident
    - not_silenced
    - fatigue_check
    - check-ping-proxy

---
type: CheckConfig
api_version: core/v2
metadata:
  annotations:
    fatigue_check/occurrences: "3"
  labels:
    sensu.io/workflow: sensu-flow
  name: check-AdminLoginAttempts
spec:
  command: 'check_snmp -P {{.labels.snmp_protocol | default "3"}} -timeout={{.labels.snmp_timeout | default "90"}} -retries={{.labels.snmp_retries | default "5"}} -hostname={{.labels.hostname}} -secname={{.labels.snmp_secname}} -authpasswd={{.labels.snmp_authpasswd}} -privpasswd={{.labels.snmp_privpasswd}} -seclevel={{.labels.snmp_seclevel | default "public"}} -authproto={{.labels.snmp_authproto | default "SHA"}} -privproto={{.labels.snmp_privproto | default "AES"}} -oid=.1.3.6.1.4.1.23629.1.5.1.2.1.16.{{.labels.snmp_serialnumber}} -label=AdminLoginAttempts -warning=3: -critical=1:'
  handlers: []
  high_flap_threshold: 60
  interval: 300
  low_flap_threshold: 20
  output_metric_format: nagios_perfdata
  output_metric_handlers:
    - influxdb
  proxy_entity_name: check-AdminLoginAttempts
  proxy_requests:
    entity_attributes:
      - entity.entity_class == 'proxy'
      - entity.labels.proxy_type == 'check-snmp-loginattempts'
    splay: true
    splay_coverage: 90
  publish: true
  round_robin: true
  runtime_assets:
    - sensu-plugins-monitoring-plugins
  stdin: false
  subscriptions:
    - proxy
  timeout: 160
  ttl: 0
2

Hey!

Based on the configuration you’ve provided I’m not sure sensu.CheckDependencies is what you want… or you need to adapt your config so CheckDependencies works.

I’m suspicious that your CheckConfig for your proxy check is collapsing all the generated events into a single event. I think, the proxy_entity_name attribute is causing all runs of the check by all proxy entities to rewrite the entity name to be “check-ping-proxy” Can you double check that for me?

sensu.CheckDependancies only makes sense to me if you either have
a) events from different entities
b) events from different checks from the same entity

Right now based on your configuration, I don’t think have either case.

3

OK,
I’ll try to explain in more detail how my setup looks like.

I monitor several systems via SNMP. Each of these systems is also tested with a ping check.
For the SNMP check I use:
check_snmp
Each of the systems has not only one check but 3-12.
Here are of course 3-12 different checks in sensu meant. So each has a different name and proxy_entity_name.
E.g:

  • OperationErrors
  • PartitionStorageAllocatedBytes
  • PartitionsFree
    Now I want to prevent 4-13 alerts in case of a failure, but only 1-2 alerts maximum per proxy/agent.

I didn’t quite understand your statement with the parameter proxy_entity_name. I have always used this until now without I think understanding exactly why and how this is used.

That’s why I thought that the method CheckDependancies is the right one.
But currently I can’t get this to work.
And I haven’t found a real-live-example either here in the forum or on Github yet.

My desired result would be for starters,
if a ping fails, no alert should communicate a message via handler(opsgenie).

4

Hey,

First… in your filter named ‘check-ping-proxy’ you have

sensu.CheckDependencies("check-ping-proxy")

The intepretation of this line is that CheckDependencies will be checking to see if any Sensu events with the check name “check-ping-proxy” are non-zero status. When called by a handler attached to “check-ping-proxy”, as in your configuration, this only does anything useful if there are multiple events with a check name of “check-ping-proxy”

Let’s check if that’s true:

sensuctl event list --field-selector 'event.check.name == "check-ping-proxy"'

If there is only one event returned using that field-selector filter then CehckDependencies either not what you want, or you are asking CheckDependencies to look at the wrong check name, or you have a misconfigured proxy check configuration. So let’s start there.
Does that filtered event list return more than on event, from different entities? Or does it return just one event with entity name check-ping-proxy

5

I get several entities back here, of course. Since I use this also in several Entitys-proxys.
But what is the right way to reduce a CheckDependencies in only one entity?
So I am looking for a Keepalive alternative for Entitys-proxys.

6

Here is a recent example for a better understanding. I have 3 proxys Entitys

appliance1 (only (icmp/snmp allows)
- check-ping-proxy → DOWN
- snmp-disksize → DOWN
- snmp-temperature → DOWN
- snmp-fanspeed → DOWN
appliance2 (only snmp allow)
- check-ping-proxy → UP
- snmp-disksize → UP
- snmp-temperature → UP
- snmp-fanspeed → DOWN
appliance3 (only snmp allow)
- check-ping-proxy → UP
- snmp-disksize → UP
- snmp-temperature → UP
- snmp-fanspeed → UP

The result should be,
that I get only two alerts in Opsgenie.
appliance1 → check-ping-proxy
appliance2 → snmp-fanspeed

Currently I get 4 alerts.
Four from appliance1. That’s three too many for me.

In other words I need a keepalive for proxies

7

I have now created a separate handler for filter. With this it works.

check-ping-proxy.yaml

type: EventFilter
api_version: core/v2
metadata:
labels:
Sensu | Page not found sensu-flow
name: filter-ping-proxy
spec:
action: allow
expressions:
- sensu.CheckDependencies(“check-ping-proxy”)

However, I don’t find the solution elegant. Since I now need a filter for each proxy check so a handler.

Unfortunately my custom filter does not work yet. can someone tell me why.

type: EventFilter
api_version: core/v2
metadata:
labels:
Sensu | Page not found sensu-flow
name: check-ping-proxy
spec:
action: deny
expressions:
- ‘(function () { var pingEvent = sensu.EventStatus(“check-ping-proxy”); if (pingEvent != 0) { return true; } else{ return false; } })()’

8

So I think the BSM feature is actually what you probably need to use, it lets you build more expressive polling of Sensu events and map/reduce them into alerts into “service” level events that you can then alert on.