Filter seems is not working for the events that send to pagerduty

melw · April 15, 2021, 8:07pm

I have a check that integrate with pagerduty and will page me once there are more than three Criticals events, with using the sensu-plugin/handler-pagerduty.rb (sensu-plugins-pagerduty/handler-pagerduty.rb at master · sensu-plugins/sensu-plugins-pagerduty · GitHub) The event mapped the into sensugo with sensu-plugin/sensu-handler.rb at master · sensu-plugins/sensu-plugin · GitHub

Now the issue is I got paged for even a single error event(there are mainly two issues in this case. First, the occurrences filter here is not working, and second issue is, it couldn’t override the non-critical event into a correct event severity. i.e. when the event send to pagerduty, all kinds of events treated at a Critical level. For the second issue I find this plugin file could solve the issue sensu-plugins-pagerduty/mutator-pagerduty-priority-override.rb at master · sensu-plugins/sensu-plugins-pagerduty · GitHub, but occurrences filter not working to page us only if above 3 events is still a problem at this moment)

Could you please have a look at the issue, this is weird because the occurrences filter is working for email and slack alert handlers, only when sending pager duty, it is not working. Is there a way can solve this issue?

The event json file:

{
  "check": {
    "command": "echo testing..... && /4xx-error-rates.py --endpoint \"%\" --status_code \"401\"",
    "handlers": [
      "pd-test"
    ],
    "high_flap_threshold": 0,
    "interval": 300,
    "low_flap_threshold": 0,
    "publish": true,
    "runtime_assets": null,
    "subscriptions": [
      "runner",
      "runner-stg"
    ],
    "proxy_entity_name": "api-checker",
    "check_hooks": null,
    "stdin": false,
    "subdue": null,
    "ttl": 0,
    "timeout": 0,
    "round_robin": true,
    "duration": 0.10803472,
    "executed": 1618516354,
    "history": [
      {
        "status": 127,
        "executed": 1618511247
      },
      {
        "status": 127,
        "executed": 1618511547
      },
      {
        "status": 0,
        "executed": 1618511547
      },
      {
        "status": 127,
        "executed": 1618511848
      },
      {
        "status": 127,
        "executed": 1618512148
      },
      {
        "status": 127,
        "executed": 1618512449
      },
      {
        "status": 127,
        "executed": 1618512749
      },
      {
        "status": 127,
        "executed": 1618513049
      },
      {
        "status": 127,
        "executed": 1618513350
      },
      {
        "status": 127,
        "executed": 1618513650
      },
      {
        "status": 127,
        "executed": 1618513951
      },
      {
        "status": 127,
        "executed": 1618514251
      },
      {
        "status": 127,
        "executed": 1618514552
      },
      {
        "status": 127,
        "executed": 1618514852
      },
      {
        "status": 127,
        "executed": 1618515152
      },
      {
        "status": 127,
        "executed": 1618515453
      },
      {
        "status": 127,
        "executed": 1618515753
      },
      {
        "status": 127,
        "executed": 1618516054
      },
      {
        "status": 127,
        "executed": 1618516056
      },
      {
        "status": 0,
        "executed": 1618516056
      },
      {
        "status": 127,
        "executed": 1618516354
      }
    ],
    "issued": 1618516354,
    "output": "testing.....\nsh: /4xx-error-rates.py: No such file or directory\n",
    "state": "failing",
    "status": 127,
    "total_state_change": 20,
    "last_ok": 1618516056,
    "occurrences": 1,
    "occurrences_watermark": 1,
    "output_metric_format": "",
    "output_metric_handlers": null,
    "env_vars": null,
    "metadata": {
      "name": "apiv3_errors_401",
      "namespace": "stg",
      "labels": {
        "sensu.io/managed_by": "sensuctl",
        "notification": "APIV3 403 Error Rate"
      }
    },
    "secrets": null,
    "is_silenced": false,
    "scheduler": "etcd"
  },
  "entity": {
    "entity_class": "proxy",
    "system": {
      "network": {
        "interfaces": null
      },
      "libc_type": "",
      "vm_system": "",
      "vm_role": "",
      "cloud_provider": "",
      "processes": null
    },
    "subscriptions": [
      "entity:api-checker"
    ],
    "last_seen": 0,
    "deregister": false,
    "deregistration": {},
    "metadata": {
      "name": "api-checker",
      "namespace": "stg"
    },
    "sensu_agent_version": ""
  },
  "id": "0cc06dd9-959d-4a7c-be2b-2df4986aa757",
  "metadata": {
    "namespace": "stg"
  },
  "sequence": 1,
  "timestamp": 1618516354
}

The handler and the filter configuration:

type: Handler
api_version: core/v2
metadata:
  name: 'pd-test'
  namespace: stg
spec:
  type: set
  filters:
   - is_incident
   - not_silenced
   - emb-occurrences_3_filter
  handlers:
   - pagerduty_api_sre

---
type: Handler
api_version: core/v2
metadata:
  name: 'pagerduty_api_sre'
  namespace: stg
spec:
  command: "/opt/sensu-plugins-ruby/embedded/bin/handler-pagerduty.rb -j pagerduty_api_sre --map-go-event-into-ruby"
  type: pipe
  filters:
  - is_incident
  - not_silenced

Filter configuration

---
  type: EventFilter
  api_version: core/v2
  metadata:
    name: "emb-occurrences_3_filter"
    namespace: stg
  spec:
    action: allow
    expressions:
    - event.check.occurrences >= 3

jspaleta · April 16, 2021, 2:37am

Hey,
I’ll try to reproduce this as soon as I can.
The fact that you say its working for email with the same filter is very interesting.
Can you provide the handler definition for the thing that is working as you expect so I can compare?

melw · April 16, 2021, 5:12pm

Hi,

Looks like the filter is also not working now , it worked once when I trigger the execute button to generate a testing error event from UI. I’m using this plugin for email handler

github.com

sensu-plugins/sensu-plugins-mailer/blob/master/bin/handler-mailer.rb

#!/usr/bin/env ruby
#
# Sensu Handler: mailer
#
# This handler formats alerts as mails and sends them off to a pre-defined recipient.
#
# Copyright 2012 Pal-Kristian Hamre (https://github.com/pkhamre | http://twitter.com/pkhamre)
#
# Released under the same terms as Sensu (the MIT license); see LICENSE
# for details.

# Note: The default mailer config is fetched from the predefined json config file which is "mailer.json" or any other
#       file defined using the "json_config" command line option. The mailing list could also be configured on a per client basis
#       by defining the "mail_to" attribute in the client config file. This will override the default mailing list where the
#       alerts are being routed to for that particular client.

require 'sensu-handler'
require 'mail'
require 'timeout'
require 'erubis'

This file has been truncated. show original

I’m so confused why the filter is not working

---
type: Handler
api_version: core/v2
metadata:
  name: "handler-filter-test"
  namespace: stg
spec:
  type: set
  filters:
  - not_silenced
  - is_incident
  - emb-occurrences_3_filter
  handlers:
  - slack-integration-tests
  - mailer

---
type: Handler
api_version: core/v2
metadata:
  name: 'mailer'
  namespace: stg
spec:
  command: "/opt/sensu-plugins-ruby/embedded/bin/handler-mailer.rb -j test-mail -c html -s '[Sensu]' --map-go-event-into-ruby"
  type: pipe
  filters:
  - is_incident
  - not_silenced

jspaleta · April 16, 2021, 6:12pm

okay,
well that’s a little less weird.

melw · April 16, 2021, 6:22pm

so why the filter with occurrences checking expression doesn’t work in sensu go? Shall I have to use the fatigue_check asset for implement the occurrences keyword from sensu core?

jspaleta · April 16, 2021, 6:24pm

i need to play with your filter a bit to understand what’s going on. I’ll look at it today

melw · April 19, 2021, 6:05pm

Hi jspaleta,

I doubt this could be some configuration issues in the server side, since we have two systems(dev build and the production one which including multiple namespaces for testing production namespace and a real production namespace). And the same setup in the dev build system is working. Do you have any hint of which configuration could be related with and I can look into the differences of the two systems

anatolijd · April 21, 2021, 10:57am

Hi ,

Isn’t it because pd-test is a handler set and not a regular filter ?

Filters specified in a handler set definition have no effect. Filters must be specified in individual handler definitions.

melw · April 21, 2021, 6:42pm

oh! This is a nice hint! I just checked, this note is from version 6.0, and I’m using the version 6.2, does this rule applied to the latest version as well?

jspaleta · April 21, 2021, 7:11pm

yes

Ref: https://docs.sensu.io/sensu-go/6.2/observability-pipeline/observe-filter/filters/#event-filter-example-minimum-required-attributes

Small note, you can navigate to the documentation verison that matches the version you are using by using the version dropdown at the top of the docs page.

Topic		Replies	Views
Handler behavior when check goes from warning to critical Sensu Classic (EOL)	2	694	September 10, 2014
Filters on sensu plugins Sensu Classic (EOL)	1	431	May 6, 2015
sensu-api /event/resolve issue Sensu Classic (EOL)	3	444	November 22, 2018
Handler not always sending resolve events Sensu Classic (EOL)	3	1460	December 3, 2019
Simulate the "occurrences" check attribute from old Sensu Sensu Go	3	443	April 26, 2022

Filter seems is not working for the events that send to pagerduty

Related topics