Introducing curated configurations: a new way to contribute and share operational expertise!

jspaleta · April 24, 2020, 8:04pm

Hey!

As some of you might already be aware, we’ve been working on a new idea to make it possible to share curated Sensu resource configurations and to help bootstrap new Sensu users into effective monitoring workloads more quickly. By making use of Sensu Go features such as assets and secrets management, it’s finally possible to share monitoring workflow configurations widely and safely.

We’ve started the process, created several curated workflows, and we think it’s ready to share, but we need your help to take it further. This is a great opportunity to share operational knowledge with the rest of the Sensu community. If there is a handler or check you have operational experience with, we’d like to invite you to get involved in the configuration curation! If your interested take a look at:

https://github.com/sensu-community/monitoring-checks
https://github.com/sensu-community/monitoring-pipelines

Okay that was the tease, here’s the longer rundown.

We’ve split the configuration curation into two separate repositories. One for checks that run on a Sensu agent, and a separate repository for pipelines that run on a Sensu backend. These repositories are designed to work together, but we felt it made logical sense to split the functionality at the backend/agent boundary.

Take a look at the nginx metrics check from the collection to get a sense of what a curated configuration looks like:

github.com

sensu-community/monitoring-checks/blob/master/nginx/metrics-nginx.yaml

---
# Description
#   Provide NGINX metrics from the status module
#
# Instructions
#   1. Add the "nginx" subscription to agents that should run this check.
#   2. Ensure the ngnix status module is enabled on the server(s) to be checked.
#      http://nginx.org/en/docs/http/ngx_http_status_module.html
#   3. More configuration options are available for this plugin; please see the
#      plugin documentation for more details.
#
# Documentation
#   - Usage: https://github.com/sensu-plugins/sensu-plugins-nginx#usage
#
# Contributors
#   The following individuals have contributed to this configuration template:
#   - Todd Campbell, @nixwiz
#   - Caleb Hailey, @calebhailey
type: CheckConfig
api_version: core/v2

This file has been truncated. show original

The curated check configurations not only include the check resource but also the versioned asset resources as well. All you need to do is install the single yaml file to have an unpublished check staged into your Sensu namespace that is ready to interact with handler sets defined from the curated monitoring-pipelines. Our hope is these curated checks will get people up and running with common monitoring tasks that can then be easily optimized once installed.

Speaking of, the curated monitoring pipelines on the other hand run on the Sensu backend, and are meant to make it easier for new users to integrate with the service integrations they need to interact with. These curated pipeline configurations make use of Sensu secrets management, so that sensitive information isn’t encoded into the publicly shared resources. Take a look at the curated pagerduty pipeline for example:

github.com

sensu-community/monitoring-pipelines/blob/master/incident-management/pagerduty.yaml

---
# Description
#   The Sensu PagerDuty Handler is a Sensu Event Handler that alerts to
#   the PagerDuty incident reponse service.
#
# Instructions
#   1. Add "pagerduty" to the "incident-management" handler set.
#
#      To create the "incident-management" handler set, please run:
#
#      $ sensuctl handler create incident-management --type set --handlers pagerduty
#
#      If you already have a handler set called "incident-management", please run:
#
#      $ sensuctl edit handler incident-management
#
#      NOTE: the sensuctl edit command will open your default editor; please
#      modify the "handlers" attribute by adding "pagerduty".
#
#   2. Configure the following secrets using your preferred Sensu Go Secrets

This file has been truncated. show original

This curation defines a handler and an asset, but also defines a Sensu secret resource, to instruct the Sensu backend to pull the pagerduty token from the backend environment. This makes it safe to share the configuration without fear of leaking sensitive information,

Please take a look at these configuration repositories and see if there’s an operational workflow in your environment that you could help curate. We’ve stubbed in the directory structure to help give a sense of the coverage that is possible based on the Sensu plugins that exist, but we could really use operator expertise to fill in the gaps. If you’re interested in helping, take a look at the Readme in the repos, we’ve tried to document what we think makes for a great curated configuration.

We are also planning to hold a workshop session around curations during the upcoming virtual summit, May 19-21. We’d love to work together on this with people who want to contribute to this effort during that workshop. Please register for the our virtual summit and join the live conversation.

Dr_Ogg · April 27, 2020, 7:52pm

looks like a great start, would be nice to see a reference to were to install the asset from in the check definition itself, so when you create the new check definition, sensu could on its own check for the required assets and install. or if you had an internal asset server, to reference when instead. Also would be nice to have some references for humans, eg were did this check come from? does it have a home page, and is there a playbook to resolving this issue for humans to follow when resolving?

jspaleta · April 27, 2020, 8:22pm

Hey!

We’re definitely trying to make sure the yaml files contain useful comment blocks, so its possible for operators to read along through the yaml comments and make appropriate edits to adapt the configuration. I’m not sure i fully understand some of what your suggesting, it might be useful talk through your suggestions for an existing curation, the check-http should be familiar enough for most people reading along.

github.com

sensu-community/monitoring-checks/blob/master/http/check-http.yaml

---
# Description
#   Check that an HTTP URL is accessible and returns a 200 response.
#
# Instructions
#   1. Add the "http" subscription to agents that should run this check.
#   2. More configuration options are available for this plugin; please see the
#      plugin documentation for more details.
#
# Documentation
#   - Usage: https://github.com/sensu-plugins/sensu-plugins-http#usage-examples
#
# Contributors
#   The following individuals have contributed to this configuration template:
#   - Todd Campbell, @nixwiz
type: CheckConfig
api_version: core/v2
metadata:
  name: check-http
spec:

This file has been truncated. show original

This yaml file starts with a comment block that includes a reference to the check’s upstream documentation. Are you looking for a more expansive reference as to where the check came from?

I’m not sure what you meant when as “to where to install the asset from the check” in your comments. The yaml defines the check and a both the needed assets together in a single file, pulling assets from the public bonsai asset collection. If a user needs to use a privately hosted asset, it should be a simple matter of editing this file and changing the referenced assets in the check definition. Is there something else you were looking for? Do you have an idea on how to better comment the file?

The playbook comment is a bit broader. We don’t have a space yet for published remediation playbooks that match up with the templates, but if there was interest in operators in our community to share that, we could definitely make a space for that. It’s sort of a chicken and egg problem, I’d love to come back and add curated remediation playbook suggestions into the check comment blocks once we have some curated playbooks published. The related repository for monitoring-pipelines does anticipate the development of curated remediation pipelines. If people wanted to start contributing curated playbooks, they probably could start with PRs to the monitoring-pipelines repository for now.

Thanks for the feedback!
-Jef

Dr_Ogg · April 27, 2020, 8:43pm

simply, the check should have the URL to the asset, such that we don’t need to include the required assets in file to get bootstrapped .

jspaleta · April 27, 2020, 9:03pm

Ah you want to have a way to encode an equivalent of sensuctl asset add statement inside a check so when creating a check definition and interact with an asset repository api so the asset resource is generated and installed if needed behind the scenes? That’s an interesting idea.

Topic		Replies	Views
[Webinar] Practical monitoring as code Webinars	0	330	February 27, 2021
Sensu docs digest for March 2022 Sensu Docs docs-digest	0	225	March 31, 2022
Sensu docs digest for March 2021 Sensu Docs docs-digest	1	380	April 14, 2021
Sensu docs digest for May 2021 Sensu Docs docs-digest	0	314	May 28, 2021
Sensu docs digest for February 2021 Sensu Docs docs-digest	0	292	February 26, 2021

Introducing curated configurations: a new way to contribute and share operational expertise!

Related topics