Pattern for migrating 100's of checks

Sensu Go
1

For our upcoming sensu-go setup, we want to move away from overly complicated ansible playbooks for checks. to a template version.

I think I’m in the wrong path: if I want to generate checks from static data, let’s say I have 2 of them, configuration looks pretty simple:

sensu_checks:
    - name: check_cpu_usage
      runbook: "https://confluence.corp.com/display/DevOps/Redis+sentinel+alert+troubleshooting"
      namespace: default
      annotations: {}
      labels:
        key1: value
      command: >-
        check-cpu.rb -c 95 -w 70
      handlers: 
        - notifyallthethings
      interval: 100
      publish: true
      subscriptions:
        - defaults
   -  name: api_errors_5XX_aggregate
    custom:
      notification: APIError Rate
    source: api-checker
    command: >-
      api-error-rates.py
      -a {{ new_relic_nrql_token }}
      -c {{ sensu_thresholds['critical']['5xx'] }}
      -w {{ sensu_thresholds['warning']['5xx'] }}
      -r 5
    subscribers:
    -runner
    interval: "{{ sensu_default_interval }}"
    occurrences: "{{ sensu_default_occurrences }}"
    handlers: APIError500Handler
    refresh: "{{ refresh_default }}"

However, generating the check config it’s getting complicated because I have to account for the optional properties:

{%- for check in sensu_go_checks['sensu_checks'] -%}
---
type: CheckConfig
api_version: core/v2
metadata:
  annotations:
    sensu.io.json_attributes: |
      { 
        "runbook" :  "{{ check['runbook'] }}"
      }
    {%- for k,v in check['annotations'].items() %}
    {{ k }}: '{{ v }}'
    {%- endfor %}
  labels:
    {%- for k,v in check['labels'].items()  %}
    {{ k }}: '{{ v }}'
    {%- endfor %}
  name: {{ check['name'] }}
  namespace: {{ check['namespace'] }}
spec:
  command: {{ check['command'] }}
 ... (a ton of if-for statements for every property in check)
  {%- if check['round_robin'] %}
  round_robin: true 
  {%- endif %}
  subscriptions:
  {%- for s in check['subscriptions'] %}
  - {{ s }}
  {% endfor %}
{% endfor %}

On top of it, Now I have to do sensuctl create --file my_big.yml instead of just copying the files to the server.

I feel I’m implementing sensu classic again in sensu-go, Is there a pattern to:

  1. avoid so many conditions by token substitution
  2. re-use the same yml template for every similar check, just change parameters
  3. integrate with a repo, so each time checks are built by CI, they get updated to sensu

Thanks a lot for your help and to @jspaleta for starting the conversation.
M.

1 Like
2

Hey!

This is a great topic for discussion. I expect a lot of people who were previously relying on configuration management templating to generate static stand alone checks client side are probably grappling with similar problems now.

Personally, I think this is a situation where you want to use agent level token substitution and define the Sensu resources as static configurations for your configuration management to upload to the sensu-backend server without doing any config management specific templating.

This pattern should be much easier to audit. And depending on your workload, you may find that you can reduce the number of unique check definitions in the system with entity level token substitution. Depends on your workload for sure.

So real high level pattern for now:

  1. use static declarative Sensu Go resource definitions
    (checks, handlers, filters, mutators)
    optionally keep these in a dedicated repository. Use this as a resource for your config management.
    The idea here is these resources should be valid to upload manually using sensuctl as is without manipulation from the configuration management.
    Note: It’s a great idea to setup production specific Sensu Go namespace(s) and have your configuration management set the namespace to use when uploading these Sensu Go resources. This is a really good use case for Sensu Go RBAC, to help you manually test changes in one namespace, without worry of disrupting the namespace your production automation uses to deploy approved changes.

  2. config management generated agent.yaml for agent entities
    place agent specific metadata in labels and annotations using config management templating.

  3. config management generated entity resource configs for proxy entities
    place proxy entity specific metadata in labels and annotations using config management templating

  4. update the static resource definitions to reference the correct entity level labels and annotations.
    For checks this would mean using token substitution. This effectively replaces any host inventory specific templating prevoiusly used in config management to differentiate check details.
    For filters,mutators and handlers, this means generally means referencing the correct event attribute.

For migrating an existing collection of checks, I’d start with the checks configuration as rendered by the existing configuration management to make sure all the cm specific templating was rendered. I’d then run the translator against those resources and work to rebuild the new collection.

Pro tip, until the sensu-translator tool learns how to output yaml natively, you can use sensuctl to re-export back to yaml if you prefer to keep your new collection of Sensu Go in yaml format.

I’ll try to come back with a worked example based on what you provided.

2 Likes