Jef Practice: Build handlers that are annotation override aware

tl;dr
Annotation override support makes it really use to re-use handler resource definitions. The Sensu Plugin SDK makes it trivially easy to add annotation override support for golang based handlers. If you are writing handlers, make then annotation override aware!

The longer story…
Annotation overrides make it possible for handlers to use check or entity entity annotations to override commandline configuration, allow you to control how a handler command works based on contextual information in your entity or check definitions. We’ve baked into the golang SDK to make it easy for everyone building handlers to take advantage of this.
It’s a really powerful pattern, and the SDK makes it very easy to implement.
https://github.com/sensu-community/sensu-plugin-sdk#annotations-configuration-options-override

Basically once you define the annotation keyspace the handler will use, you just have to give each option that should be overridable via annotation a Path attribute, and the SDK logic handles the annotation overrides as part of configuration option parsing. The keyspace is essentially an annotation string prefix,

The obviously first use case is using annotation overrides to provide a flexible contact routing by annotating either the entity or the check for handlers that are used to contact a human.

For the golang sensu-email-handler
you can use entity and check level annotation overrides to create flexible email alert routing with a fallback email defined by the --toEmail commandline option as a catchall.

But this isn’t just for contact routing, the annotation overrides can be used for remediation patterns as well. For example the Ansible Tower handler uses the annotation overrides to let you defined different job templates to use for different entity contexts because you may need to run a different remediation playbook for an entity in AWS versus something running in physical data center. Using this mechanism, you only need to defined the single handler, and its entity specific or check specific configuration is an annotation in the entity or check resource.

In both cases, the handler resource definition stays simple and reusable, minimizing duplication.