Jef Practice: Use explicit configuration path for Ruby Sensu Plugin configs

Sensu Go
1

In Sensu Classic it was typical best practice to include specialized plugin configuration in the default Sensu configuration as json files under /etc/sensu/conf.d/ and have all the configuration for all plugins loaded as part of normal operation.

But in Sensu Go I find now that I prefer to segregate my plugin configuration into distinct plugin specific configuration paths to isolate configuration between plugins and teams. This helps isolate human errors in configuration syntax and provides a mechanism by which I can let different teams manage their own configurationā€¦by giving them separate configuration directories in which to hold needed plugin configuration.

Letā€™s take the existing opsgenie sensu plugin as an example.
It provides a handler that expect to find specalized opsgenie json configuration that includes information like the api access key. By default this handler, like all handlers based on the sensu-plugin gem, will look in /etc/sensu/conf.d/ for configuration. But I donā€™t want the default behavior. I want to explicitly use: /etc/sensu/teamA/opsgenie.json where teamA corresponds to the specific Sensu Go namespace i want to run the handler under.

What I can do in the Sensu Go handler is set the environment variable:
SENSU_CONFIG_FILES=/etc/sensu/teamA/opsgenie.json

Now my opsgenie handler is reading ONLY the configuration from that single file. A file only members of teamA uses for their workflow. For teamB, Iā€™d have a similar subdirectory where teamB can have its opsgenie configuration defined.

The SENSU_CONFIG_FILES envvar has always been available to support alternative locations for configuration as part of Sensu Classic server and client operation. It supports a list of files using the ā€˜:ā€™ path separator. With Sensu Go, its even more powerful because individual checks, handlers, and mutators can set that envvar explicitly for much more granular control over what configuration is uploaded as part of plugin operation. Itā€™s very flexible and great for workflows that give teams end-to-end control over their monitoring configuration.

2 Likes
2

Now if only we had handler token substitutionā€¦ then you could combine this pattern with tokens from the event (e.g. check or entity label/annotation data)ā€¦

:stuck_out_tongue:

3

This sparked a discussion between Ben and myself for post Sensu Classic EOL changes at the end of the yearā€¦ we could extend the base sensu-plugin package in a couple of different ways for all ruby plugins to benefit from.

My idea in the discussion was if we can expose the namespace that the handler/mutator/check is operating in via an envvar, the sensu-plugin config_files logic could use the auto detected namespace and extend the default config file paths to merge into settings automatically.

Itā€™s safer to do that sort of work after Classic EOLs to reduce the testing burden and to make sure we donā€™t accidently break existing classic users who are using a custom path override already.

4

I should definitely add this really only works if if all your teams are considered ā€œtrustedā€ as handlers running under different namespaces will still be able to read each others configurationā€¦and will be able to see sensitive information such as api keys and the like.

The benefit here, is you can give the sensu user running the backend service read only access to the configā€¦and give write access to each teamā€¦to prevent one team from accidentally adjusting a config and breaking another teamā€™s monitoring workflow.

Trying to run Sensu completely as multi-tenency service where namespaces are completely untrusted relative to each otherā€¦ is possibleā€¦but not with the community ruby plugins at present. Weā€™ll need to decide if its worth supporting that pattern, and figure out how to support sensitive information stored as encrypted values in the readonly filesystem configsā€¦ or via some other encrypted means.