Sensu Go plugin distribution model

I am migrating our legacy Sensu Core setup to Sensu Go.

With Sensu Core, I would sensu-install a plugin on the agent itself. However, this does not seem to be the case anymore in the docs: there is no mention of installing plugins on agents there at all. This gives me the idea that plugins are distributed to agents with certain subscriptions from the Sensu Go server after adding the asset there.

Question: Is this correct? If not, how are Sensu plugins (in this case plugins from Bonsai) distributed to agent machines?

Also, the documentation is really vague about this. It should be more clear.

There is no reason you could not continue to use sensu-install to install a plugin on the agent itself. The only requirement checks have is that the command specified be available on the system (and either in the PATH if not fully qualified).

Assets in Sense Go look to alleviate the need of installing the plugins on the agent (either through the build process or via configuration management) in advance. Instead they are downloaded and installed when referenced and then cached for future use.

Assets themselves are nothing more than tar or zip archives that consist of a plugin’s bin/, lib/, and include/ directories. These archive files are downloaded from the specified URL in the asset definition. This URL can be anything that is accessible to the agent (or backend in the case of handler and mutator assets), whether it be Bonsai, GitHub, an internal asset repository like Artifactory, or simply a web server running NGINX. Unless you also run a web server on your Sensu Go backend, assets are not distributed from the Sensu Go backed itself.

Here’s the workflow from the perspective of an agent wanting to run a check that’s available in the sensu/monitoring-plugins asset on Bonsai. It assumes the asset definition exists as if you ran sensuctl asset add sensu/monitoring-plugins.

  1. Agent has subscription for “linux” checks
  2. Check “check_disk” is part of the “linux” subscription and has a runtime asset defined of “sensu/monitoring-plugins”
  3. The Agent is scheduled to run “check_disk” at the next interval
  4. The Agent sees that asset “sensu/monitoring-plugins” is needed and the current asset definition has a SHA512 of XXYYXXX (shortened for brevity)
  5. The agent checks it cache to see if that asset currently exists locally (/var/cache/sensu/sensu-agent/XXYYXXX)
  6. If it doesn’t exist, it is downloaded from the specified URL and unarchived into that cache directory
  7. The bin/ directory from that cached location is added to PATH, the lib/ directory is added to LD_LIBRARY_PATH
  8. The check command is executed

The beauty of this model is that you don’t have to worry about PATH, etc. Assets that are in Bonsai are fronted by a CDN to provide performant downloads. Many of the existing Sensu Core plugins are available on Bonsai and make use of a Ruby runtime asset.

Right, that’s what I thought.
Thanks.

What’s still not clear to me, and what doesn’t seem to be clear to other either 1 2, is:

  • Executing checks with commands from assets with sudo
  • Having the sensu user be able to find the path to commands from assets

With Sensu Core, I added secure_path to sudoers. I cannot do the same with Sensu Go, as the path to an asset is unknown (/var/cache/sensu/sensu-agent/$somethingunknown). Then, I would allow the sensu user to execute anything in /opt/sensu/embedded/bin, which I - again - cannot do as the path to asset’s commands are unknown.

How are others solving this issue?

Hoping someone can explain how they’re running commands on the agent machine without a predictable assets location.

I also find the sensu go assets location extremely frustrating. The docs do not give the explanation that @todd has mentioned and so I’ve spent a significant time debugging where the command is installed. For now I’ve resorted to installing packages through debian packages so I have a predictable path.

@todd is the explanation you gave documented anywhere in the sensu go docs?

That explanation can be found here.

Hey,

yeah anything that requires sudo becomes difficult to recommand because there are so many trade-offs. I’m always wary about trying to automate solutions around sudo users and assets because inevitably it has security ramifications. So HUGE caveat on what I’m about to suggest.

Because of the way secure_path option works, it probably interferes with how sensu-agent updates the PATH. Assets are designed so that only the paths defined for the assets you need are prefixed into the default executable path. You definitely do not want to include all possible assets that are cached.

Instead of mucking with sudoers, try this workaround…
for example I’m able to run this check command using the system-profile-linux asset:

sudo PATH=$PATH bash -c "system-profile-linux"

All i had to do was give sensu user the agent is running as NOPASSWORD access to run commands, with no mucking with the secure-path

Now should this work? probably not this seems like a bug in the intent for secure_path, but it is what it is.

Now you may be able to go further and lock down that sensu user or group in sudoers further than I did, but the key is being able to pass the PATH the agent wants to run into the environment sudo is setting up that bash reads in. In fact there’s probably room to replace that bash call with a more restricted shell. Ultimately what is going on here is that sudo secure_path allows for bash to be run, and then the root user is just running commands under bash.

Like I said HUGE caveats here with regard to security. Once you let the sensu agent run bash under sudo you’ve got keys to the kingdom really you can basically run anything under bash like this. So please think this through and lock down the sensu user accordingly.