Sensu Go plugin distribution model

I am migrating our legacy Sensu Core setup to Sensu Go.

With Sensu Core, I would sensu-install a plugin on the agent itself. However, this does not seem to be the case anymore in the docs: there is no mention of installing plugins on agents there at all. This gives me the idea that plugins are distributed to agents with certain subscriptions from the Sensu Go server after adding the asset there.

Question: Is this correct? If not, how are Sensu plugins (in this case plugins from Bonsai) distributed to agent machines?

Also, the documentation is really vague about this. It should be more clear.

There is no reason you could not continue to use sensu-install to install a plugin on the agent itself. The only requirement checks have is that the command specified be available on the system (and either in the PATH if not fully qualified).

Assets in Sense Go look to alleviate the need of installing the plugins on the agent (either through the build process or via configuration management) in advance. Instead they are downloaded and installed when referenced and then cached for future use.

Assets themselves are nothing more than tar or zip archives that consist of a plugin’s bin/, lib/, and include/ directories. These archive files are downloaded from the specified URL in the asset definition. This URL can be anything that is accessible to the agent (or backend in the case of handler and mutator assets), whether it be Bonsai, GitHub, an internal asset repository like Artifactory, or simply a web server running NGINX. Unless you also run a web server on your Sensu Go backend, assets are not distributed from the Sensu Go backed itself.

Here’s the workflow from the perspective of an agent wanting to run a check that’s available in the sensu/monitoring-plugins asset on Bonsai. It assumes the asset definition exists as if you ran sensuctl asset add sensu/monitoring-plugins.

  1. Agent has subscription for “linux” checks
  2. Check “check_disk” is part of the “linux” subscription and has a runtime asset defined of “sensu/monitoring-plugins”
  3. The Agent is scheduled to run “check_disk” at the next interval
  4. The Agent sees that asset “sensu/monitoring-plugins” is needed and the current asset definition has a SHA512 of XXYYXXX (shortened for brevity)
  5. The agent checks it cache to see if that asset currently exists locally (/var/cache/sensu/sensu-agent/XXYYXXX)
  6. If it doesn’t exist, it is downloaded from the specified URL and unarchived into that cache directory
  7. The bin/ directory from that cached location is added to PATH, the lib/ directory is added to LD_LIBRARY_PATH
  8. The check command is executed

The beauty of this model is that you don’t have to worry about PATH, etc. Assets that are in Bonsai are fronted by a CDN to provide performant downloads. Many of the existing Sensu Core plugins are available on Bonsai and make use of a Ruby runtime asset.

Right, that’s what I thought.
Thanks.

What’s still not clear to me, and what doesn’t seem to be clear to other either 1 2, is:

  • Executing checks with commands from assets with sudo
  • Having the sensu user be able to find the path to commands from assets

With Sensu Core, I added secure_path to sudoers. I cannot do the same with Sensu Go, as the path to an asset is unknown (/var/cache/sensu/sensu-agent/$somethingunknown). Then, I would allow the sensu user to execute anything in /opt/sensu/embedded/bin, which I - again - cannot do as the path to asset’s commands are unknown.

How are others solving this issue?

1 Like

Hoping someone can explain how they’re running commands on the agent machine without a predictable assets location.

I also find the sensu go assets location extremely frustrating. The docs do not give the explanation that @todd has mentioned and so I’ve spent a significant time debugging where the command is installed. For now I’ve resorted to installing packages through debian packages so I have a predictable path.

@todd is the explanation you gave documented anywhere in the sensu go docs?

That explanation can be found here.

Hey,

yeah anything that requires sudo becomes difficult to recommand because there are so many trade-offs. I’m always wary about trying to automate solutions around sudo users and assets because inevitably it has security ramifications. So HUGE caveat on what I’m about to suggest.

Because of the way secure_path option works, it probably interferes with how sensu-agent updates the PATH. Assets are designed so that only the paths defined for the assets you need are prefixed into the default executable path. You definitely do not want to include all possible assets that are cached.

Instead of mucking with sudoers, try this workaround…
for example I’m able to run this check command using the system-profile-linux asset:

sudo PATH=$PATH bash -c "system-profile-linux"

All i had to do was give sensu user the agent is running as NOPASSWORD access to run commands, with no mucking with the secure-path

Now should this work? probably not this seems like a bug in the intent for secure_path, but it is what it is.

Now you may be able to go further and lock down that sensu user or group in sudoers further than I did, but the key is being able to pass the PATH the agent wants to run into the environment sudo is setting up that bash reads in. In fact there’s probably room to replace that bash call with a more restricted shell. Ultimately what is going on here is that sudo secure_path allows for bash to be run, and then the root user is just running commands under bash.

Like I said HUGE caveats here with regard to security. Once you let the sensu agent run bash under sudo you’ve got keys to the kingdom really you can basically run anything under bash like this. So please think this through and lock down the sensu user accordingly.

Not sure why this has been made so difficult. I just want to install a plugin, then know exactly where it is. I don’t like the complexity with ‘runtime assets’, not knowing where that asset actually is, and thus having to come up with awkward ‘solutions’ to be able to sudo. The use case to sudo execute a plugin does not seem so rare for a monitoring system.

Anyway, can I just keep doing sensu-install for years to come, or is that a soon to be unsupported concept?

hey,
Just to be clear, you do not have to use assets. In fact there are some workloads where people want assets disabled entirely…and that’s fine…the Sensu agent actually has a tunable to disable asset downloading entirely to support that use case. And really everything that is an asset is just a tarball that you can install and unpack manually as part of CM if you want really tight control over what is install on the system and where.

sensu-install is a thin wrapper over rubygem, so even if it goes a way at some unspecified time in the future you’ll still be able to use the gem tool. sensu-install only supports Sensu plugins written in ruby and packaged and published as gems. How long you’ll be able to use those gems depends on how much forward momentum contributors have to maintain the plugins you are using. There’s no effort to to prevent gem installed ruby plugins from working. Some of those gems, especially the handler plugins might need to be updated to work with the Sensu Go event json, but we tried to anticipate that by providing wrapper logic to convert the event json back to what the handler expects…as a forward compatible change in the lastest sensu-plugin gem.

However, much of the newer work is around plugins written in golang and statically cross compiled instead of Ruby as the static Golang binaries work really well for a larger cross section of workloads, including sensu agent sidecar containers. The Ruby stuff, just because it relies on the heavy Ruby environment to be present…doesn’t fit well in the containerized workloads so much. Ironically, the sudo issues aren’t a big deal in that workload generally either because right now the agent is run as root in the official Sensu container right now.

Previously it was possible to rely heavily on Ruby’s handling of shared libraries, but assets even though more complicated provide a way to support a wider range of languages. They aren’t just binaries, but can also include libraries… we can’t just dump all of that into the same path… because different assets may require different compiled versions of the same library.

All of that being said, there’s certainty room to improve assets. I put in this straw proposal to extend assets to optionally allow for predictable binary installs awhile ago.
https://github.com/sensu/sensu-go/issues/3046

Hi,

Thanks for your response. This is clear, but sensu-plugins-ruby - which provides sensu-install - is not available for Debian, which doesn’t make things easier.

Uhm I’m pretty sure the repo has a deb package.

Yes, but Debian does not seem to be supported. deb https://packagecloud.io/sensu/community/debian/ %s main, where %s is buster or stretch, returns a 404. Ubuntu does seem to be available.

This issue has been open for over a year: https://github.com/sensu/sensu-plugins-omnibus/issues/3

Anyway, this is a bit off topic.

Hmm… that’s a bug. Let me see if I can get that corrected…in the packagecloud setup.

In the meantime you should be able to manually install the deb…or edit the repo definition on your system. I do that to get yum repo working on fedora.

Any progress on that would be very welcome :slight_smile:

Also you have the option to install a ruby environment with something like rvm and use ruby gem command to install existing sens-plugins ruby gems. The sensu-plugins-ruby package isn’t strictly needed. The sensu-install executable is a very thin wrapper over ruby gem.

Honestly, even for the rpm based systems I’m using. I don’t bother installing that package, and rely on rvm instead to give me a ruby environment where I can’t use assets.

Thanks. I feel like I should stick to the documentation to maintain a supported setup, though…

Thanks for everyone’s replies in this topic. I ended up using runtime assets and preserving the environment by passing PATH when sudo’ing. My Sensu Go deployment is ready…

1 Like