I am migrating our legacy Sensu Core setup to Sensu Go.
With Sensu Core, I would sensu-install a plugin on the agent itself. However, this does not seem to be the case anymore in the docs: there is no mention of installing plugins on agents there at all. This gives me the idea that plugins are distributed to agents with certain subscriptions from the Sensu Go server after adding the asset there.
Question: Is this correct? If not, how are Sensu plugins (in this case plugins from Bonsai) distributed to agent machines?
Also, the documentation is really vague about this. It should be more clear.
There is no reason you could not continue to use sensu-install to install a plugin on the agent itself. The only requirement checks have is that the command specified be available on the system (and either in the PATH if not fully qualified).
Assets in Sense Go look to alleviate the need of installing the plugins on the agent (either through the build process or via configuration management) in advance. Instead they are downloaded and installed when referenced and then cached for future use.
Assets themselves are nothing more than tar or zip archives that consist of a pluginās bin/, lib/, and include/ directories. These archive files are downloaded from the specified URL in the asset definition. This URL can be anything that is accessible to the agent (or backend in the case of handler and mutator assets), whether it be Bonsai, GitHub, an internal asset repository like Artifactory, or simply a web server running NGINX. Unless you also run a web server on your Sensu Go backend, assets are not distributed from the Sensu Go backed itself.
Hereās the workflow from the perspective of an agent wanting to run a check thatās available in the sensu/monitoring-plugins asset on Bonsai. It assumes the asset definition exists as if you ran sensuctl asset add sensu/monitoring-plugins.
Agent has subscription for ālinuxā checks
Check ācheck_diskā is part of the ālinuxā subscription and has a runtime asset defined of āsensu/monitoring-pluginsā
The Agent is scheduled to run ācheck_diskā at the next interval
The Agent sees that asset āsensu/monitoring-pluginsā is needed and the current asset definition has a SHA512 of XXYYXXX (shortened for brevity)
The agent checks it cache to see if that asset currently exists locally (/var/cache/sensu/sensu-agent/XXYYXXX)
If it doesnāt exist, it is downloaded from the specified URL and unarchived into that cache directory
The bin/ directory from that cached location is added to PATH, the lib/ directory is added to LD_LIBRARY_PATH
The check command is executed
The beauty of this model is that you donāt have to worry about PATH, etc. Assets that are in Bonsai are fronted by a CDN to provide performant downloads. Many of the existing Sensu Core plugins are available on Bonsai and make use of a Ruby runtime asset.
Whatās still not clear to me, and what doesnāt seem to be clear to other either 12, is:
Executing checks with commands from assets with sudo
Having the sensu user be able to find the path to commands from assets
With Sensu Core, I added secure_path to sudoers. I cannot do the same with Sensu Go, as the path to an asset is unknown (/var/cache/sensu/sensu-agent/$somethingunknown). Then, I would allow the sensu user to execute anything in /opt/sensu/embedded/bin, which I - again - cannot do as the path to assetās commands are unknown.
I also find the sensu go assets location extremely frustrating. The docs do not give the explanation that @todd has mentioned and so Iāve spent a significant time debugging where the command is installed. For now Iāve resorted to installing packages through debian packages so I have a predictable path.
@todd is the explanation you gave documented anywhere in the sensu go docs?
yeah anything that requires sudo becomes difficult to recommand because there are so many trade-offs. Iām always wary about trying to automate solutions around sudo users and assets because inevitably it has security ramifications. So HUGE caveat on what Iām about to suggest.
Because of the way secure_path option works, it probably interferes with how sensu-agent updates the PATH. Assets are designed so that only the paths defined for the assets you need are prefixed into the default executable path. You definitely do not want to include all possible assets that are cached.
Instead of mucking with sudoers, try this workaroundā¦
for example Iām able to run this check command using the system-profile-linux asset:
sudo PATH=$PATH bash -c "system-profile-linux"
All i had to do was give sensu user the agent is running as NOPASSWORD access to run commands, with no mucking with the secure-path
Now should this work? probably not this seems like a bug in the intent for secure_path, but it is what it is.
Now you may be able to go further and lock down that sensu user or group in sudoers further than I did, but the key is being able to pass the PATH the agent wants to run into the environment sudo is setting up that bash reads in. In fact thereās probably room to replace that bash call with a more restricted shell. Ultimately what is going on here is that sudo secure_path allows for bash to be run, and then the root user is just running commands under bash.
Like I said HUGE caveats here with regard to security. Once you let the sensu agent run bash under sudo youāve got keys to the kingdom really you can basically run anything under bash like this. So please think this through and lock down the sensu user accordingly.
Not sure why this has been made so difficult. I just want to install a plugin, then know exactly where it is. I donāt like the complexity with āruntime assetsā, not knowing where that asset actually is, and thus having to come up with awkward āsolutionsā to be able to sudo. The use case to sudo execute a plugin does not seem so rare for a monitoring system.
Anyway, can I just keep doing sensu-install for years to come, or is that a soon to be unsupported concept?
hey,
Just to be clear, you do not have to use assets. In fact there are some workloads where people want assets disabled entirelyā¦and thatās fineā¦the Sensu agent actually has a tunable to disable asset downloading entirely to support that use case. And really everything that is an asset is just a tarball that you can install and unpack manually as part of CM if you want really tight control over what is install on the system and where.
sensu-install is a thin wrapper over rubygem, so even if it goes a way at some unspecified time in the future youāll still be able to use the gem tool. sensu-install only supports Sensu plugins written in ruby and packaged and published as gems. How long youāll be able to use those gems depends on how much forward momentum contributors have to maintain the plugins you are using. Thereās no effort to to prevent gem installed ruby plugins from working. Some of those gems, especially the handler plugins might need to be updated to work with the Sensu Go event json, but we tried to anticipate that by providing wrapper logic to convert the event json back to what the handler expectsā¦as a forward compatible change in the lastest sensu-plugin gem.
However, much of the newer work is around plugins written in golang and statically cross compiled instead of Ruby as the static Golang binaries work really well for a larger cross section of workloads, including sensu agent sidecar containers. The Ruby stuff, just because it relies on the heavy Ruby environment to be presentā¦doesnāt fit well in the containerized workloads so much. Ironically, the sudo issues arenāt a big deal in that workload generally either because right now the agent is run as root in the official Sensu container right now.
Previously it was possible to rely heavily on Rubyās handling of shared libraries, but assets even though more complicated provide a way to support a wider range of languages. They arenāt just binaries, but can also include librariesā¦ we canāt just dump all of that into the same pathā¦ because different assets may require different compiled versions of the same library.
All of that being said, thereās certainty room to improve assets. I put in this straw proposal to extend assets to optionally allow for predictable binary installs awhile ago. https://github.com/sensu/sensu-go/issues/3046
Thanks for your response. This is clear, but sensu-plugins-ruby - which provides sensu-install - is not available for Debian, which doesnāt make things easier.
Yes, but Debian does not seem to be supported. deb https://packagecloud.io/sensu/community/debian/ %s main, where %s is buster or stretch, returns a 404. Ubuntu does seem to be available.
Hmmā¦ thatās a bug. Let me see if I can get that correctedā¦in the packagecloud setup.
In the meantime you should be able to manually install the debā¦or edit the repo definition on your system. I do that to get yum repo working on fedora.
Also you have the option to install a ruby environment with something like rvm and use ruby gem command to install existing sens-plugins ruby gems. The sensu-plugins-ruby package isnāt strictly needed. The sensu-install executable is a very thin wrapper over ruby gem.
Honestly, even for the rpm based systems Iām using. I donāt bother installing that package, and rely on rvm instead to give me a ruby environment where I canāt use assets.
Thanks for everyoneās replies in this topic. I ended up using runtime assets and preserving the environment by passing PATH when sudoāing. My Sensu Go deployment is readyā¦
