Issues with ruby checks sensu-plugin/metric/cli (LoadError)

Sensu Go
1

I’m transitioning a legacy Sensu implementation to Sensu-go and have been making good progress. I have a number of checks successfully processing, but I’ve been running into load errors with some of my checks.

Let’s follow my process with the sensu-plugins syslog check. The original repo is not set up as an asset, so I forked it and installed the necessary gems, placing them in the lib folder under the ruby version. I then packaged this up, created a github release, and created a private Bonsai asset.

I placed the information for this asset in my configuration management as my other working checks, and have the check generating the following error:

/var/cache/sensu/sensu-agent/a28952fd93fc63db1f8988c7bc40b0ad815eb9f35ef7317d6caf5d77ecfbfd824a9db54184400aa0c81c29b34cb48c7e8c6e3f17891aaf84cafa3c134266a61a/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- sensu-plugin/metric/cli (LoadError) from /var/cache/sensu/sensu-agent/a28952fd93fc63db1f8988c7bc40b0ad815eb9f35ef7317d6caf5d77ecfbfd824a9db54184400aa0c81c29b34cb48c7e8c6e3f17891aaf84cafa3c134266a61a/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require' from /var/cache/sensu/sensu-agent/20ce4f19e1deafbec834e41bb8a4f613d0abd1f2c12fb242936ed7a620e19f9779e0782958763d608878da237d59b28c3c986846ff5a9be76565cf7e84da3b11/bin/metrics-syslog-ng.rb:17:in main'

  • sensu-agent version 5.17.1, build 50b2c99e444ff3a0a3b2a6fbd58de988bccbbb9a, built 2020-01-31T19:27:40Z

  • sensu-backend version 5.17.1, build 50b2c99e444ff3a0a3b2a6fbd58de988bccbbb9a, built 2020-01-31T19:27:14Z

I verified the sensu-plugin gem was present within the check asset. One thing I have noticed is the difference in ruby version between the above error message and the runtime asset.

I then decompiled the ruby runtime asset and repackaged it with every gem my checks may need, but it’s now complaining /usr/bin/env ruby is not present.

Something about my understanding of the way checks are supposed to execute must be fundamentally flawed. How can I resolve this?

2

Can you post your check definition (sensuctl check info <check-name> --format yaml)?

3

@Capt-Tom just an FYI here, Ruby plugins aren’t the most straightforward things to convert to assets. We have a guide (Contributing Assets for Existing Ruby Sensu Plugins) that walks through this process, as just dropping the gems off in /lib won’t yield a successfully packaged asset. That guide should walk you through successfully packing that repo up as an asset.

4

I rebuilt my testing environment, as apparently if I leave it overnight a token expires and I’m not sure how to refresh it.

vagrant@base-server-ubuntu-1604:~$ sensuctl check info check-syslog --format yaml
Error: GET "/api/core/v2/namespaces/default/checks/check-syslog": failed to update configuration with new refresh token (open /home/vagrant/.config/sensu/sensuctl/cluster: permission denied)


vagrant@base-server-ubuntu-1604:~$ sensuctl check info check-syslog --format yaml
type: CheckConfig
api_version: core/v2
metadata:
  name: check-syslog
  namespace: default
spec:
  check_hooks: null
  command: metrics-syslog-ng.rb
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 60
  low_flap_threshold: 0
  output_metric_format: ""
  output_metric_handlers: null
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - sensu-ruby-runtime
  - sensu-plugins-syslog-ng
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - system
  timeout: 0
  ttl: 500
5

Thanks for the link, I haven’t come across that guide in researching this.

I’ll pick this apart and see where it gets me. This is a fairly involved process for what’s essentially running a script.

That guide is also mostly just paperwork associated with making a change. It would be very helpful to get a more in depth technical explanation of how runtime assets are loaded into the environment used for evaluating these checks and how their components interact.

For example, from a very high level it makes perfect sense that a ruby check should be packaged as an asset, and then combined with a ruby runtime that will help when the check is called. However that leaves a lot unsaid; what’s the order these assets are loaded in? Is there any way of increasing the verbosity of the process? How exactly does Sensu use this runtime to create an environment within which it can provide everything necessary for the check to execute successfully?

Thank you very much for your response and kind direction.

1 Like
6

@Capt-Tom great questions! And thanks for your contributions!

The Sensu Assets documentation helps explain how assets work under the hood, but it’s essentially management of the Sensu execution environment (literally managing environment variables to dynamically set $PATH, $LD_LIBRARY_PATH, and $CPATH).

https://docs.sensu.io/sensu-go/latest/reference/assets/#asset-format-specification

At runtime, the Sensu services (e.g. the agent or backend) fetch assets, verify their integrity using the provided sha512 value, then unpack the assets into the service cache directory (e.g. /var/cache/sensu/sensu-agent/<sha512>/). Once unpacked, the Sensu service modifies $PATH to include the corresponding cache directories for the corresponding asset.

Asset load order doesn’t matter, thanks to how $PATH resolution works. If we add a ruby runtime to the $PATH and an asset containing a Ruby script with a “shebang” like #!/usr/bin/env ruby – well, you can probably guess what happens at this point – “it just works” because our ruby runtime has been added to $PATH.

My personal favorite thing about the implementation is that it’s lightweight (based on GNU tar and *nix environment principles), but also flexible enough to package plugins which require dynamically linked libraries (which are looked up using $LD_LIBRARY_PATH). So even old Nagios C plugins can become portable using Sensu Assets.

I hope this helps!

7

Many thanks for your reply Caleb. I’d poked around the cache directories as I’ve seen them referenced in the traces I was receiving, but your pathing descriptions are helpful.

I was able to unpack the build process Aaron mentioned and modify and build it locally to produce an asset that’s allowing all my product-specific checks to execute correctly.

Happy trails.

1 Like