Getting errors related to ruby environment from RHEL6 agents after installing a new agent on RHEL7 host. How is this possible?

Sensu Go
1

Getting errors related to ruby environment from RHEL6 agents after installing a new agent on RHEL7 host. How is this possible?

Good day smart people!
I’m hoping someone will be able to help untangle this perplexing predicament

What I have:
I’m running Sensu-GO 6.2.7-4449 on a RHEL7 VM, installed via the rpm package. This environment has agents installed on 38 hosts, (also version 6.2.7-4449) which are a mix of RHEL6 and RHEL7 VMs. All agents were also installed from the rpm package. This system has been running nicely with no issues for well over a year. Although I do patch monthly via yum update, so the Sensu version has been incrementing as patches become available from the repo. Anyway….

What happened:
This afternoon, I installed and started up an additional Sensu agent on a new RHEL7 VM, bringing the total up to 39 agents. After starting up the new agent, I started getting alerts from all RHEL6 agents which run checks via the “sensu/sensu-ruby-runtime” runtime assets such as check-cpu.rb, check-http.rb, …etc.

These ruby checks are now broken on all agents running on RHEL6 hosts. The same check on RHEL7 hosts were not impacted. Other non-ruby checks running on RHEL6 hosts are also working. There are no error messages in the server or agent logs.

I am at a complete lost how adding an additional Sensu agent on a new node could have broken the ruby checks on only the RHEL6 hosts. Let me be clear. The only change was installing an agent on a completely new RHEL7 VM. Other than this new agent starting up and connecting to the backend-url, nothing else was touched.

More weirdness:

All failing checks on some RHEL6 hosts are all returning this error

/usr/bin/env: ruby: No such file or directory

Failing checks on other RHEL6 hosts are all returning errors similar to this

/var/cache/sensu/sensu-agent/45ed66c95e71b9f44604a65752262ba8f506c39d8877f032b74ee9b155318f8229de2657ce65785fa1b46024d1b42166769dfe93f61610bf704f8bf5663f1bc8/lib/bundler/setup.rb:2: uninitialized constant RUBY_ENGINE (NameError)
	from /var/cache/sensu/sensu-agent/45ed66c95e71b9f44604a65752262ba8f506c39d8877f032b74ee9b155318f8229de2657ce65785fa1b46024d1b42166769dfe93f61610bf704f8bf5663f1bc8/bin/check-swap-percent.rb:13:in `require'
	from /var/cache/sensu/sensu-agent/45ed66c95e71b9f44604a65752262ba8f506c39d8877f032b74ee9b155318f8229de2657ce65785fa1b46024d1b42166769dfe93f61610bf704f8bf5663f1bc8/bin/check-swap-percent.rb:13

I have zero experience with ruby, but these errors suggest something become hosed in the ruby environment. Does anyone have advice on how this could have become broken and what steps might resolve this? Please let me know what additional information might be helpful.

Thank you
Tim

2

Hello,
latest version of the ruby runtime asset does not have a centos6 build, as centos6 is has now reached eol.

Most likely you’ve updated to the lastest ruby-runtime-asset version in your asset resources.

3

Thanks for your reply. Help me understand the versioning issue. Should I reinstall the agents on the RHEL6 hosts with a lesser version? The docs suggest that v6.2 is supported on Centos6: Supported platforms and distributions - Sensu Docs

Thanks

4

its not the sensu agents… its the ruby runtime asset.

What you probably want to do is use sensuctl asset add with addtional options so that you can install a previous ruby-runtime-version with centos6 build using a versioned name using the renaming option. Then make sure the checks running on the rhel6-alike hosts reference the correct asset name.
Example:

sensuctl asset add sensu/sensu-ruby-runtime:0.0.11 -r "sensu/sensu-ruby-runtime:0.0.11"

Then you just have to make sure the checks that need to run on centos6 have “sensu/sensu-ruby-runtime:0.0.11” as a runtime asset.

You should also look into transitioning from the ruby plugins to the golang based assets. Because the golang can be statically compiled, they are much easier to use as runtime assets.
For example, this golang based plugin may provide you with equivalent swap check information without the need for the ruby-runtime.
https://bonsai.sensu.io/assets/sensu/check-memory-usage

hope this helps

5

Ok, I think you’ve pointed me in the right direction. I really appreciate the help. Thank you