Getting errors related to ruby environment from RHEL6 agents after installing a new agent on RHEL7 host. How is this possible?
Good day smart people!
I’m hoping someone will be able to help untangle this perplexing predicament
What I have:
I’m running Sensu-GO 6.2.7-4449 on a RHEL7 VM, installed via the rpm package. This environment has agents installed on 38 hosts, (also version 6.2.7-4449) which are a mix of RHEL6 and RHEL7 VMs. All agents were also installed from the rpm package. This system has been running nicely with no issues for well over a year. Although I do patch monthly via yum update, so the Sensu version has been incrementing as patches become available from the repo. Anyway….
This afternoon, I installed and started up an additional Sensu agent on a new RHEL7 VM, bringing the total up to 39 agents. After starting up the new agent, I started getting alerts from all RHEL6 agents which run checks via the “sensu/sensu-ruby-runtime” runtime assets such as check-cpu.rb, check-http.rb, …etc.
These ruby checks are now broken on all agents running on RHEL6 hosts. The same check on RHEL7 hosts were not impacted. Other non-ruby checks running on RHEL6 hosts are also working. There are no error messages in the server or agent logs.
I am at a complete lost how adding an additional Sensu agent on a new node could have broken the ruby checks on only the RHEL6 hosts. Let me be clear. The only change was installing an agent on a completely new RHEL7 VM. Other than this new agent starting up and connecting to the backend-url, nothing else was touched.
All failing checks on some RHEL6 hosts are all returning this error
/usr/bin/env: ruby: No such file or directory
Failing checks on other RHEL6 hosts are all returning errors similar to this
/var/cache/sensu/sensu-agent/45ed66c95e71b9f44604a65752262ba8f506c39d8877f032b74ee9b155318f8229de2657ce65785fa1b46024d1b42166769dfe93f61610bf704f8bf5663f1bc8/lib/bundler/setup.rb:2: uninitialized constant RUBY_ENGINE (NameError) from /var/cache/sensu/sensu-agent/45ed66c95e71b9f44604a65752262ba8f506c39d8877f032b74ee9b155318f8229de2657ce65785fa1b46024d1b42166769dfe93f61610bf704f8bf5663f1bc8/bin/check-swap-percent.rb:13:in `require' from /var/cache/sensu/sensu-agent/45ed66c95e71b9f44604a65752262ba8f506c39d8877f032b74ee9b155318f8229de2657ce65785fa1b46024d1b42166769dfe93f61610bf704f8bf5663f1bc8/bin/check-swap-percent.rb:13
I have zero experience with ruby, but these errors suggest something become hosed in the ruby environment. Does anyone have advice on how this could have become broken and what steps might resolve this? Please let me know what additional information might be helpful.