Embedded vs System Ruby runtime with v0.17.0


#1

Hi,

Going a bit nuts today as older installation done with v0.16.0 is working.

I just did a fresh installation of Sensu 0.17.0, with basic system check (like from community plugins check-cpu,…) on ubuntu (using omnibus package). I’m not using the embedded ruby runtime but the system one (ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]) so I can deploy easily additional gems like pg, twilio… I installed the sensu-plugin gem in this case. If i try a check via command line - it works. As soon as I define it as a check on a given client - i keep on getting the error below output:

{“timestamp”:“2015-03-19T13:21:51.508363+0000”,“level”:“info”,“message”:“publishing check result”,“payload”:{“client”:“msi-cloudsql-test-sgi2”,“check”:{“name”:“system-load_metrics”,“issued”:1426771311,“command”:"/etc/sensu/plugins/systems/load-metrics.rb",“executed”:1426771311,“duration”:0.055,“output”:"/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require’: cannot load such file – sensu-plugin/metric/cli

(LoadError)\n\tfrom /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'\n\tfrom /etc/sensu/plugins/systems/load-metrics.rb:41:in'\n",“status”:1}}}

The crazy part - these errors only occur on sensu client (agent) but work on the sensu server itself (client part).

Same set up / configuration with the previous release v0.16.0. Anything change that could explain the issue ? As indicated it’s not an issue with the system ruby not finding the gem sensu-plugin - as the command run successfully directly on command line.

Thanks for any pointer.

Cheers,

Sylvain

Ps;

gem list returns

*** LOCAL GEMS ***

json (1.8.2)

mixlib-cli (1.5.0)

pg (0.18.1)

sensu-plugin (1.1.0)


#2

I would note that you say you have 1.9.3 installed on the server, but your error message seems to indicate that you have 1.9.1 installed on the client. Probably not a big deal, but that strikes me as a configuration you’d probably want to avoid in general.

···

On Thu, Mar 19, 2015 at 1:00 PM, Sylvain Gibier sylvain@munichconsulting.de wrote:

Hi,

Going a bit nuts today as older installation done with v0.16.0 is working.

I just did a fresh installation of Sensu 0.17.0, with basic system check (like from community plugins check-cpu,…) on ubuntu (using omnibus package). I’m not using the embedded ruby runtime but the system one (ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]) so I can deploy easily additional gems like pg, twilio… I installed the sensu-plugin gem in this case. If i try a check via command line - it works. As soon as I define it as a check on a given client - i keep on getting the error below output:

{“timestamp”:“2015-03-19T13:21:51.508363+0000”,“level”:“info”,“message”:“publishing check result”,“payload”:{“client”:“msi-cloudsql-test-sgi2”,“check”:{“name”:“system-load_metrics”,“issued”:1426771311,“command”:"/etc/sensu/plugins/systems/load-metrics.rb",“executed”:1426771311,“duration”:0.055,“output”:"/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require’: cannot load such file – sensu-plugin/metric/cli

(LoadError)\n\tfrom /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'\n\tfrom /etc/sensu/plugins/systems/load-metrics.rb:41:in'\n",“status”:1}}}

The crazy part - these errors only occur on sensu client (agent) but work on the sensu server itself (client part).

Same set up / configuration with the previous release v0.16.0. Anything change that could explain the issue ? As indicated it’s not an issue with the system ruby not finding the gem sensu-plugin - as the command run successfully directly on command line.

Thanks for any pointer.

Cheers,

Sylvain

Ps;

gem list returns

*** LOCAL GEMS ***

json (1.8.2)

mixlib-cli (1.5.0)

pg (0.18.1)

sensu-plugin (1.1.0)


#3

Hi Glenn -

Yeah - it can be confusing but on ubuntu trusty - even if ruby 1.9.3 is installed, the actual path is under 1.9.1. I fixed the issue by overriding the USER=root in the /etc/default/sensu configuration file. It seems somehow even if gem where installed globally - they were not picked up by the default sensu user.

Cheers,

s.

···

On Thursday, March 19, 2015 at 9:55:04 PM UTC+1, Glenn McAllister wrote:

I would note that you say you have 1.9.3 installed on the server, but your error message seems to indicate that you have 1.9.1 installed on the client. Probably not a big deal, but that strikes me as a configuration you’d probably want to avoid in general.

On Thu, Mar 19, 2015 at 1:00 PM, Sylvain Gibier syl...@munichconsulting.de wrote:

Hi,

Going a bit nuts today as older installation done with v0.16.0 is working.

I just did a fresh installation of Sensu 0.17.0, with basic system check (like from community plugins check-cpu,…) on ubuntu (using omnibus package). I’m not using the embedded ruby runtime but the system one (ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]) so I can deploy easily additional gems like pg, twilio… I installed the sensu-plugin gem in this case. If i try a check via command line - it works. As soon as I define it as a check on a given client - i keep on getting the error below output:

{“timestamp”:“2015-03-19T13:21:51.508363+0000”,“level”:“info”,“message”:“publishing check result”,“payload”:{“client”:“msi-cloudsql-test-sgi2”,“check”:{“name”:“system-load_metrics”,“issued”:1426771311,“command”:"/etc/sensu/plugins/systems/load-metrics.rb",“executed”:1426771311,“duration”:0.055,“output”:"/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require’: cannot load such file – sensu-plugin/metric/cli

(LoadError)\n\tfrom /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'\n\tfrom /etc/sensu/plugins/systems/load-metrics.rb:41:in'\n",“status”:1}}}

The crazy part - these errors only occur on sensu client (agent) but work on the sensu server itself (client part).

Same set up / configuration with the previous release v0.16.0. Anything change that could explain the issue ? As indicated it’s not an issue with the system ruby not finding the gem sensu-plugin - as the command run successfully directly on command line.

Thanks for any pointer.

Cheers,

Sylvain

Ps;

gem list returns

*** LOCAL GEMS ***

json (1.8.2)

mixlib-cli (1.5.0)

pg (0.18.1)

sensu-plugin (1.1.0)


#4

the version 1.9.x don’t have more support…

Please use ruby 2.1 or 2.2 in sensu 0.17.

tks

···

Em sexta-feira, 20 de março de 2015 14:53:54 UTC-3, Sylvain Gibier escreveu:

Hi Glenn -

Yeah - it can be confusing but on ubuntu trusty - even if ruby 1.9.3 is installed, the actual path is under 1.9.1. I fixed the issue by overriding the USER=root in the /etc/default/sensu configuration file. It seems somehow even if gem where installed globally - they were not picked up by the default sensu user.

Cheers,

s.

On Thursday, March 19, 2015 at 9:55:04 PM UTC+1, Glenn McAllister wrote:

I would note that you say you have 1.9.3 installed on the server, but your error message seems to indicate that you have 1.9.1 installed on the client. Probably not a big deal, but that strikes me as a configuration you’d probably want to avoid in general.

On Thu, Mar 19, 2015 at 1:00 PM, Sylvain Gibier syl...@munichconsulting.de wrote:

Hi,

Going a bit nuts today as older installation done with v0.16.0 is working.

I just did a fresh installation of Sensu 0.17.0, with basic system check (like from community plugins check-cpu,…) on ubuntu (using omnibus package). I’m not using the embedded ruby runtime but the system one (ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]) so I can deploy easily additional gems like pg, twilio… I installed the sensu-plugin gem in this case. If i try a check via command line - it works. As soon as I define it as a check on a given client - i keep on getting the error below output:

{“timestamp”:“2015-03-19T13:21:51.508363+0000”,“level”:“info”,“message”:“publishing check result”,“payload”:{“client”:“msi-cloudsql-test-sgi2”,“check”:{“name”:“system-load_metrics”,“issued”:1426771311,“command”:"/etc/sensu/plugins/systems/load-metrics.rb",“executed”:1426771311,“duration”:0.055,“output”:"/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require’: cannot load such file – sensu-plugin/metric/cli

(LoadError)\n\tfrom /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'\n\tfrom /etc/sensu/plugins/systems/load-metrics.rb:41:in'\n",“status”:1}}}

The crazy part - these errors only occur on sensu client (agent) but work on the sensu server itself (client part).

Same set up / configuration with the previous release v0.16.0. Anything change that could explain the issue ? As indicated it’s not an issue with the system ruby not finding the gem sensu-plugin - as the command run successfully directly on command line.

Thanks for any pointer.

Cheers,

Sylvain

Ps;

gem list returns

*** LOCAL GEMS ***

json (1.8.2)

mixlib-cli (1.5.0)

pg (0.18.1)

sensu-plugin (1.1.0)


#5

I've read your email, but I want to just confirm what you said that:
* System ruby 1.9.1 (or 1.9.3?) is installed on the *client* (where
sensu-agent runs)
* The sensu-plugin gem is installed using the *gem* binary of that
same ruby? (/usr/bin/gem versus /usr/bin/gem1.9.1?)
* Double check that EMBEDDED_RUBY=false on the *client*
* And what is the output of gem1.9.1 list on the client?

Remember that all of these check-command are executed the on the
client. The fact that they work on the server doesn't actually matter
(unless the server is also a client. Also handlers of course)

To me, it does look like you are using the system ruby, but
sensu-plugin is not installed, as you said. I bet it was installed
either on the wrong system or with the wrong gem binary.

···

On Thu, Mar 19, 2015 at 10:00 AM, Sylvain Gibier <sylvain@munichconsulting.de> wrote:

Hi,

Going a bit nuts today as older installation done with v0.16.0 is working.

I just did a fresh installation of Sensu 0.17.0, with basic system check
(like from community plugins check-cpu,...) on ubuntu (using omnibus
package). I'm not using the embedded ruby runtime but the system one (ruby
1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]) so I can deploy easily
additional gems like pg, twilio... I installed the sensu-plugin gem in this
case. If i try a check via command line - it works. As soon as I define it
as a check on a given client - i keep on getting the error below output:

{"timestamp":"2015-03-19T13:21:51.508363+0000","level":"info","message":"publishing
check
result","payload":{"client":"msi-cloudsql-test-sgi2","check":{"name":"system-load_metrics","issued":1426771311,"command":"/etc/sensu/plugins/systems/load-metrics.rb","executed":1426771311,"duration":0.055,"output":"/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in
`require': cannot load such file -- sensu-plugin/metric/cli
(LoadError)\n\tfrom /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in
`require'\n\tfrom /etc/sensu/plugins/systems/load-metrics.rb:41:in
`<main>'\n","status":1}}}

The crazy part - these errors only occur on sensu client (agent) but work on
the sensu server itself (client part).

Same set up / configuration with the previous release v0.16.0. Anything
change that could explain the issue ? As indicated it's not an issue with
the system ruby not finding the gem sensu-plugin - as the command run
successfully directly on command line.

Thanks for any pointer.

Cheers,

Sylvain

Ps;

gem list returns

*** LOCAL GEMS ***

json (1.8.2)
mixlib-cli (1.5.0)
pg (0.18.1)
sensu-plugin (1.1.0)


#6

Hi Kyle,

I’m using ubuntu trusty - with default ruby runtime installed on it via apt-get. I’m running on AWS where I actually created a default AMI for all my servers where i have at the sensu agent installed + all required gems required for my services (pg, … including sensu-plugin). So I’m 100% of using the system ruby (EMBEDDED_RUBY=false) as same AMI used on the different instances…

A week ago - before sensu v1.7.0 was released, I had no issues of running my checks. Few days back I had to set up a new environment - and just wanted to update my custom AMI with additional plugins - that when trying to run the same sensu checks - it was failing with the error of not finding the sensu-plugin. I can’t explain why - but it seems that with the new version of sensu - gems installed against the system ruby - they were not ‘reachable’ by the sensu user installed via the omnibus package.

Overriding the default user to be ROOT - where i could via shell validating that the different checks were running … - did allow the sensu-client runtime finding the custom gems.

I will revisit the user privilege issue once I re-package the latest sensu into my own bosh release for my cloudfoundry installation next week.

Cheers,

sylvain

···

On Sat, Mar 21, 2015 at 3:23 PM, Kyle Anderson kyle@xkyle.com wrote:

I’ve read your email, but I want to just confirm what you said that:

  • System ruby 1.9.1 (or 1.9.3?) is installed on the client (where

sensu-agent runs)

  • The sensu-plugin gem is installed using the gem binary of that

same ruby? (/usr/bin/gem versus /usr/bin/gem1.9.1?)

  • Double check that EMBEDDED_RUBY=false on the client

  • And what is the output of gem1.9.1 list on the client?

Remember that all of these check-command are executed the on the

client. The fact that they work on the server doesn’t actually matter

(unless the server is also a client. Also handlers of course)

To me, it does look like you are using the system ruby, but

sensu-plugin is not installed, as you said. I bet it was installed

either on the wrong system or with the wrong gem binary.

On Thu, Mar 19, 2015 at 10:00 AM, Sylvain Gibier

sylvain@munichconsulting.de wrote:

Hi,

Going a bit nuts today as older installation done with v0.16.0 is working.

I just did a fresh installation of Sensu 0.17.0, with basic system check

(like from community plugins check-cpu,…) on ubuntu (using omnibus

package). I’m not using the embedded ruby runtime but the system one (ruby

1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]) so I can deploy easily

additional gems like pg, twilio… I installed the sensu-plugin gem in this

case. If i try a check via command line - it works. As soon as I define it

as a check on a given client - i keep on getting the error below output:

{“timestamp”:“2015-03-19T13:21:51.508363+0000”,“level”:“info”,“message”:"publishing

check

result",“payload”:{“client”:“msi-cloudsql-test-sgi2”,“check”:{“name”:“system-load_metrics”,“issued”:1426771311,“command”:"/etc/sensu/plugins/systems/load-metrics.rb",“executed”:1426771311,“duration”:0.055,“output”:"/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in

`require’: cannot load such file – sensu-plugin/metric/cli

(LoadError)\n\tfrom /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in

`require’\n\tfrom /etc/sensu/plugins/systems/load-metrics.rb:41:in

`’\n",“status”:1}}}

The crazy part - these errors only occur on sensu client (agent) but work on

the sensu server itself (client part).

Same set up / configuration with the previous release v0.16.0. Anything

change that could explain the issue ? As indicated it’s not an issue with

the system ruby not finding the gem sensu-plugin - as the command run

successfully directly on command line.

Thanks for any pointer.

Cheers,

Sylvain

Ps;

gem list returns

*** LOCAL GEMS ***

json (1.8.2)

mixlib-cli (1.5.0)

pg (0.18.1)

sensu-plugin (1.1.0)