sensu-plugins/sensu-plugins-kubernetes is not working as expected.
<!-- I am trying to add my kubernetes cluster monitoring through Sensu-go, So far I have added assets sensu-plugins/sensu-plugins-kubernetes (https://bonsai.sensu.io/assets/sensu-plugins/sensu-plugins-kubernetes/versions/5.0.2) and sensu/sensu-ruby-runtime (https://bonsai.sensu.io/assets/sensu/sensu-ruby-runtime/versions/0.2.1) -->
#sensuctl asset list
Name URL Hash
───────────────────────────────────────── ───────────────────────────────────────────────────────────────────────────────────────────── ──────────
sensu-plugins/sensu-plugins-kubernetes //assets.bonsai.sensu.io/.../sensu-plugins-kubernetes_5.0.2_debian9_linux_amd64.tar.gz 0766bb5
sensu-plugins/sensu-plugins-kubernetes //assets.bonsai.sensu.io/.../sensu-plugins-kubernetes_5.0.2_debian_linux_amd64.tar.gz 91d971c
sensu-plugins/sensu-plugins-kubernetes //assets.bonsai.sensu.io/.../sensu-plugins-kubernetes_5.0.2_centos7_linux_amd64.tar.gz edc2ff2
sensu-plugins/sensu-plugins-kubernetes //assets.bonsai.sensu.io/.../sensu-plugins-kubernetes_5.0.2_centos6_linux_amd64.tar.gz ed7cf1d
sensu-plugins/sensu-plugins-kubernetes //assets.bonsai.sensu.io/.../sensu-plugins-kubernetes_5.0.2_alpine3.8_linux_amd64.tar.gz 715964e
sensu-plugins/sensu-plugins-kubernetes //assets.bonsai.sensu.io/.../sensu-plugins-kubernetes_5.0.2_alpine_linux_amd64.tar.gz 986248f
sensu/sensu-ruby-runtime //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.2.1_ruby-2.4.4_debian_linux_amd64.tar.gz ff3ba13
sensu/sensu-ruby-runtime //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.2.1_ruby-2.4.4_centos6_linux_amd64.tar.gz 702d49d
sensu/sensu-ruby-runtime //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.2.1_ruby-2.4.4_amzn2_linux_amd64.tar.gz 34d8b1b
sensu/sensu-ruby-runtime //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.2.1_ruby-2.4.4_amzn1_linux_amd64.tar.gz ca25106
sensu/sensu-ruby-runtime //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.2.1_ruby-2.4.4_centos8_linux_amd64.tar.gz 0da51b4
sensu/sensu-ruby-runtime //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.2.1_ruby-2.4.4_centos7_linux_amd64.tar.gz 9204c46
sensu/sensu-ruby-runtime //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.2.1_ruby-2.4.4_alpine_linux_amd64.tar.gz eea1c2e
I was getting below error when setting up the alert.
/var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require': Error loading shared library libffi.so.6: No such file or directory (needed by /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/lib/ruby/2.4.0/gems/ffi-1.13.1/lib/ffi_c.so) - /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/lib/ruby/2.4.0/gems/ffi-1.13.1/lib/ffi_c.so (LoadError)
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/lib/ruby/2.4.0/gems/ffi-1.13.1/lib/ffi.rb:6:in `rescue in <top (required)>'
from /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/lib/ruby/2.4.0/gems/ffi-1.13.1/lib/ffi.rb:3:in `<top (required)>'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/lib/ruby/2.4.0/gems/http-parser-1.2.1/lib/http-parser.rb:3:in `<top (required)>'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/lib/ruby/2.4.0/gems/http-4.4.1/lib/http/response/parser.rb:3:in `<top (required)>'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/lib/ruby/2.4.0/gems/http-4.4.1/lib/http/connection.rb:6:in `<top (required)>'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/lib/ruby/2.4.0/gems/http-4.4.1/lib/http/client.rb:9:in `<top (required)>'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/lib/ruby/2.4.0/gems/http-4.4.1/lib/http.rb:8:in `<top (required)>'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/lib/ruby/2.4.0/gems/kubeclient-4.9.1/lib/kubeclient/watch_stream.rb:2:in `<top (required)>'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/lib/ruby/2.4.0/gems/kubeclient-4.9.1/lib/kubeclient.rb:16:in `<top (required)>'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/lib/ruby/2.4.0/bundler/gems/sensu-plugins-kubernetes-a82804509e76/lib/sensu-plugins-kubernetes/client.rb:3:in `<top (required)>'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/lib/ruby/2.4.0/bundler/gems/sensu-plugins-kubernetes-a82804509e76/lib/sensu-plugins-kubernetes/cli.rb:4:in `<top (required)>'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/lib/ruby/2.4.0/bundler/gems/sensu-plugins-kubernetes-a82804509e76/bin/check-kube-pods-running.rb:51:in `<top (required)>'
from /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/bin/check-kube-pods-running.rb:14:in `load'
from /var/cache/sensu/sensu-agent/715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd/bin/check-kube-pods-running.rb:14:in `<main>'
{
"type": "CheckConfig",
"api_version": "core/v2",
"metadata": {
"name": "dnv.test.k8s.pod.running",
"namespace": "dnv.test",
"created_by": "admin"
},
"spec": {
"check_hooks": null,
"command": "check-kube-pods-running.rb --time 60 --in-cluster",
"env_vars": null,
"handlers": [
"slack-warning"
],
"high_flap_threshold": 0,
"interval": 60,
"low_flap_threshold": 0,
"output_metric_format": "",
"output_metric_handlers": null,
"pipelines": [],
"proxy_entity_name": "",
"publish": true,
"round_robin": false,
"runtime_assets": [
"sensu-plugins/sensu-plugins-kubernetes",
"sensu/sensu-ruby-runtime"
],
"secrets": null,
"stdin": false,
"subdue": null,
"subscriptions": [
"entity:aks-nodepool2-12345678-vmss000002"
],
"timeout": 0,
"ttl": 0
}
}
Description of steps I’ve taken to attempt to solve the issue.
<!-- Basically sensu-plugins/sensu-plugins-kubernetes is looking for libffi.so.6 file from sensu/sensu-ruby-runtime but instead libffi.so.6 , it has libffi.so.7 and due to that checks is failing, to cross check, I exec in to sensu pod and copy libffi.so.7 as libffi.so.6 under container path "/var/cache/sensu/sensu-agent/eea1c2e*" and it's started working -->
# kubectl exec -it sensu-agent-bhmgd /bin/sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ # cd /var/cache/sensu/sensu-agent/
/var/cache/sensu/sensu-agent # ls
715964e8e927f608a605ac3d6e45721ba706bbb2c7f222de8bdf0a1bba0b8aa6c50ccd1e7ba07ed68c96ab83961b8238de6d8e0ad836e43a71008fda240f8bcd
assets.db
eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df
fa25e317ba8aa3e23d9e3d3c54081ca4529aeba2e7ee437a2bf1ed8cfb3fbf70cd0380eb8cc427a9be17515e437f6346949381cc7ce47f37493e32eb8372ded8
queue.db
/var/cache/sensu/sensu-agent # cd eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/
/var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df # ls
bin include lib share
/var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df # cd lib/
/var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib # ls
ld-musl-x86_64.so.1 libcrypto.so.1.1 libgdbm.so.6 libm.so.6 libreadline.so.8 libssl.so.1.1 libyaml-0.so.2
libc.so.6 libffi.so.7 libgdbm_compat.so.4 libncursesw.so.6 librt.so.1 libstdc++.so.6 libz.so.1
libcrypt.so.1 libgcc_s.so.1 libgmp.so.10 libpthread.so.0 libruby-static.a libutil.so.1 ruby
/var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib # cp libffi.so.7 libffi.so.6
/var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib # ls -ltrh
total 52M
drwxr-xr-x 6 root root 4.0K Feb 1 09:44 ruby
-rwxr-xr-x 1 root root 97.8K Feb 1 09:44 libz.so.1
-rwxr-xr-x 1 root root 105.7K Feb 1 09:44 libyaml-0.so.2
-rwxr-xr-x 1 root root 590.6K Feb 1 09:44 libutil.so.1
-rwxr-xr-x 1 root root 1.6M Feb 1 09:44 libstdc++.so.6
-rwxr-xr-x 1 root root 511.5K Feb 1 09:44 libssl.so.1.1
-rw-r--r-- 1 root root 41.6M Feb 1 09:44 libruby-static.a
-rwxr-xr-x 1 root root 590.6K Feb 1 09:44 librt.so.1
-rwxr-xr-x 1 root root 286.7K Feb 1 09:44 libreadline.so.8
-rwxr-xr-x 1 root root 590.6K Feb 1 09:44 libpthread.so.0
-rwxr-xr-x 1 root root 363.0K Feb 1 09:44 libncursesw.so.6
-rwxr-xr-x 1 root root 590.6K Feb 1 09:44 libm.so.6
-rwxr-xr-x 1 root root 405.7K Feb 1 09:44 libgmp.so.10
-rwxr-xr-x 1 root root 13.6K Feb 1 09:44 libgdbm_compat.so.4
-rwxr-xr-x 1 root root 53.8K Feb 1 09:44 libgdbm.so.6
-rw-r--r-- 1 root root 97.8K Feb 1 09:44 libgcc_s.so.1
-rwxr-xr-x 1 root root 38.0K Feb 1 09:44 libffi.so.7
-rwxr-xr-x 1 root root 2.5M Feb 1 09:44 libcrypto.so.1.1
-rwxr-xr-x 1 root root 590.6K Feb 1 09:44 libcrypt.so.1
-rwxr-xr-x 1 root root 590.6K Feb 1 09:44 libc.so.6
-rwxr-xr-x 1 root root 590.6K Feb 1 09:44 ld-musl-x86_64.so.1
-rwxr-xr-x 1 root root 38.0K Feb 1 09:45 libffi.so.6
/var/cache/sensu/sensu-agent/eea1c2eb928e59d94f39e8c2c90315ca6145590462f2bab60f2d7798f15ec0caaf0310047aeba478901651a78fce9b23c848a016cfa133f43aa99bc880d4d4df/lib #
Now I am looking for solution to either get libffi.so.6 in to sensu/sensu-ruby-runtime it self or sensu-plugins/sensu-plugins-kubernetes can use libffi.so.7 which is available under sensu/sensu-ruby-runtime.
Please do suggest. I am very new in to Sensu and need your help to get this fixed.
Operating system information
<!--Please select the operating system that you're using Sensu Go on-->
- Ubuntu