Asset usage GLIBC error

Sensu Go
1

Hello,

I got an issue with the asset mecanism, i got a sensu docker stack with persistant etcd storage, a nginx front to secure thes acces between agents end server, everything worked, i can add check, assets, handlers, remote servers, but when i tried to use a checks-cpu or another, i got this message :

ruby: relocation error: /var/cache/sensu/sensu-agent/8b5191be68c09fd76533a67857bc52152c8660814c113fa0137fc5c7e104d096656b675a65e330e82a88b43368f9f63fae2735e79e1e54f828b03f1520afe16b/lib/libpthread.so.0: symbol __libc_dl_error_tsd version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

My check go a reference on my asset, the asset is in the on the server (agent) 's cache directory , i use the official runtime from bonsai, did i miss something ?

Best regards

Franck

2

Hello,

I’ll need more info in order to figure out if this was a configuration error or a bug in the asset itself.

  1. what docker container are use using to run the Sensu agent? The official sensu/sensu container?
  2. which ruby runtime version were you using: version, platform and arch
    If you provide the url used in your asset definition that should provide all the information
  3. Which asset binary were you using: version platform and arch
    If you provide the url used in your asset definition that should provide all the information
3

Hey!
We did fix a bug recently in the alpine platform binary. Can you please try the 0.0.7 version and see if that helps.

1 Like
4

Hi,
I am also facing the same issue, though I am using the updated asset definition i.e. 0.0.8. Here’s what the error looks like for disk plugin on asset-
ruby: relocation error: /var/cache/sensu/sensu-agent/44507f99f316c969b8e02a2194aeec5d0ff6d6f9a5ad656b7e0af1339f4aed95ec333577a8880ffab6b8c494475a6154694dbb3129ef73baf1b4f1f3cf1f60dd/lib/libpthread.so.0: symbol __libc_dl_error_tsd, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

And if I use any other version on it then it will give errors like these -
/usr/bin/env: ruby: No such file or directory

or
/usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in 'require': cannot load such file -- sensu-plugin/check/cli (LoadError) from /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in 'require' from /var/cache/sensu/sensu-agent/0ce9d52b270b77f4cab754e55732ae002228201d0bd01a89b954a0655b88c1ee6546e2f82cfd1eec04689af90ad940cde128e8867912d9e415f4a58d7fdcdadf/lib/ruby/2.4.0/bundler/gems/sensu-plugins-disk-checks-73a6f8b6f566/bin/check-disk-usage.rb:28:in '<top (required)>' from /var/cache/sensu/sensu-agent/0ce9d52b270b77f4cab754e55732ae002228201d0bd01a89b954a0655b88c1ee6546e2f82cfd1eec04689af90ad940cde128e8867912d9e415f4a58d7fdcdadf/bin/check-disk-usage.rb:14:in 'load' from /var/cache/sensu/sensu-agent/0ce9d52b270b77f4cab754e55732ae002228201d0bd01a89b954a0655b88c1ee6546e2f82cfd1eec04689af90ad940cde128e8867912d9e415f4a58d7fdcdadf/bin/check-disk-usage.rb:14:in <main>'

Here is a new error that I got just recently -

error getting assets for event: TypeError: Cannot access member 'split' of undefined
Here’s the ruby runtime asset definition -

---
type: Asset
api_version: core/v2
metadata:
  name: sensu-ruby-runtime
  namespace: SafeGuard
  labels:
  annotations:
    sensio.io.bonsai.url: https://assets.bonsai.sensu.io/03d08cdfc649500b7e8cd1708bb9bb93d91fea9e/sensu-ruby-runtime_0.0.8_ruby-2.4.4_centos7_linux_amd64.tar.gz
    sensio.io.bonsai.tier: Community
    sensio.io.bonsai.version: 0.0.8
    sensio.io.bonsai.tags:
spec:
  url: https://assets.bonsai.sensu.io/03d08cdfc649500b7e8cd1708bb9bb93d91fea9e/sensu-ruby-runtime_0.0.8_ruby-2.4.4_centos7_linux_amd64.tar.gz
  sha512: 8d6e7ef9bab057493ab25c4d5b9d68d324600b032b467e09cc26450141c1d4868fcd74704b7fc2c4523c23021c80291ee53f95f139d11a03b9fbf506d3455fbe
  filters:
    - entity.system.os == 'linux'
    - entity.system.arch == 'amd64'
    - entity.system.platform_family == 'rhel'
    - entity.system.platform_version.split('.')[0] == '7'

Can you please help in this matter as this is a blocker for our project?

Regards,
Rohan

5

Hey,
Just to be clear is this CentOS 7 install you are trying to use the asset on?

I’m working on a pre-release to try to clean up the libthread error now.

The underlying problem is, the set of shared libraries included in the runtime asset is over zealous and is including several libraries provided by glibc as ruby has to link against them. When in fact all glibc provided libraries should be considered part of the OS. The error crops up only in situations where the OS glibc doesn’t match the glibc libraries in the asset itself and doesn’t crop up in initial testing because the libraries on the host and in the asset match in the test environment.

Real quick hack, if you can’t wait for an official fix… you should be able to remove the offending glibc provided libraries from the asset tarball you are using right now and serve the asset locally instead of using the asset provided at the bonsai url.

The other error you are hitting has to do with the new filter rule, meant to make sure the asset only runs on rhel family with version 7. If the rule is misfiring for you, you can remove the filter statements in the config.

Can you provide the detailed entity system information as seen by sensu?

sensuctl entity info --format yaml

I’m interested specifically in the spec system attribute section. You may want to redact the network section, as it may contain ip address information you don’t need to share.

6

So I just tested the 0.0.8 release of ruby runtime on a CentOS 7 install, and I’m not seeing any errors with the sensu-ruby-runtime.

Are you using another asset as well as the ruby-runtime-asset? The relocation error you pasted is referencing a directory not associated with the sensu-ruby-runtime 0.0.8 release, so that makes me think the problem is with a different asset, whatever asset has the sha512sum starting with “44507f”.

And here is where things get a little complicated…
because of the way LD_LIBRARY_PATH based runtime overrides work… your check is seeing multiple library paths, one for each asset configured to be used with the check.
The offending libthread library could actually be in the directory:
/var/cache/sensu/sensu-agent/44507f99f316c969b8e02a2194aeec5d0ff6d6f9a5ad656b7e0af1339f4aed95ec333577a8880ffab6b8c494475a6154694dbb3129ef73baf1b4f1f3cf1f60dd/lib/

which is associated with a different asset. I don’t know which asset that is. The directory matches the sha512sum in the asset definition. so you should be able to determine which asset from your configuraiton.

Can you please share the check definition and the definitions for each runtime asset used in the check?

7

Here is a helpful diagnostic check to help you figure out which asset is providing which linked library so you can track down future errors like this. Add additional runtime assets as needed to recreate the environment for the check that is failing. Note the ordering of the runtime assets in your check config may impact the ordering of the directories listed in the LD_LIBRARY_PATH env variable.

The command just outputs which libraries are being linked in. As you add additional assets into the environment the libraries being linked to may change. So make sure you include all assets being used in the check that is failing.

type: CheckConfig
api_version: core/v2
metadata:
  name: ruby_ldd
  namespace: default
spec:
  check_hooks: null
  command: ldd $(which ruby)
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 30
  low_flap_threshold: 0
  output_metric_format: ""
  output_metric_handlers: null
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - sensu-ruby-runtime_centos7_amd64
  stdin: false
  subdue: null
  subscriptions:
  - diagnostic
  timeout: 10
  ttl: 0