Building native extensions for ruby-pg succeeds, but require fails

Sensu Go
1

Hello,

I’m using sensu-ruby-runtime to install pg gem. As it requires native extensions I had to rename libcrypto.so.1.0.0 to libcrypto.so.10, libgmp.so.10 to libgmp.so and libssl.so.1.0.0 to libssl.so.10 in order for it to successfully compile.

It compiled after these changes, but now when I require it inside my plugin I’m getting the following error:

/var/cache/sensu/sensu-agent/8d768d1fba545898a8d09dca603457eb0018ec6829bc5f609a1ea51a2be0c4b2d13e1aa46139ecbb04873449e4c76f463f0bdfbaf2107caf37ab1c8db87d5250/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require’: Error relocating /var/cache/sensu/sensu-agent/8d768d1fba545898a8d09dca603457eb0018ec6829bc5f609a1ea51a2be0c4b2d13e1aa46139ecbb04873449e4c76f463f0bdfbaf2107caf37ab1c8db87d5250/lib/ruby/gems/2.4.0/gems/pg-1.1.3/lib/pg_ext.so: PQsslAttribute: symbol not found - /var/cache/sensu/sensu-agent/8d768d1fba545898a8d09dca603457eb0018ec6829bc5f609a1ea51a2be0c4b2d13e1aa46139ecbb04873449e4c76f463f0bdfbaf2107caf37ab1c8db87d5250/lib/ruby/gems/2.4.0/gems/pg-1.1.3/lib/pg_ext.so (LoadError)

As far as I can understand, since it’s a dynamically-linked library it tries to load system libs and those are not of the right version. Assuming this is correct, are there any recommendations on how to fix that issue?

Best,
Konstantin

2

Are you doing this as part of a asset build process managed by TravisCI, as per the normal pattern for asset enablement? Is there a github repo I can look at?

I’ve found what generally works is to vacuum up the corresponding system libs needed by any library built (excluding system libs provided by libc,this is very important as it turns out) and provide them as well in the assets /lib.

I’ve been working towards a re-usable pattern for native extension gems, but I haven’t gotten there yet.

What I do have as a starting point is a Dockerfile based build process that uses an ldd script to selectively find the dependent system libraries and include them in the asset.

For an example, take a look at the sensu-ruby-runtime provided Dockerfile:
https://github.com/sensu/sensu-ruby-runtime/blob/5123017d3dadf2067fa90fc28275b92e9b586885/Dockerfile.debian

This Dockerfile attempts to discover all the required dynamically linked system libraries needed and include them in the asset lib. The script explictly excludes the libraries i know are part of the glibc package on debian.

So in your build process, after you have the native extensions built, it should be sufficient to run a similar ldd script to populate the asset lib/ directory with any additional dynlibs needed from your build system.

I can’t stress this enough, you have to exclude the libc provided libraries, or else dynamic linking may not work. On each linux distribution family (its not just a musl versus glibc difference) these are a different set of libraries, so the Dockerfiles have the default set of libs to exclude tuned accordingly.

3

Thanks for your response.

I’ve been building that extension manually (getting a tty on a container and updating PATHs) just to see how it goes.

From what I can tell, all required libs are provided along with sensu-ruby-runtime, though some of them have to be renamed in order for the extension to compile.

If I had to build my base system from scratch and then package assets, then following your Dockferfile workflow would totally make sense. But, as I mentioned above, it seems that everything should just work with the provided Alpine-based Sensu-go container that I’m using to build this native extension - after all, the same exact workflow has been executed to get sensu-ruby-runtime built for that particular distro/arch combination.

4

the renaming of libs is just odd. I don’t completely follow what that’s required. Is this an alpine-ism? I haven’t run into that particular requirement yet in my prototyping some other plugin assets. I need to find the time to build the postgres plugin asset via the travisCI workflow and figure out what’s going on.

5

My exposure to low-level languages is pretty limited, so unfortunately I can’t help much in that department. I can see that the first error during compilation references libpq-fe.h header file and that, in turn, includes PQsslAttribute referenced in my error message above.

I’ll keep digging to see if I can figure out something new.

Thank you!

6

After playing some more with my installation, I figured that my plugin actually works when I run it manually (executing something shell command like sh -c "/usr/bin/env ruby my_plugin.rb"). However, when Sensu tries to run the same check I’m getting the same issue described above.

Here’s the state of ENV when Sensu runs the check:

{"RUBY_MAJOR"=>"2.7", "HOSTNAME"=>"8fb9f4bac467", "LD_LIBRARY_PATH"=>"/var/cache/sensu/sensu-agent/505cf2f8cabbcf8be64149a522d9080025366cc2d5ccafe38e842131e37234a46811a2126b3480c1b9f4faa1912984be5fb9a9aca7b456027ccd5df964c56792/lib::", "SHLVL"=>"1", "HOME"=>"/root", "BUNDLE_APP_CONFIG"=>"/usr/local/bundle", "RUBY_VERSION"=>"2.7.0", "PATH"=>"/var/cache/sensu/sensu-agent/505cf2f8cabbcf8be64149a522d9080025366cc2d5ccafe38e842131e37234a46811a2126b3480c1b9f4faa1912984be5fb9a9aca7b456027ccd5df964c56792/bin:/usr/local/bundle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bundle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "GEM_HOME"=>"/usr/local/bundle", "RUBY_DOWNLOAD_SHA256"=>"27d350a52a02b53034ca0794efe518667d558f152656c2baaf08f3d0c8b02343", "PWD"=>"/root", "BUNDLE_SILENCE_ROOT_WARNING"=>"1", "CPATH"=>"/var/cache/sensu/sensu-agent/505cf2f8cabbcf8be64149a522d9080025366cc2d5ccafe38e842131e37234a46811a2126b3480c1b9f4faa1912984be5fb9a9aca7b456027ccd5df964c56792/include::"}

Here’s ENV when I run it manually:

{"RUBY_MAJOR"=>"2.7", "HOSTNAME"=>"8fb9f4bac467", "SHLVL"=>"2", "HOME"=>"/root", "OLDPWD"=>"/root", "BUNDLE_APP_CONFIG"=>"/usr/local/bundle", "RUBY_VERSION"=>"2.7.0", "TERM"=>"xterm", "PATH"=>"/usr/local/bundle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "GEM_HOME"=>"/usr/local/bundle", "RUBY_DOWNLOAD_SHA256"=>"27d350a52a02b53034ca0794efe518667d558f152656c2baaf08f3d0c8b02343", "PWD"=>"/var/cache/sensu/sensu-agent/505cf2f8cabbcf8be64149a522d9080025366cc2d5ccafe38e842131e37234a46811a2126b3480c1b9f4faa1912984be5fb9a9aca7b456027ccd5df964c56792/bin", "BUNDLE_SILENCE_ROOT_WARNING"=>"1"}

And here’s the error:

/usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require': Error loading shared library libssl.so.10: No such file or directory (needed by /var/cache/sensu/sensu-agent/505cf2f8cabbcf8be64149a522d9080025366cc2d5ccafe38e842131e37234a46811a2126b3480c1b9f4faa1912984be5fb9a9aca7b456027ccd5df964c56792/lib/libpq.so.5) - /usr/local/bundle/gems/pg-1.2.1/lib/pg_ext.so (LoadError)
	from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	from /usr/local/bundle/gems/pg-1.2.1/lib/pg.rb:5:in `<top (required)>'
	from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:168:in `require'
	from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:168:in `rescue in require'
	from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:156:in `require'
	from /var/cache/sensu/sensu-agent/505cf2f8cabbcf8be64149a522d9080025366cc2d5ccafe38e842131e37234a46811a2126b3480c1b9f4faa1912984be5fb9a9aca7b456027ccd5df964c56792/bin/db_helpers.rb:3:in `<top (required)>'
	from /var/cache/sensu/sensu-agent/505cf2f8cabbcf8be64149a522d9080025366cc2d5ccafe38e842131e37234a46811a2126b3480c1b9f4faa1912984be5fb9a9aca7b456027ccd5df964c56792/bin/ratio_check.rb:10:in `require_relative'
	from /var/cache/sensu/sensu-agent/505cf2f8cabbcf8be64149a522d9080025366cc2d5ccafe38e842131e37234a46811a2126b3480c1b9f4faa1912984be5fb9a9aca7b456027ccd5df964c56792/bin/ratio_check.rb:10:in `<main>'
/usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require': cannot load such file -- pg (LoadError)
	from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	from /var/cache/sensu/sensu-agent/505cf2f8cabbcf8be64149a522d9080025366cc2d5ccafe38e842131e37234a46811a2126b3480c1b9f4faa1912984be5fb9a9aca7b456027ccd5df964c56792/bin/db_helpers.rb:3:in `<top (required)>'
	from /var/cache/sensu/sensu-agent/505cf2f8cabbcf8be64149a522d9080025366cc2d5ccafe38e842131e37234a46811a2126b3480c1b9f4faa1912984be5fb9a9aca7b456027ccd5df964c56792/bin/ratio_check.rb:10:in `require_relative'
	from /var/cache/sensu/sensu-agent/505cf2f8cabbcf8be64149a522d9080025366cc2d5ccafe38e842131e37234a46811a2126b3480c1b9f4faa1912984be5fb9a9aca7b456027ccd5df964c56792/bin/ratio_check.rb:10:in `<main>'
7

looks like from this error you need to ensure that shared library is bundled into the asset’s lib/ directory when you roll up the asset tarball.

What’s happening is one of the libs in the asset needs that specific shared library but its not on the system anywhere.

Bacause this is alpine (this is alpine container right?)… most likely the system installed version of libpq.so used when you manually run is linked against libressl NOT openssl.
But when you build against ruby-runtime its linking and looking for openssl.

Shared libraries are the tricky bit to get right with assets. Alpine always seems to have the most interesting cornercases because of the way Alpine choice its defaults…it breaks a lot of assumptions by choosing alternative implementations for some core bits of tech (libc and ssl implementations for example). And the difference between openssl and ressl are…one of those things you don’t see in the older linux distriubtions.

Long term there is huge benefit to moving to statically compiled golang for assets to avoid this problem.

Inspired by this thread, I’ve gone back and put some work into getting sensu-plugins-postgres and run into some build issues around openssl versus ressl libraries that definitely seem to correlate with your problem here. I’ll report back with links to my test assets once i get things building in CI for you to look at… especially the alpine Dockerfiles I use to build the assets.

8

So also what’s really interesting is your error mentions:

/usr/local/bundle/gems/pg-1.2.1/lib/pg_ext.so

which is outside of your installed asset.

I would expect that pg_ext.so to be picked up from inside your asset.

So it looks like what is happening is your container running sensu has the postgres native extension installed outside of your asset already… and but your asset is providing libpg.so

So something is definitely a messed up your build ennvironment.

Checking really quickly in the sensu-plugins-postgres asset ive built locally. pg_ext.so is provided in the asset tarball.

So i’ll work on getting test assets up built in CI as part of a github release for you to look at tonight.

9

Jef, thank you very much for trying to help, I really appreciate it.

I should’ve mentioned that before posting my previous message: I started with ruby:alpine (latest) image and added Sensu-go to that image. That’s why my Ruby path is set to /usr/local/* and the Ruby version is 2.7.0. I also removed sensu-ruby-runtime altogether and using system Ruby (2.7.0) to run my asset.

Your explanation about static and dynamic libs makes total sense and that’s pretty much was my understanding at this point.

What I don’t understand is how it is possible for me to successfully run my check manually (as I said earlier by executing ruby ratio_check.rb or even sh -c "ratio_check.rb" when the bin dir is present at PATH) while Sensu fails with missing pg dependency and invalid shared libs. That part is really confusing to me.

10

it has to do with the contents of the asset itself.

you have libpg.so in your assets lib/ directory and its getting picked up as the prefered libpg.so over the system provided one.

BUT the one in your asset is built against openssl not ressl and openssl isn’t available on the system.

If you removed libpg.so from your asset, it would probably work.

The real problem here is that you are sort of mixing half asset with half on system and getting strange results because the asset isnt providing all the libs

to do a postgres asset correctly you really have to find a way to include the pg_ext native extension library into the asset… and not use the pg_ext in the containter.

the chain of shared libraries on your system looks like this

pg_ext.so from outside your asset installed in /usr/local requires libpg.so

So when you run things manually… the linker sees the libpg.so provided by the system and everything works…because that system provided libpg.so depends on libressl which is on the system.

But when you run from inside sensu… the linker’s search path is manipulated to first look into the assets lib directory…and the libpg.so there is built against libopenssl which is not provided by the system nor is it in your asset…so the dynlib look up fails.

The linker search path manipulation is what makes assets work. But it only works if you ensure all the shared libs (except for libc libraries) that you build against are also in the asset.

ldd is a really useful tool for exploring what is going on in terms of linker operation.

1 Like