Jef Practice: Dealing with unexpected https verification errors using openssl cert directory override

td;dr
openssl libraries included in assets may have a compile time default certificate directory that differs from agent OS configuration.

Set the SSL_CERT_DIR path in the agent-wide environment to explicitly correct the problem for all checks run by the agent.

The longer version
Sometimes when using assets that bundle an openssl library, this compile time default path may be the wrong directory for a particular agent that the asset is being used with.

Maybe the agent is a stripped down container and the certs just aren’t provided.

Maybe the asset was built using custom openssl library with different build time settings than what the host OS assumes.

Maybe the pre-installed certs in the container are out of date.

Whatever the reason, a check that needs to do SSL verification will fail when using that asset because the openssl library won’t have access to the correct certs.

The fix
You can set the SSL_CERT_DIR environment variable in the agent-wide environment to explicitly set directory for all checks the agent runs.

For agents run as containers, this is done by passing runtime envvar options into the container (check with your container orchestrator of choice on specifics). Also don’t forget to volume mount the correct certs if they are missing!

For agents run as systemd services, you can set explicit environment variables
in either configured EnvironmentFile directives:
/etc/default/sensu-agent or /etc/sysconfig/sensu-agent

Note default cert locations for some common linux distributions.

  • CentOS 7 : /etc/pki/tls/certs/
  • Debian 9 : /usr/lib/ssl/certs symlink pointing to ‘/etc/ssl/certs’
  • Ubuntu 18.04 : /usr/lib/ssl/certs symlink pointing to ‘/etc/ssl/certs’
  • Alpine 3.10 : `/etc/ssl/certs’