Wss-connection issues on sensu deployed by puppet

Sensu Go
1

Hello there,

all my Sensu checks are failing after rebooting my VMs (could be coincidence)

These are the errors the Sensu logs are giving me:

Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“error”:“remote error: tls: bad certificate”,“level”:“error”,“msg”:“websocket connection error”,“time”:“2021-07-04T22:56:27+02:00”}
Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“level”:“warning”,“msg”:“http: TLS handshake error from 1.2.3.4.148:59160: remote error: tls: bad certificate”,“time”:"2021-07-04T22:56:27+
Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“error”:“remote error: tls: bad certificate”,“level”:“error”,“msg”:“websocket connection error”,“time”:“2021-07-04T22:56:27+02:00”}
Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“level”:“warning”,“msg”:“http: TLS handshake error from 1.2.3.4.142:52638: remote error: tls: bad certificate”,“time”:"2021-07-04T22:56:27+
Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“error”:“remote error: tls: bad certificate”,“level”:“error”,“msg”:“websocket connection error”,“time”:“2021-07-04T22:56:27+02:00”}
Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“level”:“warning”,“msg”:“http: TLS handshake error from 1.2.3.4.140:60170: remote error: tls: bad certificate”,“time”:"2021-07-04T22:56:27+
Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“error”:“remote error: tls: bad certificate”,“level”:“error”,“msg”:“websocket connection error”,“time”:“2021-07-04T22:56:27+02:00”}
Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“level”:“warning”,“msg”:“http: TLS handshake error from 1.2.3.4.146:43198: remote error: tls: bad certificate”,“time”:"2021-07-04T22:56:27+
Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“error”:“remote error: tls: bad certificate”,“level”:“error”,“msg”:“websocket connection error”,“time”:“2021-07-04T22:56:27+02:00”}
Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“level”:“warning”,“msg”:“http: TLS handshake error from 1.2.3.4.145:36068: remote error: tls: bad certificate”,“time”:"2021-07-04T22:56:27+

This is backend.yml:

---
state-dir: "/var/lib/sensu/sensu-backend"
api-url: https://sensu-backend.example.org:8080
cert-file: "/etc/sensu/ssl/cert.pem"
key-file: "/etc/sensu/ssl/key.pem"
trusted-ca-file: "/etc/sensu/ssl/ca.crt"
2

Setting use ssl to false (https://forge.puppet.com/modules/sensu/sensu#advanced-ssl) solved the problem for now… But it’s not really elegant…

3

try do downgrade agents to 6.3.
I had same issue:

you can add an environment variable as mentionned:
https://golang.google.cn/doc/go1.15#commonname

4

Hi and thanks for your answer.

Could you please tell mich, which env var you set? In the config hash of sensu::agent I got this at the moment:

socket-port: 4033

Thanks a lot :slight_smile:

5

Hey,
So this is possibly related to a documented golang deprecation associated with how golang cyrpto package handles TLS certificates. The golang 1.15 release notes have a way to enable the deprecated behavior, but that’s just a short term workaround. If this is your problem, you’ll need to get the certs updated to include the SAN extension.

Ref:
https://golang.org/doc/go1.15#commonname

We’ve added a note in the Sensu troubleshooting documentation
https://docs.sensu.io/sensu-go/latest/operations/maintain-sensu/troubleshoot/#commonname-deprecation-in-go-115

Just as an aside, the change in cert handling has been a long time coming. This was actually deprecated by an RFC back in 2000, and as far as I personally know golang 1.15 is the first tls implementation in common use that has deprecated the behavior. But I fully expect other TLS implementations to follow suit in the next year now that golang has done it.

6

Hi and thank you for the update,

so it is no longer possible to use the sensu puppet-module without a custom cert?

7

I believe there is work inside puppet specifically to update how certs are being built to conform to best practices. I’m not an active puppet user, so I’m not following the puppet issue closely so I don’t know the full details of what part of the puppet stack needs to be changed.

Here’s a reference to a puppetCA specific issue discussion:
https://tickets.puppetlabs.com/browse/SERVER-2338

In the meantime please try setting the GODEBUG environment variable to the value documented in the Golang release notes for sensu backend and agent. That should allow for your existing certs to be used until your puppet infra can issue conformant certs.

8

Hi,

as I am no GOLANG expert: Is the GODEBUG-variable just a system-variable I could set using puppet or is it something I could set just within the puppet module of sensu?

9

It’s just an environment variable.
Assuming for example sensu-backend is running as a systemd service using the the systemd service file as packaged in the official sensu packages you have several ways to add an environment variable to a running service:

For sensu-backend as packaged your choices are:

  1. /etc/default/sensu-backend or /etc/sysconfig/sensu-backend . This works for official sensu packages because the provided systemd service files is configured to attempt to read those files for environment variable information.
  2. extend the systemd service configuration with the necessary environment variable statement in a file under /etc/systemd/system/sensu-backend.service.d/. This is a general technique for all systemd services so local admins don’t have to edit vendor provided default service configuration.

Similarly you can set an environment variable for sensu-agent, with file location names changed accordingly.

For systems without systemd, the official sensu packages use a sysvinit script that knows how to do 1. from above.

10

Based on what I’m reading elsewhere, it appears that

GODEBUG=x509ignoreCN=0

is the sufficient envvar configuration. Let me know if this works for you. We can get it added to our troubleshooting documentation. Right now all my privately generated certs conform to the RFC best practice so I’m having trouble reproducing (like I said not a puppet user) so I can’t test if this GODEBUG configuration works.

Also here’s a discussion from 2017 the behavior deprecation for context:
https://frasertweedale.github.io/blog-redhat/posts/2017-07-11-cn-deprecation.html

11

Thank you for your support.

I have to check this and make sure this is working on all our setups from integration up to production (it’s a very long way for us :wink: )

12

Also, if you can report back with regard to anything you find out about how puppet plans to address this, for example if there is a minimum version of puppet that needs to be used to avoid the deprecation workaround. Or if there is a specific puppet configuration change associated with how certs are created that you can share with other users.

Our documented examples for generating self-signed certs uss cfssl tooling have the necessary SAN listing I believe, which is why I didn’t see the problem on the upgrade in my test systems. But we can’t realistically document all possible ways to generate certs (sort of outside the scope of our docs)

But once there’s an understood way to avoid this problem with puppet generated certs, we can probably add the specific puppet learning to our troubleshooting guide to help the next person who runs into this. Outside of puppet now, I’m not sure what other automatic cert generation tooling is impacted this. For example I’m pretty sure lets encrypt generated certs are RFC compliant. Hopefully puppet has a solution for users already or will soon. More and more things are going to start deprecating/disabling support for these out of compliance certs.

13

The puppet server is acting as a CA itself and in this role it’s signing all certs of the managed hosts. Replacing the cert, which comes with a very long validity period, is not that easy, especially when there are other tools like Foreman (theforeman.org) which make use of the Puppet CA, too.

I will watch this, but I guess I won’t reply back within the next weeks/months. Maybe it would be better, if the sensu module would create a cert itself.

14

Hmm that’s an interesting suggestion.

Just to be clear, so from an internal process pov, it would be much easier/faster for you to have bespoke self-signed certs generated by the sensu puppet module?

15

For me it would be a lot faster, as I would need to replace all certs, not only for the puppet server but also for the foreman-proxies, which are acting as some kind of puppet-proxyserver.

For me it would be the best case to avoid this as I need to check a lot of possible side-effects.

16

so instead of sensu module doing itself…
How about using an existing puppet openssl module to generate the certs and place them on disk prior to sensu module running?

I think if you use
https://forge.puppet.com/modules/camptocamp/openssl and use the openssl ‘template’ configuration (file ending in .cnf ) with alternative names defined similar to what is discussed here you probably get to where you need for self-signed certs.

Sorry, I’m not a puppet user myself so I can’t give you a working example of integrating the openssl module, but I’m pretty confident it should work once you have the correct openssl configuration with alternative names specified.

17

We had the same issue here, triggered yesterday when the sensu-agents updated.

We created a self signed certificate for our sensu domain, with the domain repeated in the SAN area and for around 1000 days.

Then for our Sensu backend we added

{
	"sensu::backend::ssl_cert_source": "puppet:///modules/ssl/ssl/ix/sensugo.ourdomain.crt",
	"sensu::backend::ssl_key_source": "puppet:///modules/ssl/ssl/ix/sensugo.ourdomain.key"
}

Then our in our base.json file which Hiera includes in all calls

{
	"sensu::ssl_ca_source": "puppet:///modules/ssl/ssl/pd/ca.ourdomain.crt",
}

Next time puppet ran it updated our sensu backend and the agents, restarted them and all started working again.

HTH

2 Likes