# Wss-connection issues on sensu deployed by puppet

**URL:** <https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637>\
**Category:** Sensu Go\
**Created:** [July 4, 2021, 9:03pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637 "2021-07-04T21:03:18Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![ahoiroman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/ahoiroman/32/508_2.png) [@ahoiroman](https://discourse.sensu.io/u/ahoiroman)\
**Post date:** [July 4, 2021, 9:03pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/1 "2021-07-04T21:03:18Z")

</div>

Hello there,

all my Sensu checks are failing after rebooting my VMs (could be coincidence)

These are the errors the Sensu logs are giving me:

> Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“error”:“remote error: tls: bad certificate”,“level”:“error”,“msg”:“websocket connection error”,“time”:“2021-07-04T22:56:27+02:00”}  
> Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“level”:“warning”,“msg”:“http: TLS handshake error from 1.2.3.4.148:59160: remote error: tls: bad certificate”,“time”:"2021-07-04T22:56:27+  
> Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“error”:“remote error: tls: bad certificate”,“level”:“error”,“msg”:“websocket connection error”,“time”:“2021-07-04T22:56:27+02:00”}  
> Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“level”:“warning”,“msg”:“http: TLS handshake error from 1.2.3.4.142:52638: remote error: tls: bad certificate”,“time”:"2021-07-04T22:56:27+  
> Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“error”:“remote error: tls: bad certificate”,“level”:“error”,“msg”:“websocket connection error”,“time”:“2021-07-04T22:56:27+02:00”}  
> Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“level”:“warning”,“msg”:“http: TLS handshake error from 1.2.3.4.140:60170: remote error: tls: bad certificate”,“time”:"2021-07-04T22:56:27+  
> Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“error”:“remote error: tls: bad certificate”,“level”:“error”,“msg”:“websocket connection error”,“time”:“2021-07-04T22:56:27+02:00”}  
> Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“level”:“warning”,“msg”:“http: TLS handshake error from 1.2.3.4.146:43198: remote error: tls: bad certificate”,“time”:"2021-07-04T22:56:27+  
> Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“error”:“remote error: tls: bad certificate”,“level”:“error”,“msg”:“websocket connection error”,“time”:“2021-07-04T22:56:27+02:00”}  
> Jul 04 22:56:27 sb1 sensu-backend[3788]: {“component”:“agentd”,“level”:“warning”,“msg”:“http: TLS handshake error from 1.2.3.4.145:36068: remote error: tls: bad certificate”,“time”:"2021-07-04T22:56:27+

This is `backend.yml`:

```auto
---
state-dir: "/var/lib/sensu/sensu-backend"
api-url: https://sensu-backend.example.org:8080
cert-file: "/etc/sensu/ssl/cert.pem"
key-file: "/etc/sensu/ssl/key.pem"
trusted-ca-file: "/etc/sensu/ssl/ca.crt"

```

---

<div class="post-metadata">

**Author:** ![ahoiroman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/ahoiroman/32/508_2.png) [@ahoiroman](https://discourse.sensu.io/u/ahoiroman)\
**Post date:** [July 4, 2021, 9:27pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/2 "2021-07-04T21:27:08Z")

</div>

Setting `use ssl` to `false` ([https://forge.puppet.com/modules/sensu/sensu#advanced-ssl](https://forge.puppet.com/modules/sensu/sensu#advanced-ssl)) solved the problem for now… But it’s not really elegant…

---

<div class="post-metadata">

**Author:** ![elfranne](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/elfranne/32/594_2.png) [@elfranne](https://discourse.sensu.io/u/elfranne)\
**Post date:** [July 6, 2021, 1:16pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/3 "2021-07-06T13:16:22Z")

</div>

try do downgrade agents to 6.3.  
I had same issue:

> <https://github.com/sensu/sensu-go/issues/4340>
>
> \### Initial bug report below, see first comment.
> 
> 
> 
> When using a backend bui…ld with go \>1.15 you can run into issues with certificates
> 
> Some agents will not be able to connect to the backend and will log the following error:
> \`\`\`json
> {"component":"agent","error":"x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0","level":"error","msg":"reconnection attempt failed","time":"2021-06-29T11:07:51+02:00"}
> \`\`\`
> More info from Golang release notes: https://golang.google.cn/doc/go1.15#commonname
> 
> 
> \## Expected Behavior
> sensu-agent should connect to the backend
> 
> 
> \## Current Behavior
> sensu-agent cannot connect to the backend
> 
> \## Possible Solution
> As noted in the error you can set the environment variable on the \*\*agent\*\* and restarting the sensu agent process after setting the environment variable.
> 
> 
> \## Steps to Reproduce (for bugs)
> The issue is only present on our Windows agent. I' ll be looking deeper on the certificates and how they were generated.
> 
> 
> \## Your Environment
> sensu-backend running Ubuntu 18.04 with backend build yesterday ( https://github.com/sensu/sensu-go/tree/d1480a492024045acac62772f62666204ad7f93d ) 
> 
> sensu-agent 6.4.0.4826 on Windows 2012 (and most likely also 6.3 too)

you can add an environment variable as mentionned:  
[https://golang.google.cn/doc/go1.15#commonname](https://golang.google.cn/doc/go1.15#commonname)

---

<div class="post-metadata">

**Author:** ![ahoiroman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/ahoiroman/32/508_2.png) [@ahoiroman](https://discourse.sensu.io/u/ahoiroman)\
**Post date:** [July 11, 2021, 7:35am UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/4 "2021-07-11T07:35:22Z")

</div>

Hi and thanks for your answer.

Could you please tell mich, which env var you set? In the config hash of `sensu::agent` I got this at the moment:

```auto
socket-port: 4033

```

Thanks a lot 🙂

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [July 15, 2021, 5:58pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/5 "2021-07-15T17:58:05Z")

</div>

Hey,  
So this is possibly related to a documented golang deprecation associated with how golang cyrpto package handles TLS certificates. The golang 1.15 release notes have a way to enable the deprecated behavior, but that’s just a short term workaround. If this is your problem, you’ll need to get the certs updated to include the SAN extension.

Ref:  
[https://golang.org/doc/go1.15#commonname](https://golang.org/doc/go1.15#commonname)

We’ve added a note in the Sensu troubleshooting documentation  
[https://docs.sensu.io/sensu-go/latest/operations/maintain-sensu/troubleshoot/#commonname-deprecation-in-go-115](https://docs.sensu.io/sensu-go/latest/operations/maintain-sensu/troubleshoot/#commonname-deprecation-in-go-115)

Just as an aside, the change in cert handling has been a long time coming. This was actually deprecated by an RFC back in 2000, and as far as I personally know golang 1.15 is the first tls implementation in common use that has deprecated the behavior. But I fully expect other TLS implementations to follow suit in the next year now that golang has done it.

---

<div class="post-metadata">

**Author:** ![ahoiroman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/ahoiroman/32/508_2.png) [@ahoiroman](https://discourse.sensu.io/u/ahoiroman)\
**Post date:** [July 15, 2021, 6:11pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/6 "2021-07-15T18:11:55Z")

</div>

Hi and thank you for the update,

so it is no longer possible to use the sensu puppet-module without a custom cert?

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [July 15, 2021, 6:56pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/7 "2021-07-15T18:56:54Z")

</div>

I believe there is work inside puppet specifically to update how certs are being built to conform to best practices. I’m not an active puppet user, so I’m not following the puppet issue closely so I don’t know the full details of what part of the puppet stack needs to be changed.

Here’s a reference to a puppetCA specific issue discussion:  
[https://tickets.puppetlabs.com/browse/SERVER-2338](https://tickets.puppetlabs.com/browse/SERVER-2338)

In the meantime please try setting the GODEBUG environment variable to the value documented in the Golang release notes for sensu backend and agent. That should allow for your existing certs to be used until your puppet infra can issue conformant certs.

---

<div class="post-metadata">

**Author:** ![ahoiroman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/ahoiroman/32/508_2.png) [@ahoiroman](https://discourse.sensu.io/u/ahoiroman)\
**Post date:** [July 15, 2021, 7:08pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/8 "2021-07-15T19:08:39Z")

</div>

Hi,

as I am no GOLANG expert: Is the GODEBUG-variable just a system-variable I could set using puppet or is it something I could set just within the puppet module of sensu?

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [July 15, 2021, 7:16pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/9 "2021-07-15T19:16:37Z")

</div>

It’s just an environment variable.  
Assuming for example sensu-backend is running as a systemd service using the the systemd service file as packaged in the official sensu packages you have several ways to add an environment variable to a running service:

For sensu-backend as packaged your choices are:

1. `/etc/default/sensu-backend` or `/etc/sysconfig/sensu-backend` . This works for official sensu packages because the provided systemd service files is configured to attempt to read those files for environment variable information.
2. extend the systemd service configuration with the necessary environment variable statement in a file under `/etc/systemd/system/sensu-backend.service.d/`. This is a general technique for all systemd services so local admins don’t have to edit vendor provided default service configuration.

Similarly you can set an environment variable for sensu-agent, with file location names changed accordingly.

For systems without systemd, the official sensu packages use a sysvinit script that knows how to do 1. from above.

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [July 15, 2021, 7:37pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/10 "2021-07-15T19:37:48Z")

</div>

> [@ahoiroman](#):
>
> GODEBUG

Based on what I’m reading elsewhere, it appears that

```auto
GODEBUG=x509ignoreCN=0 

```

is the sufficient envvar configuration. Let me know if this works for you. We can get it added to our troubleshooting documentation. Right now all my privately generated certs conform to the RFC best practice so I’m having trouble reproducing (like I said not a puppet user) so I can’t test if this GODEBUG configuration works.

Also here’s a discussion from 2017 the behavior deprecation for context:  
[https://frasertweedale.github.io/blog-redhat/posts/2017-07-11-cn-deprecation.html](https://frasertweedale.github.io/blog-redhat/posts/2017-07-11-cn-deprecation.html)

---

<div class="post-metadata">

**Author:** ![ahoiroman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/ahoiroman/32/508_2.png) [@ahoiroman](https://discourse.sensu.io/u/ahoiroman)\
**Post date:** [July 15, 2021, 7:48pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/11 "2021-07-15T19:48:46Z")

</div>

Thank you for your support.

I have to check this and make sure this is working on all our setups from integration up to production (it’s a very long way for us 😉 )

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [July 15, 2021, 11:57pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/12 "2021-07-15T23:57:29Z")

</div>

Also, if you can report back with regard to anything you find out about how puppet plans to address this, for example if there is a minimum version of puppet that needs to be used to avoid the deprecation workaround. Or if there is a specific puppet configuration change associated with how certs are created that you can share with other users.

Our documented examples for generating self-signed certs uss cfssl tooling have the necessary SAN listing I believe, which is why I didn’t see the problem on the upgrade in my test systems. But we can’t realistically document all possible ways to generate certs (sort of outside the scope of our docs)

But once there’s an understood way to avoid this problem with puppet generated certs, we can probably add the specific puppet learning to our troubleshooting guide to help the next person who runs into this. Outside of puppet now, I’m not sure what other automatic cert generation tooling is impacted this. For example I’m pretty sure lets encrypt generated certs are RFC compliant. Hopefully puppet has a solution for users already or will soon. More and more things are going to start deprecating/disabling support for these out of compliance certs.

---

<div class="post-metadata">

**Author:** ![ahoiroman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/ahoiroman/32/508_2.png) [@ahoiroman](https://discourse.sensu.io/u/ahoiroman)\
**Post date:** [July 16, 2021, 6:37am UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/13 "2021-07-16T06:37:56Z")

</div>

The puppet server is acting as a CA itself and in this role it’s signing all certs of the managed hosts. Replacing the cert, which comes with a very long validity period, is not that easy, especially when there are other tools like Foreman ([theforeman.org](http://theforeman.org)) which make use of the Puppet CA, too.

I will watch this, but I guess I won’t reply back within the next weeks/months. Maybe it would be better, if the sensu module would create a cert itself.

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [July 16, 2021, 6:09pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/14 "2021-07-16T18:09:00Z")

</div>

Hmm that’s an interesting suggestion.

Just to be clear, so from an internal process pov, it would be much easier/faster for you to have bespoke self-signed certs generated by the sensu puppet module?

---

<div class="post-metadata">

**Author:** ![ahoiroman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/ahoiroman/32/508_2.png) [@ahoiroman](https://discourse.sensu.io/u/ahoiroman)\
**Post date:** [July 16, 2021, 6:29pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/15 "2021-07-16T18:29:19Z")

</div>

For me it would be a lot faster, as I would need to replace all certs, not only for the puppet server but also for the foreman-proxies, which are acting as some kind of puppet-proxyserver.

For me it would be the best case to avoid this as I need to check a lot of possible side-effects.

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [July 16, 2021, 6:45pm UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/16 "2021-07-16T18:45:40Z")

</div>

so instead of sensu module doing itself…  
How about using an existing puppet openssl module to generate the certs and place them on disk prior to sensu module running?

I think if you use  
[https://forge.puppet.com/modules/camptocamp/openssl](https://forge.puppet.com/modules/camptocamp/openssl) and use the openssl ‘template’ configuration (file ending in .cnf ) with alternative names defined similar to what is discussed [here](https://geekflare.com/san-ssl-certificate/) you probably get to where you need for self-signed certs.

Sorry, I’m not a puppet user myself so I can’t give you a working example of integrating the openssl module, but I’m pretty confident it should work once you have the correct openssl configuration with alternative names specified.

---

<div class="post-metadata">

**Author:** ![jedihomer](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jedihomer/32/470_2.png) [@jedihomer](https://discourse.sensu.io/u/jedihomer)\
**Post date:** [July 26, 2021, 10:42am UTC](https://discourse.sensu.io/t/wss-connection-issues-on-sensu-deployed-by-puppet/2637/17 "2021-07-26T10:42:29Z")

</div>

We had the same issue here, triggered yesterday when the sensu-agents updated.

We created a self signed certificate for our sensu domain, with the domain repeated in the SAN area and for around 1000 days.

Then for our Sensu backend we added

```json
{
	"sensu::backend::ssl_cert_source": "puppet:///modules/ssl/ssl/ix/sensugo.ourdomain.crt",
	"sensu::backend::ssl_key_source": "puppet:///modules/ssl/ssl/ix/sensugo.ourdomain.key"
}

```

Then our in our base.json file which Hiera includes in all calls

```json
{
	"sensu::ssl_ca_source": "puppet:///modules/ssl/ssl/pd/ca.ourdomain.crt",
}

```

Next time puppet ran it updated our sensu backend and the agents, restarted them and all started working again.

HTH