Hi, we’re trying to get our sensu-go backend up and running with TLS. I can’t get the peers to talk over etcd. I’m not using an external etcd cluster and I’m getting malformed net/http which makes me think it’s trying to communicate over http on an https endpoint. Any thoughts to this regard or an example backend.yml would help.
The error I’m trying to solve is the following:
My existing backend.yml is the following:
The certs I’m using have the CN of Agent and the IP SANs for both nodes in the cluster. (I have a 3rd unconfigured node). These are self-signed certs from our Hashicorp vault CA. All nodes are Ubuntu 18.04 running Sensu-Go 5.1. I’m most concerned I’m missing a “hidden” configuration parameter.
I’ve recently tried changing the cluster token to spawn a new etcd raft db and I’ve also shut down the cluster and deleted the existing data store to ensure there are no old messages floating around. I’ve also added etcd-client-cert-auth: true to the backend.yml to no effect.
The cluster does bootstrap if I comment out all of the TLS config lines (lines 23-35) and change all the endpoints to HTTP.
Looking over your configuration, nothing is immediately standing out to me as incorrect.
Here is a backend.yml that works for me:
---
##
# general configuration
##
state-dir: "/var/lib/sensu/sensu-backend"
##
# agent configuration
##
#agent-host: "[::]" # listen on all IPv4 and IPv6 addresses
#agent-port: 8081
##
# api configuration
##
#api-host: "[::]" # listen on all IPv4 and IPv6 addresses
#api-port: 8080
api-url: "https://10.0.0.1:8080"
##
# dashboard ssl configuration
##
cert-file: "/etc/sensu/certs/backend-1.pem"
key-file: "/etc/sensu/certs/backend-1-key.pem"
trusted-ca-file: "/etc/pki/ca-trust/source/anchors/ca.pem"
##
# etcd ssl configuration
##
etcd-cert-file: "/etc/sensu/certs/backend-1.pem"
etcd-key-file: "/etc/sensu/certs/backend-1-key.pem"
etcd-trusted-ca-file: "/etc/pki/ca-trust/source/anchors/ca.pem"
etcd-client-cert-auth: true
etcd-peer-cert-file: "/etc/sensu/certs/backend-1.pem"
etcd-peer-key-file: "/etc/sensu/certs/backend-1-key.pem"
etcd-peer-trusted-ca-file: "/etc/pki/ca-trust/source/anchors/ca.pem"
etcd-peer-client-cert-auth: true
#insecure-skip-tls-verify: false
##
# store configuration for backend-1/10.0.0.1
##
etcd-listen-client-urls: "https://10.0.0.1:2379"
etcd-advertise-client-urls: "https://10.0.0.1:2379"
etcd-listen-peer-urls: "https://10.0.0.1:2380"
etcd-initial-cluster: "backend-1=https://10.0.0.1:2380,backend-2=https://10.0.0.2:2380,backend-3=https://10.0.0.3:2380"
etcd-initial-advertise-peer-urls: "https://10.0.0.1:2380"
etcd-initial-cluster-state: "new"
#etcd-initial-cluster-token: "sensu"
etcd-name: "backend-1"
##
# dashboard configuration
##
#dashboard-host: "[::]" # listen on all IPv4 and IPv6 addresses
#dashboard-port: 3000
##
# other
##
#config-file: ""
#debug: false
#deregistration-handler: ""
#log-level: "warn"
The only difference I see is the value for state-dir being /var/lib/sensu/sensu-backend in my example.
What operating system are the backends running on? Your certs could also be the source. I know the error you are getting, as I’ve ran into myself a few times for different reasons.
All of the backends are running Ubuntu 16.04 currently. I just recreated all the instances the other day and i’ve moved the state-dir to /var/lib/sensu/sensu-backend since that’s now the default with 5.1.1. I was having plenty of problems getting the cluster to talk even without TLS after doing some additional tests.
Now i have 2 members talking without TLS with new hosts and new data store. I’ll try to convert it to TLS again and see what happens.
Did you get this going ? I have the same issue. etcd cluster without TLS is working without any issues. New setup with TLS is not starting correctly and thus Sensu is not starting.
Im attaching my config for one of three nodes
Hi Darth,
Did you get this going ? I have the same issue. etcd cluster without TLS is working without any issues. New setup with TLS is not starting correctly and thus Sensu is not starting.
I was on 4.6 I think. I don’t recall. I’m on 5.1 now. I never got the built in etcd cluster to work under TLS. The config options and what the related to in the etcd API we’re confusing me.
My plan, when I have time is to leverage the auto TLS option made available in 5.1
Oof. 5.1 is quite old in terms of our releases. We just pushed out 5.12 yesterday, so I’m keen to see if this was an issue with one of the early releases and upgrading might address the issue. I’d also be keen to see if this is a shortcoming of our docs and we need to do a better job of documenting setting up TLS.
@alektrawicki
i’ve got a brand bew 3 node cluster up using 5.12. Going to work on changing it to use TLS next… I’ll report back. I’m going to try to document my cert creation process step by step to make sure.
One question, how did you create your certs? And were you able to test that your certs work outside the context of sensu’s configs?
I did get an error when I changed an IP on one of the servers and then generated a new one without issues. So I guess they work. I havent tested them outside of Sensu/etcd. I can try to check with curl.
@alektrawicki
Hmm can you actually set the ipaddress to 0.0.0.0 like that in a cert? I mean you are basically saying that cert applies to all ipv4 address. Is that allowed?
I get an error.
drift@sat-xXXX-01:~$ curl --cacert /usr/local/share/ca-certificates/sensu/ca.pem -L https://100.100.0.xx:2380/version
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
Now that is perhaps indicative of a problem with the cert and I’m not expert enough in tls certs to know what that means.
I’m able to use that curl test with a DNS wildcard cert I made with the openssl cmdline tooling.
I can’t use the guide instructions because I don’t have a new enough version of go to use the golang package… it won’t compile form me with this older golang version i have locally.
So i a more manual openssl process…which sort of works…but only if i create a fake domain name. Still havent gotten the ip addresses to work with my curl test without using -k flag.
when I use the ip address… curl --cacert /usr/local/share/ca-certificates/sensu/rootCA.crt -L https://192.168.14.24:2380/version
returns: curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.
but adding -k ignores the error…which is what i would expect for a wild card dns cert.