# Sensu Go 6.0.0 upgrade issue

**URL:** <https://discourse.sensu.io/t/sensu-go-6-0-0-upgrade-issue/2008>\
**Category:** Sensu Go\
**Created:** [August 12, 2020, 9:25am UTC](https://discourse.sensu.io/t/sensu-go-6-0-0-upgrade-issue/2008 "2020-08-12T09:25:00Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![TakeTwo](https://avatars.discourse-cdn.com/v4/letter/t/b3f665/32.png) [@TakeTwo](https://discourse.sensu.io/u/TakeTwo)\
**Post date:** [August 12, 2020, 9:25am UTC](https://discourse.sensu.io/t/sensu-go-6-0-0-upgrade-issue/2008/1 "2020-08-12T09:25:00Z")

</div>

Hi,

I’m following the [instructions](https://docs.sensu.io/sensu-go/latest/operations/maintain-sensu/upgrade/#upgrade-to-sensu-go-60-from-a-5x-deployment), but I’m having issues with step 3 upgrading sensu-backend.

**?**  **Do you really want to upgrade your Sensu 5.x database to 6.x? This operation cannot be undone; make sure you back up your database!** Yes

{“component”:“store”,“level”:“warning”,“msg”:“migrating etcd database to a new version”,“time”:“2020-08-12T10:55:46+02:00”}

{“component”:“store”,“database\_version”:1,“level”:“error”,“msg”:“error upgrading database”,“time”:“2020-08-12T10:55:46+02:00”}

{“component”:“sensu-enterprise”,“error”:“the namespace production does not exist”,“level”:“fatal”,“msg”:“error executing sensu-backend”,“time”:“2020-08-12T10:55:46+02:00”}

I don’t have a namespace called production, so not sure where this is coming from.

sensuctl namespace list

**Name**

──────

**dev**

**prod**

Sensu version before upgrade: 5.21.0  
OS: CentOS Linux release 7.8.2003 (Core)

---

<div class="post-metadata">

**Author:** ![palourde](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/palourde/32/60_2.png) [@palourde](https://discourse.sensu.io/u/palourde)\
**Post date:** [August 12, 2020, 2:34pm UTC](https://discourse.sensu.io/t/sensu-go-6-0-0-upgrade-issue/2008/2 "2020-08-12T14:34:29Z")

</div>

Hi @TakeTwo

I have a feeling there’s some unexpected keys in etcd that might refer to a `production` namespace, that maybe you previously deleted?

Just to verify that, could you try to [install `etcdctl`](https://etcd.io/docs/v3.4.0/dl-build/) on the backend machine, and list all keys with `production` within their path (you might have to adjust some flags if you configured etcd with TLS authentication)

```auto
etcdctl get /sensu.io --prefix --keys-only | grep default

```

If some keys are returned, you might have to manually delete them, using something like this:

```auto
etcdctl del /sensu.io/path/to/key

```

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [August 12, 2020, 7:49pm UTC](https://discourse.sensu.io/t/sensu-go-6-0-0-upgrade-issue/2008/3 "2020-08-12T19:49:56Z")

</div>

@TakeTwo  
Just a quick command fix up… grep for production instead of default

```auto
etcdctl get /sensu.io --prefix --keys-only | grep production

```

So a little bit more on this. I’m not sure how to get into this situation. I just deleted a namespace, made sure there were still resource keys in the namespace and then did the upgrade. I wasn’t able to reproduce the error starting from 5.21.1.

Here’s what I did step by step.

1. ensure namespace `action_CICD` exists
2. populate role, rolebindings and a check in the namespace
3. 

```auto
etcdctl get /sensu.io --prefix --keys-only | grep action_CICD`
/sensu.io/api/internal/metricsd/v1/metrics/action_CICD/entity_gauges
/sensu.io/api/internal/metricsd/v1/metrics/action_CICD/event_gauges
/sensu.io/api/internal/metricsd/v1/metrics/action_CICD/keepalive_gauges
/sensu.io/checks/action_CICD/test
/sensu.io/namespaces/action_CICD
/sensu.io/rbac/rolebindings/action_CICD/namespace-admins
/sensu.io/rbac/rolebindings/action_CICD/namespace-operators
/sensu.io/rbac/roles/action_CICD/namespace-admin
/sensu.io/rbac/roles/action_CICD/namespace-operator

```

1. delete the namespace
2. resource related keys are still in place

```auto
/sensu.io/api/internal/metricsd/v1/metrics/action_CICD/entity_gauges
/sensu.io/api/internal/metricsd/v1/metrics/action_CICD/event_gauges
/sensu.io/api/internal/metricsd/v1/metrics/action_CICD/keepalive_gauges
/sensu.io/checks/action_CICD/test
/sensu.io/rbac/rolebindings/action_CICD/namespace-admins
/sensu.io/rbac/rolebindings/action_CICD/namespace-operators
/sensu.io/rbac/roles/action_CICD/namespace-admin
/sensu.io/rbac/roles/action_CICD/namespace-operator

```

1. did the upgrade, no problem

@TakeTwo, it would definitely would be useful to see that key output to see if we can determine if there is a specific resource key that is causing the problem for you. My quick test of just deleting a namespace didn’t cause a problem for me, so its definitely something subtle.

In the meantime, if you do see production namespaced resources in your etcdctl key output, it might be easier to add the namespace back do the upgrade then delete the namespace again after the upgrade. Though I really want to see what etcd key names are referring to production.

---

<div class="post-metadata">

**Author:** ![TakeTwo](https://avatars.discourse-cdn.com/v4/letter/t/b3f665/32.png) [@TakeTwo](https://discourse.sensu.io/u/TakeTwo)\
**Post date:** [August 13, 2020, 12:53pm UTC](https://discourse.sensu.io/t/sensu-go-6-0-0-upgrade-issue/2008/4 "2020-08-13T12:53:21Z")

</div>

Thanks for the suggestions @palourde and @jspaleta.

Unfortunately I haven’t had much luck with this, it may be a bit beyond my technical capabilities to troubleshoot.

```
etcdctl -ca-file /etc/sensu/tls/xxx.pem --cert-file /etc/sensu/tls/xxx.pem --key-file /etc/sensu/tls/xxx-key.pem get /sensu.io --prefix --keys-only | grep production

flag provided but not defined: -prefix

```

I’ve tried swapping between API 2 and 3 without any difference.

Removing these flags gives the following error;

```
Error: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:4001: connect: connection refused

; error #1: EOF

error #0: dial tcp 127.0.0.1:4001: connect: connection refused

error #1: EOF

```

Adding --endpoints “[https://localhost:2379](https://localhost:2379)” seems to allow connectivity, but not sure this is the endpoint to use for this purpose? This gives the following error;

`Error: 100: Key not found (/sensu.io) [53]`

---

<div class="post-metadata">

**Author:** ![palourde](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/palourde/32/60_2.png) [@palourde](https://discourse.sensu.io/u/palourde)\
**Post date:** [August 13, 2020, 8:18pm UTC](https://discourse.sensu.io/t/sensu-go-6-0-0-upgrade-issue/2008/5 "2020-08-13T20:18:28Z")

</div>

Could you try to run the same first command, but by prefixing the command with `ETCDCTL_API=3`, so something like

```auto
ETCDCTL_API=3 etcdctl -ca-file /etc/sensu/tls/xxx.pem --cert-file /etc/sensu/tls/xxx.pem --key-file /etc/sensu/tls/xxx-key.pem get /sensu.io --prefix --keys-only | grep production

```

---

<div class="post-metadata">

**Author:** ![TakeTwo](https://avatars.discourse-cdn.com/v4/letter/t/b3f665/32.png) [@TakeTwo](https://discourse.sensu.io/u/TakeTwo)\
**Post date:** [August 14, 2020, 6:46am UTC](https://discourse.sensu.io/t/sensu-go-6-0-0-upgrade-issue/2008/6 "2020-08-14T06:46:16Z")

</div>

Thanks @palourde, that did the trick.

```
ETCDCTL_API=3 etcdctl --cacert=/etc/sensu/tls/ca.pem --cert=/etc/sensu/tls/xxx.pem --key=/etc/sensu/tls/xxx.pem get /sensu.io --prefix --keys-only | grep production
/sensu.io/api/internal/metricsd/v1/metrics/production/entity_gauges
/sensu.io/api/internal/metricsd/v1/metrics/production/event_gauges
/sensu.io/api/internal/metricsd/v1/metrics/production/keepalive_gauges
/sensu.io/entities/production/xxx
/sensu.io/events/production/xxx/keepalive
/sensu.io/switchsets/lease/keepalived/production/xxx 

```

(Certificates and server name replaced with xxx)

I can hold off on deleting the keys if there’s some other debugging you like me to do first.

---

<div class="post-metadata">

**Author:** ![palourde](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/palourde/32/60_2.png) [@palourde](https://discourse.sensu.io/u/palourde)\
**Post date:** [August 14, 2020, 2:17pm UTC](https://discourse.sensu.io/t/sensu-go-6-0-0-upgrade-issue/2008/7 "2020-08-14T14:17:11Z")

</div>

Glad to hear this @TakeTwo!

Feel free to delete those keys; you will need to delete them one by one using a command similar to this:

```auto
ETCDCTL_API=3 etcdctl --cacert=/etc/sensu/tls/ca.pem --cert=/etc/sensu/tls/xxx.pem --key=/etc/sensu/tls/xxx.pem del /sensu.io/path/to/key

```

Do you remember what was the original version you installed on this cluster? I believe some older Sensu Go versions may have allowed you to delete a namespace that wasn’t empty, but it has been fixed for a couple of releases I think.