"failed to request new refresh token; client returned 'Post https://hostname:8080/auth/token: Forbidden'

This is somewhat weird because I find nothing at all about it in the docs and nothing relevant on Google.

6 months ago I set up a sensu go cluster, initialized the client, created assets, checks, all the stuff.
Since then I simple let it run and updated it regularly, but did not change config - just added systems / agents.

Now I want to add further checks etc. - and upon trying that I get the following error:

sensuctl check create check-glusterfs --command ‘/opt/sensu-plugins-ruby/added/check_glusterfs’ --interval 300 --subscriptions glusterfs
Error: failed to request new refresh token; client returned ‘Post https://hostname:8080/auth/token: Forbidden’

This is not only if I try creating checks - it is generally if I use sensuctl.

Do I have to reinitialize sensuctl? If yes: How?

Thanks for any help or suggestions.

Dirk

1 Like

By the way: If I run “sensuctl configure” again, neither the password I set successfully 6 months ago nor the default password of a fresh Sensu Go install work.

Hi @dirkhschulz have you updated the cluster in the last 6 months? If not, there have been a number of changes. One of the biggest being that the backend has to go through an init process (see https://docs.sensu.io/sensu-go/latest/installation/install-sensu/#3-initialize). I’m not quite sure if that’s the case of you’re updating from a pre-init version, but it’s worth noting. I’d also suggest starting the backend in debug and posting the logs when you’re trying to auth.

Hi Aaron,
thanks for your fast answer. I will try both and report back - maybe tomorrow.
Cheers,
Dirk

Hi Aaron,
this is what I got back:

sensu-backend init

{“component”:“backend.seeds”,“level”:“info”,“msg”:“store already initialized”,“time”:“2020-05-27T18:05:37+02:00”}
And if I then try to create a check I get the same error:
Error: failed to request new refresh token; client returned ‘Post https://mysensuhostname:8080/auth/token: Forbidden’
Is it possible to reset the initialization state somehow?

Dirk

This is what I find about authentication in the logs:
May 27 18:13:06 sensubackendhost sensu-backend: {“component”:“auth-providers”,“level”:“error”,“msg”:“the authentication providers watcher is no longer running”,“time”:“2020-05-27T18:13:06+02:00”}
May 27 18:13:08 sensubackendhost sensu-backend: {“component”:“auth-providers”,“level”:“info”,“msg”:“starting the authentication providers watcher”,“time”:“2020-05-27T18:13:08+02:00”}
May 27 18:13:08 sensubackendhost sensu-backend: {“component”:“auth-providers”,“level”:“info”,“msg”:“no authentication provider found”,“time”:“2020-05-27T18:13:08+02:00”}
May 27 18:13:08 sensubackendhost sensu-backend: {“component”:“store”,“key”:"/sensu.io/api/enterprise/authentication/v2/authproviders/",“level”:“debug”,“msg”:“starting a watcher”,“time”:“2020-05-27T18:13:08+02:00”}

To me that sounds the previously working authentication provider has vanished.
Did I have to reconfigure authentication providers after updating sensu-backend in the last months?

Dirk