"failed to request new refresh token; client returned 'Post https://hostname:8080/auth/token: Forbidden'

Sensu Go
1

This is somewhat weird because I find nothing at all about it in the docs and nothing relevant on Google.

6 months ago I set up a sensu go cluster, initialized the client, created assets, checks, all the stuff.
Since then I simple let it run and updated it regularly, but did not change config - just added systems / agents.

Now I want to add further checks etc. - and upon trying that I get the following error:

sensuctl check create check-glusterfs --command ‘/opt/sensu-plugins-ruby/added/check_glusterfs’ --interval 300 --subscriptions glusterfs
Error: failed to request new refresh token; client returned ‘Post https://hostname:8080/auth/token: Forbidden’

This is not only if I try creating checks - it is generally if I use sensuctl.

Do I have to reinitialize sensuctl? If yes: How?

Thanks for any help or suggestions.

Dirk

1 Like
2

By the way: If I run “sensuctl configure” again, neither the password I set successfully 6 months ago nor the default password of a fresh Sensu Go install work.

3

Hi @dirkhschulz have you updated the cluster in the last 6 months? If not, there have been a number of changes. One of the biggest being that the backend has to go through an init process (see https://docs.sensu.io/sensu-go/latest/installation/install-sensu/#3-initialize). I’m not quite sure if that’s the case of you’re updating from a pre-init version, but it’s worth noting. I’d also suggest starting the backend in debug and posting the logs when you’re trying to auth.

4

Hi Aaron,
thanks for your fast answer. I will try both and report back - maybe tomorrow.
Cheers,
Dirk

5

Hi Aaron,
this is what I got back:

sensu-backend init

{“component”:“backend.seeds”,“level”:“info”,“msg”:“store already initialized”,“time”:“2020-05-27T18:05:37+02:00”}
And if I then try to create a check I get the same error:
Error: failed to request new refresh token; client returned ‘Post https://mysensuhostname:8080/auth/token: Forbidden’
Is it possible to reset the initialization state somehow?

Dirk

6

This is what I find about authentication in the logs:
May 27 18:13:06 sensubackendhost sensu-backend: {“component”:“auth-providers”,“level”:“error”,“msg”:“the authentication providers watcher is no longer running”,“time”:“2020-05-27T18:13:06+02:00”}
May 27 18:13:08 sensubackendhost sensu-backend: {“component”:“auth-providers”,“level”:“info”,“msg”:“starting the authentication providers watcher”,“time”:“2020-05-27T18:13:08+02:00”}
May 27 18:13:08 sensubackendhost sensu-backend: {“component”:“auth-providers”,“level”:“info”,“msg”:“no authentication provider found”,“time”:“2020-05-27T18:13:08+02:00”}
May 27 18:13:08 sensubackendhost sensu-backend: {“component”:“store”,“key”:"/sensu.io/api/enterprise/authentication/v2/authproviders/",“level”:“debug”,“msg”:“starting a watcher”,“time”:“2020-05-27T18:13:08+02:00”}

To me that sounds the previously working authentication provider has vanished.
Did I have to reconfigure authentication providers after updating sensu-backend in the last months?

Dirk

7

@dirkhschulz that’s strange that the you’re seeing that message. It’s not something I would expect to happen. The configuration should persist between version upgrades. If you reconfigure sensuctl, is it still not working?

8

Hi Aaron,

if I try to reconfigure sensuctl I receive the exact same error.

Thanks for your answer!

Dirk

Dirk H. Schulz
SchulzConsult
Wiesenweg 12, 85567 Grafing
Tel. +49-8092-862568
Fax +49-8092-862572

9

@dirkhschulz this is strange indeed. The authprovider messages in the logs aren’t indicative of an issue, unless you were using LDAP at some point. What does sensuctl config view give you ? Is it using the URL/username you expect?

10

Hi Aaron,

it is using the correct URL and username.

Since I can log into the web GUI with the same credentials, I guess somehow the permission to login via sensuctl in general has been disconfigured - is this possible?

Cheers,

Dirk

SchulzConsult
Wiesenweg 12, 85567 Grafing

11

Hi @dirkhschulz,

It’s quite strange that sensuctl seems to have just lost its configuration. In general, configuration should persist across upgrades. It’s stranger still to me that the UI works just fine, whereas sensuctl is failing.

Are you able to try from a different system? Ideally we’d be able to narrow this down a bit further to see if it’s something local to the system you’re on.

12

@dirkhschulz
This is odd.

Just thinking out loud a bit… maybe very old sensuctl version relative to the sensu-backend version or vice versa?

This is definitely weird. I’ve yet to experience this myself, but I’m usually using sensuctl atmost 1 release different from sensu-backend as I update them nearly at the same time.