Check command changes not picked up by agents

Sensu Go
1

I raised the issue on github but was wondering whether someone has encountered a similar issue:

To summarise, we make a change on the backend to an existing check like updating the command that’s being executed. The agent is not picking up this change despite scheduling an ad-hoc run of that check and it is not until a period of time has elapsed (I’ve noticed 24hrs) until the check configuration associated to that event is actually updated to reflect the changes made to the check.

2

Hello @itstreet,

I see in the github issue that you are running 5.10.1. I just tried this on 5.11.0 and I am unable to reproduce the issue. The command output updates for all 5 of my entities after changing the check command and running an ad-hoc request.

Are you able to upgrade to at least 5.11.0 to see if that helps? That release includes changes to a backend component that we believe is the issue here.

https://docs.sensu.io/sensu-go/5.11/release-notes/#5-11-0-release-notes

We also have a patch release out too, but nothing in that release would help with this specific issue:

https://docs.sensu.io/sensu-go/5.11/release-notes/#5-11-1-release-notes

Once upgraded to at least 5.11.0, let us know if the issue remains. If it does:

How are you changing the check command? Through sensuctl?
Does it seem to always happen and for all entities subscribed to that check?
Does restarting the backend process resolve it?

Regards,
Richard.

1 Like
3

@itstreet :wave: I’ll add an update to what @richard mentioned here. If upgrading isn’t currently an option, the known workaround is to just restart the backend(s), which refreshes the watcher cache (currently what we believe to be the issue when this behavior is observed).

2 Likes
4

Hi everyone!

Many thanks for the quick response, I can confirm that this issue has gone away on the upgrade to 5.11.1 of the Sensu Backends. It looks like we did indeed encounter one of the noted bugs that was with high cpu utilization as our cluster was running way above normal levels of expected CPU utilization.

Many thanks for your help with this!

We can mark this as closed!

1 Like
5

Thanks for following up and letting us know @itstreet :+1: