Occasionally we receive a “Retirement Notification” from AWS informing us that an EC2 instance will be retired due to degradation of the underlying hardware hosting the instanced that the instance could already be unreachable. We often notice that the instance is already unreachable and the client has been removed from Sensu, when what we would have really wanted to happen is for the client keep-alive to throw a critical event and page out.
Why does Sensu delete the client? I noticed the EC2 attributes specification page (https://sensuapp.org/docs/latest/reference/clients.html#ec2-attributes-specification) lists various EC2 attributes, one of which is “allowed_instance_states”. We do not use this today but wondering if it applies to the CORE Sensu version which is what we use, or whether there is another way to resolve this.