Without details as to what you are trying to do…
From the log it appears that the Sensu entity resource in host.example.org in the default namespace does not exist, which is why you got a 404 error.
Should that resource exist? I don’t know, I’d have to see your puppet configuration to understand what is expected.
My best guess here is you have used sensu_agent_entity_config from the puppet module prior to creating the agent_entity in puppet. The sensu_agent_entity_config will attempt to validate the entity resource exists before it tries to change it based on what I’m seeing in the puppet module code.
But that’s a best guess based on the logic flow in the module source code, without seeing your puppet config I can’t provide anything better than a best guess.
If you are using a sensu_agent_entity_config definition for host.example.com then you should preface that with a definition of sensu_entity for host.example.com that explicit creates an entity resource and marks it as an agent classed entity.
This might be a race condition when installing the agent and backend on same host during same Puppet run. The backend has to be fully functional for the sensu_agent_entity_config to be able to make API calls for the agent code. This should be a non-issue unless something is unhealthy with the backend because there is a validator that tries to query the entity several times since it’s not guaranteed that the agent entity will exist right away via API when sensu-agent starts.
Are you experiencing any errors that indicate the environment is broken after Puppet runs? Those log messages about failed to validate could just be part of what happens when agent + backend are first installed on the same host.
I just installed sensu backend in the first step (no error) and tried to install the agent on the same host again.
This causes the following error:
]# puppet agent --test
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for host.example.org
Info: Applying configuration version '1622106608'
Notice: /Stage[main]/Sensu::Agent/File[sensu_agent_config]/ensure: defined content as '{md5}1c918e0c8a52c38a4332f7c2a32fb530'
Info: /Stage[main]/Sensu::Agent/File[sensu_agent_config]: Scheduling refresh of Service[sensu-agent]
Notice: /Stage[main]/Sensu::Agent/Service[sensu-agent]: Triggered 'refresh' from 1 event
Notice: Failed to connect to validate entity host.example.org; sleeping 2 seconds before retry
Notice: Failed to connect to validate entity host.example.org; sleeping 2 seconds before retry
Notice: Failed to connect to validate entity host.example.org; sleeping 2 seconds before retry
Notice: Failed to connect to validate entity host.example.org; sleeping 2 seconds before retry
Notice: Failed to connect to validate entity host.example.org; sleeping 2 seconds before retry
Notice: Failed to connect validate entity host.example.org within timeout window of 10 seconds; giving up.
Notice: /Stage[main]/Sensu::Backend::Default_resources/Sensu_namespace[default]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Resources/Sensu_check[check-cpu]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend::Default_resources/Sensu_cluster_role[admin]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend::Default_resources/Sensu_cluster_role[cluster-admin]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend::Default_resources/Sensu_cluster_role[edit]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend::Default_resources/Sensu_cluster_role[system:agent]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend::Default_resources/Sensu_cluster_role[system:user]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend::Default_resources/Sensu_cluster_role[view]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend::Default_resources/Sensu_cluster_role_binding[cluster-admin]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend::Default_resources/Sensu_cluster_role_binding[system:user]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend::Agent_resources/Sensu_cluster_role[puppet:agent_entity_config]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend::Agent_resources/Sensu_user[puppet-agent_entity_config]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend::Agent_resources/Sensu_cluster_role_binding[puppet:agent_entity_config]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend/Sensu_user[agent]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend::Default_resources/Sensu_cluster_role_binding[system:agent]/ensure: created (corrective)
Error: Unable to query entity data for entity host.example.org: Unable to make API request at https://10.11.48.123:8080/api/core/v2/namespaces/default/entities/host.example.org: Net::HTTPUnauthorized
Error: /Stage[main]/Sensu::Agent/Sensu::Agent::Subscription[linux]/Sensu_agent_entity_config[sensu::agent::subscription linux]/ensure: change from 'absent' to 'present' failed: Unable to query entity data for entity host.example.org: Unable to make API request at https://10.11.48.123:8080/api/core/v2/namespaces/default/entities/host.example.org: Net::HTTPUnauthorized
Info: Class[Sensu::Agent]: Unscheduling all events on Class[Sensu::Agent]
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 40.64 seconds
Based on that log output it looks like that was a run that was either backend for first time or first time with backend having include_agent_resources => true because it looks like the agent was attempting to be applied before the backend had setup the necessary RBAC resources for the agent Puppet code to query entities via API.
The agent uses this to talk to API via Puppet code:
Notice: /Stage[main]/Sensu::Backend::Agent_resources/Sensu_cluster_role[puppet:agent_entity_config]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend::Agent_resources/Sensu_user[puppet-agent_entity_config]/ensure: created (corrective)
Notice: /Stage[main]/Sensu::Backend::Agent_resources/Sensu_cluster_role_binding[puppet:agent_entity_config]/ensure: created (corrective)
Those resources got called after the agent validation was run, so this appears to just be an ordering issue since the Puppet module doesn’t force a certain order of agent and backend. One possible solution is in your backend profile class do this:
Class['sensu::backend'] -> Class['sensu::agent']
If the above causes resource dependency cycles then you might try something a bit more targetted:
Using the collector instead of targeted resource name was just because it’s less static what the validator is called as it goes by agent entity name you define for agent config.
As I wanted to use foreman without any additional custom module, I have to think about this for a while. I guess I can not solve the ordering-problem without writing a custom module in this case.
I just mentioned that I can not log in using sensuctl because of this error:
Error: unable to authenticate with error: Get https://10.11.48.123:8080/auth: x509: cannot validate certificate for 10.11.48.123 because it doesn’t contain any IP SANs
The SSL issue with sensuctl is because you set api_host as the IP address and that IP address is not in the certificate you are using. The default behavior for the Puppet module is to use the Puppet certificates and Puppet does not add IP SAN to the certificates it generates. You will likely want to change sensu::api_host to be the FQDN of the backend server, same goes for sensu::agent::backends. If you want to use IP addresses and SSL you will have to generate certificates that have the IP SAN added which I don’t think Puppet can do but something you can easily do with openssl.