Hi,
We have 3 sensu servers running in our organisation for monitoring and autoremediation and somehow remediation framework stopped working after migrating the VMs from one ESXI to other however, ips and other policies remains intact.
Setup is as follows:
Sensu-client(send Alerts)----Sensu-Server—Autoremedy-handler(send alerts to sqs)—Alerts pulled by in house mahince which runs the remediation.
Post migration, alerts are reaching sensu server but we are not getting any messges in SQS and even none of other handlers like email, pagerduty, slack are working. upon checking sensu logs found below erros:
sensu-api.log:
{“timestamp”:“2021-06-29T13:56:13.568992+0000”,“level”:“error”,“message”:“unexpected api error”,“error”:“unable to get peer name: Transport endpoint is not connected”,“backtrace”:"/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.2/lib/em/connection.rb:597:in get_peername'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.2/lib/em/connection.rb:597:in get_peername’\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.0.0/lib/sensu/api/http_handler.rb:25:in request_details'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.0.0/lib/sensu/api/http_handler.rb:43:in log_request’\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.0.0/lib/sensu/api/http_handler.rb:392:in process_http_request'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/em-http-server-0.1.8/lib/em-http-server/server.rb:30:in receive_request’\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.2/lib/em/protocols/header_and_content.rb:99:in dispatch_request'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.2/lib/em/protocols/header_and_content.rb:73:in receive_line’\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.2/lib/em/protocols/linetext2.rb:73:in receive_data'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.2/lib/eventmachine.rb:194:in run_machine’\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/eventmachine-1.2.2/lib/eventmachine.rb:194:in run'\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.0.0/lib/sensu/api/process.rb:17:in run’\n/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-1.0.0/exe/sensu-api:10:in <top (required)>'\n/opt/sensu/bin/sensu-api:22:in load’\n/opt/sensu/bin/sensu-api:22:in `’"}
{“timestamp”:“2021-06-29T13:57:52.673939+0000”,“level”:“warn”,“message”:“transport connection error”,“reason”:“tcp connection lost”}
{“timestamp”:“2021-06-29T13:57:52.674208+0000”,“level”:“warn”,“message”:“reconnecting to transport”}
{“timestamp”:“2021-06-29T13:57:54.675710+0000”,“level”:“error”,“message”:"[amqp] Detected TCP connection failure: Errno::ECONNREFUSED"}
{“timestamp”:“2021-06-29T13:57:58.675630+0000”,“level”:“error”,“message”:"[amqp] Detected TCP connection failure: Errno::ECONNREFUSED"}
sensu-client.log
error":“Transport is not connected”}
It looks like a tcp issue but we have all ports opened and it shouldn’t be blocked anywhere. Is this possible communication between sensu servers are not working ? can this cause this errors ?
If yes, any idea how we can fix this ?
We have currently three sensu servers and I am not pretty sure how they are communicating with each other but there is haproxy installed on them.
I am pretty new to sensu so any help will be appreciated
Thanks,
Yogesh