I have tried looking at my sensu logs but I haven’t found anything useful yet.
I am running:
- Debian 10
- Sensu 6.7.1
- Managed by Puppet module
Occasionally (it happens daily but not every check) I get a warning alert for a check that failed to run:
“Check failed to run: No live threads left. Deadlock?”
It only happens on checks using the check-dns.rb handler. All of the DNS remote checks are configured to run every 300 seconds. I get around 1-3 failures like this a day.
Here is an example of the full description I get on the Warning:
Check failed to run: No live threads left. Deadlock?
1 threads, 1 sleeps current:0x0000558a483dd870 main thread:0x0000558a478d11a0
* #<Thread:0x0000558a47905ea8 sleep_forever>
rb_thread_t:0x0000558a478d11a0 native:0x00007f4791b6e740 int:0
/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/dnsruby-1.61.9/lib/dnsruby/resolver.rb:250:in `pop'
/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/dnsruby-1.61.9/lib/dnsruby/resolver.rb:250:in `send_message'
/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/dnsruby-1.61.9/lib/dnsruby/resolver.rb:200:in `query'
/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-dns-3.0.0/bin/check-dns.rb:134:in `resolve_domain'
/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-dns-3.0.0/bin/check-dns.rb:238:in `block in run'
/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-dns-3.0.0/bin/check-dns.rb:237:in `each'
/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-dns-3.0.0/bin/check-dns.rb:237:in `run'
/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugin-4.0.0/lib/sensu-plugin/cli.rb:59:in `block in <class:CLI>'
, ["/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/dnsruby-1.61.9/lib/dnsruby/resolver.rb:250:in pop'", "/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/dnsruby-1.61.9/lib/dnsruby/resolver.rb:250:in send_message'", "/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/dnsruby-1.61.9/lib/dnsruby/resolver.rb:200:in query'", "/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-dns-3.0.0/bin/check-dns.rb:134:in resolve_domain'", "/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-dns-3.0.0/bin/check-dns.rb:238:in block in run'", "/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-dns-3.0.0/bin/check-dns.rb:237:in each'", "/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-dns-3.0.0/bin/check-dns.rb:237:in run'", "/opt/sensu-plugins-ruby/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugin-4.0.0/lib/sensu-plugin/cli.rb:59:in block in <class:CLI>'"]
Any ideas as to what might be causing this? Or ways to troubleshoot?