sensu server hangs with "completing work in progress"


#1

Hello,

Our sensu-server process regularly will not restart and we have to kill it manually. This has been happening for awhile, currently we’re on version 0.25.7. The log looks like

{“timestamp”:“2016-08-12T04:13:10.097074+0000”,“level”:“warn”,“message”:“received signal”,“signal”:“TERM”}
{“timestamp”:“2016-08-12T04:13:10.097317+0000”,“level”:“warn”,“message”:“stopping”}
{“timestamp”:“2016-08-12T04:13:10.097399+0000”,“level”:“info”,“message”:“completing work in progress”,“in_progress”:{“check_results”:2,“events”:0}}

I added some extra logging in “complete_in_progress” in the file https://github.com/sensu/sensu/blob/master/lib/sensu/server/process.rb

That seems to be looping forever, with one value in “check_results”:
{“timestamp”:“2016-08-12T04:13:11.611783+0000”,“level”:“warn”,“message”:"{:check_results=>1, :events=>0}\n"}

What should be the next step to debug this issue? Should sensu have a timer to complete_in_progress and eventually abort?

Thank you


#2

You are not alone we have the same problem: