We’re running Sensu in a highly heterogenous service provider environment. On several client servers, we’ve observed that the sensu-client init script provided with the Sensu RPM fails to exit when starting the client. Running bash -x /etc/init.d/sensu-client start or bash -x /etc/init.d/sensu-service client start shows that it’s hanging on the runuser command invoked by the daemon() function call, and a strace of the process shows that it forks the runuser process then hangs in wait4(), waiting for the child process to exit.
Excerpt from the strace:
read(255, “#!/bin/bash\n\n# chkconfig: 345 90”…, 407) = 407
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, , 8) = 0
rt_sigprocmask(SIG_BLOCK, [INT CHLD], , 8) = 0
lseek(255, -9, SEEK_CUR) = 398
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f120c2269d0) = 21685
rt_sigprocmask(SIG_SETMASK, , NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], , 8) = 0
rt_sigprocmask(SIG_SETMASK, , NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], , 8) = 0
rt_sigaction(SIGINT, {0x43d060, , SA_RESTORER, 0x3398032960}, {SIG_DFL, , SA_RESTORER, 0x3398032960}, 8) = 0
wait4(-1, Starting sensu-client
At this point, the client is running normally, but the init script never exits, which has implications for integration of the client with Puppet. We haven’t been able to find anything that clearly ties together the servers affected by this issue. So far we’ve observed it on RHEL 5.8, 6.2 and 6.4, all running different versions of the initscripts package (which supplies the daemon() function) and coreutils (which provides runuser). All of our servers that have Sensu deployed are running Sensu 0.12.1-13. One thing to note is that we are running a modified version of the Sensu RPM built with Omnibus (https://github.com/Contegix/omnibus-sensu) in a different filesystem path.
Have any of you seen an issue similar to this, or have any ideas as to what might be causing it?