I have a problem with sensu configuration in a chef environment. We would like to use the default cookbook and define the different checks with the LWRP’s on the cookbook. The checks are defined in the same cookbooks as the resources they are checking. For example the check that process X is running is defined in teh X cookbook while the checks for Y are defined in the Y cookbook.
The problem with this approach is that the checks only get defined on the clients since the machine running the sensu-server isn’t running all these process cookbooks. So we have to define all the checks as standalone to get them to run. But this feels wrong.
My question now is if there is a better way to configure the system? Are there any best practices we have missed?
I’ve been experimenting with Chef cookbook patterns for Sensu, you can find my experiments at https://github.com/portertech/chef-monitor The README is currently outdated, so you will have to do a little spelunking to see what I have done.
···
On Mon, Mar 17, 2014 at 2:57 AM, Martin Forssen maf@recordedfuture.com wrote:
I have a problem with sensu configuration in a chef environment. We would like to use the default cookbook and define the different checks with the LWRP’s on the cookbook. The checks are defined in the same cookbooks as the resources they are checking. For example the check that process X is running is defined in teh X cookbook while the checks for Y are defined in the Y cookbook.
The problem with this approach is that the checks only get defined on the clients since the machine running the sensu-server isn’t running all these process cookbooks. So we have to define all the checks as standalone to get them to run. But this feels wrong.
My question now is if there is a better way to configure the system? Are there any best practices we have missed?
In my experience, you’re indeed better off defining most of your checks sensu-server-side (ahem) rather than on all nodes, as standalone checks. So your nodes only install check scripts and subscribe to channels.
And yes, it does mean you’ll need to run chef on your sensu server every time you modify/add check definitions.
A gotcha with this approach: checks based on newly added check scripts will fail until you’ve run Chef on all monitored nodes as well, because the new check script needs to be uploaded there for them to work.
I prefer to do it that way though, because check modifications that only change check parameters or frequency are near-instantaneous after the Chef run on your sensu server.
HTH
Mat
···
On Mon, Mar 17, 2014 at 3:51 PM, Sean Porter portertech@gmail.com wrote:
I’ve been experimenting with Chef cookbook patterns for Sensu, you can find my experiments at https://github.com/portertech/chef-monitor The README is currently outdated, so you will have to do a little spelunking to see what I have done.
On Mon, Mar 17, 2014 at 2:57 AM, Martin Forssen maf@recordedfuture.com wrote:
I have a problem with sensu configuration in a chef environment. We would like to use the default cookbook and define the different checks with the LWRP’s on the cookbook. The checks are defined in the same cookbooks as the resources they are checking. For example the check that process X is running is defined in teh X cookbook while the checks for Y are defined in the Y cookbook.
The problem with this approach is that the checks only get defined on the clients since the machine running the sensu-server isn’t running all these process cookbooks. So we have to define all the checks as standalone to get them to run. But this feels wrong.
My question now is if there is a better way to configure the system? Are there any best practices we have missed?
I looked at the chef-monitor cookbook. As far as I can see it assumes that all checks are defined in a data bag.
So it seems as if there is no good solution which lets you define the checks only in the cookbooks run on the client machines. At last not if you want to avoid standalone checks.