Sensu plugin (cpu monitoring) configure workflow - standalone checks

puneet336 · August 20, 2021, 1:46pm

I order to understand the sensu software’s workflow, i am trying to enable a basic cpu monitoring plugin. Objective is to setup standalone checks in a way that i can see the current cpu utilization metric from each sensu client on dashboard.
I have created 2 machines -
hostname: docker1 - planning t use this as sensu server
hostname: docker2 - planning to use this as sensu client.

[root@docker1 ~]# sensuctl entity list
         ID          Class    OS                Subscriptions                         Last Seen
─────────────────── ─────── ─────── ────────────────────────────────────── ────────────────────────────────
  docker2.self.com   agent   linux   system/linux,entity:docker2.self.com   2021-08-20 19:02:15 +0530 IST
[root@docker1 ~]#

the server is able to see the sensu client. Here’s the namespace info -

[root@docker1 ~]# sensuctl config view
=== Active Configuration
API URL:                  http://127.0.0.1:8080
Namespace:                default
Format:                   tabular
Timeout:                  15s
Username:                 sensu
JWT Expiration Timestamp: 1629465301

i attemplted to familiarize my self with the monitoring aspect of the sensu plugins by enabling a cpu monitoring plugin. I followed the information listed in the documentation of the “cpu monitoring plugin” - check-cpu-usage versions

[root@docker1 ~]# sensuctl asset add sensu/check-cpu-usage
no version specified, using latest: 0.2.2
fetching bonsai asset: sensu/check-cpu-usage:0.2.2
added asset: sensu/check-cpu-usage:0.2.2
You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["sensu/check-cpu-usage"].

on this stdout from the sensu asset add -

To add this runtime asset to the appropriate resource, populate the “runtime_assets” field with [“sensu/check-cpu-usage”].

i was unable to find this(runtime_assets) section in the /etc/sensu/backend.yml file on the sensu server (docker1). So i am unsure where & how to configure this.

here’s the list of assets -

[root@docker1 ~]# sensuctl asset list
                       Name                                                                     URL                                                 Hash
────────────────────────────────────────────────── ────────────────────────────────────────────────────────────────────────────────────────────── ──────────
  sensu-plugins/sensu-plugins-cpu-checks            //assets.bonsai.sensu.io/.../sensu-plugins-cpu-checks_4.1.0_debian9_linux_amd64.tar.gz         9dd717a
  sensu-plugins/sensu-plugins-cpu-checks            //assets.bonsai.sensu.io/.../sensu-plugins-cpu-checks_4.1.0_debian_linux_amd64.tar.gz          f0435fd
  sensu-plugins/sensu-plugins-cpu-checks            //assets.bonsai.sensu.io/.../sensu-plugins-cpu-checks_4.1.0_centos7_linux_amd64.tar.gz         8a01862
  sensu-plugins/sensu-plugins-cpu-checks            //assets.bonsai.sensu.io/.../sensu-plugins-cpu-checks_4.1.0_centos6_linux_amd64.tar.gz         f42be79
  sensu-plugins/sensu-plugins-cpu-checks            //assets.bonsai.sensu.io/.../sensu-plugins-cpu-checks_4.1.0_alpine3.8_linux_amd64.tar.gz       7a5ad2d
  sensu-plugins/sensu-plugins-cpu-checks            //assets.bonsai.sensu.io/.../sensu-plugins-cpu-checks_4.1.0_alpine_linux_amd64.tar.gz          a67676f
  sensu-plugins/sensu-plugins-cpu-checks:4.1.0      //assets.bonsai.sensu.io/.../sensu-plugins-cpu-checks_4.1.0_debian9_linux_amd64.tar.gz         9dd717a
  sensu-plugins/sensu-plugins-cpu-checks:4.1.0      //assets.bonsai.sensu.io/.../sensu-plugins-cpu-checks_4.1.0_debian_linux_amd64.tar.gz          f0435fd
  sensu-plugins/sensu-plugins-cpu-checks:4.1.0      //assets.bonsai.sensu.io/.../sensu-plugins-cpu-checks_4.1.0_centos7_linux_amd64.tar.gz         8a01862
  sensu-plugins/sensu-plugins-cpu-checks:4.1.0      //assets.bonsai.sensu.io/.../sensu-plugins-cpu-checks_4.1.0_centos6_linux_amd64.tar.gz         f42be79
  sensu-plugins/sensu-plugins-cpu-checks:4.1.0      //assets.bonsai.sensu.io/.../sensu-plugins-cpu-checks_4.1.0_alpine3.8_linux_amd64.tar.gz       7a5ad2d
  sensu-plugins/sensu-plugins-cpu-checks:4.1.0      //assets.bonsai.sensu.io/.../sensu-plugins-cpu-checks_4.1.0_alpine_linux_amd64.tar.gz          a67676f
  sensu-plugins/sensu-plugins-memory-checks:4.1.1   //assets.bonsai.sensu.io/.../sensu-plugins-memory-checks_4.1.1_debian9_linux_amd64.tar.gz      45ed66c
  sensu-plugins/sensu-plugins-memory-checks:4.1.1   //assets.bonsai.sensu.io/.../sensu-plugins-memory-checks_4.1.1_debian_linux_amd64.tar.gz       a1dff4b
  sensu-plugins/sensu-plugins-memory-checks:4.1.1   //assets.bonsai.sensu.io/.../sensu-plugins-memory-checks_4.1.1_centos7_linux_amd64.tar.gz      aca2e38
  sensu-plugins/sensu-plugins-memory-checks:4.1.1   //assets.bonsai.sensu.io/.../sensu-plugins-memory-checks_4.1.1_centos6_linux_amd64.tar.gz      aae3a9f
  sensu-plugins/sensu-plugins-memory-checks:4.1.1   //assets.bonsai.sensu.io/.../sensu-plugins-memory-checks_4.1.1_alpine3.8_linux_amd64.tar.gz    38a9e03
  sensu-plugins/sensu-plugins-memory-checks:4.1.1   //assets.bonsai.sensu.io/.../sensu-plugins-memory-checks_4.1.1_alpine_linux_amd64.tar.gz       217b083
  sensu/check-cpu-usage                             //assets.bonsai.sensu.io/.../check-cpu-usage_0.2.2_windows_amd64.tar.gz                        900cfdf
  sensu/check-cpu-usage                             //assets.bonsai.sensu.io/.../check-cpu-usage_0.2.2_darwin_amd64.tar.gz                         db81ee7
  sensu/check-cpu-usage                             //assets.bonsai.sensu.io/.../check-cpu-usage_0.2.2_linux_armv7.tar.gz                          400aacc
  sensu/check-cpu-usage                             //assets.bonsai.sensu.io/.../check-cpu-usage_0.2.2_linux_arm64.tar.gz                          bef7802
  sensu/check-cpu-usage                             //assets.bonsai.sensu.io/.../check-cpu-usage_0.2.2_linux_386.tar.gz                            a2dcb53
  sensu/check-cpu-usage                             //assets.bonsai.sensu.io/.../check-cpu-usage_0.2.2_linux_amd64.tar.gz                          2453973
  sensu/sensu-ruby-runtime:0.0.10                   //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.0.10_ruby-2.4.4_centos6_linux_amd64.tar.gz   cbee191
  sensu/sensu-ruby-runtime:0.0.10                   //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.0.10_ruby-2.4.4_debian_linux_amd64.tar.gz    a28952f
  sensu/sensu-ruby-runtime:0.0.10                   //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.0.10_ruby-2.4.4_centos_linux_amd64.tar.gz    338b88b
  sensu/sensu-ruby-runtime:0.0.10                   //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.0.10_ruby-2.4.4_alpine_linux_amd64.tar.gz    8d768d1
[root@docker1 ~]#

here’s the view of the dashboard -

from this point, i am not sure what additional settings/config changes do i need to make on sensu servers and client to get this plugin to work.
Could you please point me to a web url which has complete workflow of deploying a simple plugin like this cpu monitoring/disk check etc.?

jspaleta · August 21, 2021, 2:03am

Hey,
The backend configuration file is just enough configuration to get your sensu-backend instance up and running. For Sensu Go (compared to the previous generation of Ruby based Sensu) you no longer define your monitoring workloads in the /etc/sensu/ configuration files. Those configs are now just enough configuration so the sensu backend cluster can be stood up to the etcd cluster that holds the sensu monitoring resources. It’s the etcd cluster that holds the monitoring resources.

So in this case, what you probably want to do is reference the asset in a check resource definition and then use sensuctl (or some other api client like curl or the sensu dashboard web-ui) to add/update the check definition in the running Sensu state.

The repository readme you reference has an example check definition yaml file that you can use with sensuctl create -f

puneet336 · August 21, 2021, 6:13pm

Thank you for the response, i got invalid resource issue while attempting to reference the asset using the sensuctl create command. here’s the command sequence i followed -

on sensu agent i tried subscribing to the “system” subscription as -
[root@docker3 ~]# sudo -u sensu sensu-agent start --backend-url ws://10.5.5.1:8081 --deregister true --subscriptions system
on the backend server -


[root@docker2 ~]#  sensuctl asset add sensu/check-cpu-usage
[root@docker2 ~]# sensuctl asset list
          Name                                              URL                                      Hash
──────────────────────── ───────────────────────────────────────────────────────────────────────── ──────────
  sensu/check-cpu-usage   //assets.bonsai.sensu.io/.../check-cpu-usage_0.2.2_windows_amd64.tar.gz   900cfdf
  sensu/check-cpu-usage   //assets.bonsai.sensu.io/.../check-cpu-usage_0.2.2_darwin_amd64.tar.gz    db81ee7
  sensu/check-cpu-usage   //assets.bonsai.sensu.io/.../check-cpu-usage_0.2.2_linux_armv7.tar.gz     400aacc
  sensu/check-cpu-usage   //assets.bonsai.sensu.io/.../check-cpu-usage_0.2.2_linux_arm64.tar.gz     bef7802
  sensu/check-cpu-usage   //assets.bonsai.sensu.io/.../check-cpu-usage_0.2.2_linux_386.tar.gz       a2dcb53
  sensu/check-cpu-usage   //assets.bonsai.sensu.io/.../check-cpu-usage_0.2.2_linux_amd64.tar.gz     2453973
[root@docker2 ~]# sensuctl entity list
        ID          Class    OS             Subscriptions                     Last Seen
────────────────── ─────── ─────── ─────────────────────────────── ────────────────────────────────
  docker3.hpc.com   agent   linux   system,entity:docker3.hpc.com   2021-08-21 23:31:06 +0530 IST
[root@docker2 ~]#
[root@docker2 ~]# cat cpumon.yaml
---
type: CheckConfig
api_version: core/v2
metadata:
  name: check-cpu-usage
  namespace: default
spec:
  command: >-
    check-cpu-usage
    --critical 95
    --warning 85
    --sample-interval 2
  output_metric_format: nagios_perfdata
  output_metric_handlers:
    - influxdb
  subscriptions:
  - system
  runtime_assets:
  - sensu/check-cpu-usage
[root@docker2 ~]#  sensuctl create -f cpumon.yaml
Error: error putting resource #0 with name "check-cpu-usage" and namespace "default" (/api/core/v2/namespaces/default/checks/check-cpu-usage): resource is invalid: check interval must be greater than 0 or a valid cron schedule must be provided

did i miss out on something here ?

puneet336 · August 21, 2021, 6:55pm

update :
I updated the yaml file by adding interval, timeout -

type: CheckConfig
api_version: core/v2
metadata:
name: check-cpu-usage
namespace: default
spec:
command: >-
check-cpu-usage
–critical 95
–warning 85
–sample-interval 2
output_metric_format: nagios_perfdata
output_metric_handlers:
- influxdb
publish: true
interval: 30
timeout: 10
subscriptions:

system
runtime_assets:
sensu/check-cpu-usage

and cpu monitoring asset seems to be working fine now

Also, for the architecture overview i was going through this video and i understand now that the information is deprecated. just for confirmation - the sensu go still uses rabbit mq ?
and redis database has been replaced with etcd ?

Also, i came to know that in earlier version of sensu, the client were able to run jobs themselves (standalone checks ) and servers were also able to schedule (like nagios) jobs on client.
The standalone checks are now deprecated, des this mean that now the server-backend is responsible for scheduling monitoring like nagios? - to be specific , the steps i am following above means that server is going to schedule the jobs ?

jspaleta · August 23, 2021, 5:58pm

Yes for Sensu Go, the backend schedule checks.

Agents keep a persistent websocket connection open with a backend.

Topic		Replies	Views
Monitoring Docker Containers Status Sensu Plugins	0	251	November 6, 2021
Sensu Sandbox Set-up Getting Started	1	512	September 16, 2019
Kubernetes monitoring issue with Sensu (sensu-plugins/sensu-plugins-kubernetes) Sensu Go	3	276	March 15, 2023
Few basic questions Sensu Go sensu-go , assets , sensuctl	3	473	May 4, 2021
Sensu plugins-Please help me! Sensu Plugins	5	400	March 10, 2021

Sensu plugin (cpu monitoring) configure workflow - standalone checks

update : I updated the yaml file by adding interval, timeout -

Related Topics

update :
I updated the yaml file by adding interval, timeout -