[Issue] Sensu Checks: Multiple check results to every command execution

Hi All,

We are experiencing an issue where Sensu is forwarding 4 logs/ results every execution of a check (check-memory.rb). We are using Sensu Enterprise and all events are sent over to ELK stack using TCP and we’re not sure if this contributed to the result or we might have missed a setting or parameter in our configuration. Hope you guys can help. Thanks!

  1. In our client side, the following is the check configuration

/etc/sensu/conf.d/check.json

{

“checks”: {

“disk_check”: {

“command”: “check-disk-usage.rb -w 80 -c 90 -p /sys/kernel/debug/tracing -x proc”,

“interval”: 600,

“standalone”: true,

“handler”: “event_stream”,

“subscribers”: [ “ALL” ]

},

“check_memory”: {

“command”: “/opt/sensu/embedded/bin/check-memory.rb”,

  "interval": 60,

“standalone”: true,

“handlers”: [“event_stream”, “email”],

“subscribers”: [“memory_usage”],

“type”: “metric”

},

“check_memory_percentage”: {

“command”: “/opt/sensu/embedded/bin/metrics-memory-percent.rb”,

“interval”: 900,

“standalone”: true,

“handler”: “event_stream”,

“subscribers”: [ “ALL” ]

},

“check_cpu”: {

“command”: “/opt/sensu/embedded/bin/metrics-cpu-mpstat.rb”,

“interval”: 900,

“standalone”: true,

“handler”: “event_stream”,

“subscribers”: [ “ALL” ]

}

}

}

/etc/sensu/conf.d/client.json

{

“client”: {

“name”: “Client”,

“address”: “127.0.0.1”,

“subscriptions”: [

“memory_usage”]

}

}

  1.  In the server side, the following is our handler ( we are using Sensu Enterprise)
    
    **/etc/sensu/conf.d/eventstream.json**
    

{

“event_stream”: {

“host”: “127.0.0.1”,

“port”: 5514,

“filter_metrics”: false

}

}

  1. Attached is the screenshot of our monitoring application (Kibana). There are 4 check results/logs to every single execution of check-memory.rb (given they have same timestamp, command, etc.) - and this runs every 60 seconds which we intentionally set for testing purposes.
    Please advise. Thank you so much in advance!

Regards,

Keeshia

If i am not wrong. In sensu enterprise , we need to configure event_stream as bridge not as handler.

···

On Wednesday, April 12, 2017 at 9:55:00 AM UTC+8, Kaye wrote:

Hi All,

We are experiencing an issue where Sensu is forwarding 4 logs/ results every execution of a check (check-memory.rb). We are using Sensu Enterprise and all events are sent over to ELK stack using TCP and we’re not sure if this contributed to the result or we might have missed a setting or parameter in our configuration. Hope you guys can help. Thanks!

  1. In our client side, the following is the check configuration
    /etc/sensu/conf.d/check.json

{

“checks”: {

“disk_check”: {

“command”: “check-disk-usage.rb -w 80 -c 90 -p /sys/kernel/debug/tracing -x proc”,

“interval”: 600,

“standalone”: true,

“handler”: “event_stream”,

“subscribers”: [ “ALL” ]

},

“check_memory”: {

“command”: “/opt/sensu/embedded/bin/check-memory.rb”,

  "interval": 60,

“standalone”: true,

“handlers”: [“event_stream”, “email”],

“subscribers”: [“memory_usage”],

“type”: “metric”

},

“check_memory_percentage”: {

“command”: “/opt/sensu/embedded/bin/metrics-memory-percent.rb”,

“interval”: 900,

“standalone”: true,

“handler”: “event_stream”,

“subscribers”: [ “ALL” ]

},

“check_cpu”: {

“command”: “/opt/sensu/embedded/bin/metrics-cpu-mpstat.rb”,

“interval”: 900,

“standalone”: true,

“handler”: “event_stream”,

“subscribers”: [ “ALL” ]

}

}

}

/etc/sensu/conf.d/client.json

{

“client”: {

“name”: “Client”,

“address”: “127.0.0.1”,

“subscriptions”: [

“memory_usage”]

}

}

  1.  In the server side, the following is our handler ( we are using Sensu Enterprise)
    
    **/etc/sensu/conf.d/eventstream.json**
    

{

“event_stream”: {

“host”: “127.0.0.1”,

“port”: 5514,

“filter_metrics”: false

}

}

  1. Attached is the screenshot of our monitoring application (Kibana). There are 4 check results/logs to every single execution of check-memory.rb (given they have same timestamp, command, etc.) - and this runs every 60 seconds which we intentionally set for testing purposes.
    Please advise. Thank you so much in advance!

Regards,

Keeshia

Hi Yuva,

Thank you for your response.

Sorry for the late reply. I don’t have much experience with Sensu, so is it okay to ask if you can elaborate how we can configure event_stream as bridge? :slight_smile:

Regards.

···

On Wednesday, April 12, 2017 at 9:55:00 AM UTC+8, Kaye wrote:

Hi All,

We are experiencing an issue where Sensu is forwarding 4 logs/ results every execution of a check (check-memory.rb). We are using Sensu Enterprise and all events are sent over to ELK stack using TCP and we’re not sure if this contributed to the result or we might have missed a setting or parameter in our configuration. Hope you guys can help. Thanks!

  1. In our client side, the following is the check configuration
    /etc/sensu/conf.d/check.json

{

“checks”: {

“disk_check”: {

“command”: “check-disk-usage.rb -w 80 -c 90 -p /sys/kernel/debug/tracing -x proc”,

“interval”: 600,

“standalone”: true,

“handler”: “event_stream”,

“subscribers”: [ “ALL” ]

},

“check_memory”: {

“command”: “/opt/sensu/embedded/bin/check-memory.rb”,

  "interval": 60,

“standalone”: true,

“handlers”: [“event_stream”, “email”],

“subscribers”: [“memory_usage”],

“type”: “metric”

},

“check_memory_percentage”: {

“command”: “/opt/sensu/embedded/bin/metrics-memory-percent.rb”,

“interval”: 900,

“standalone”: true,

“handler”: “event_stream”,

“subscribers”: [ “ALL” ]

},

“check_cpu”: {

“command”: “/opt/sensu/embedded/bin/metrics-cpu-mpstat.rb”,

“interval”: 900,

“standalone”: true,

“handler”: “event_stream”,

“subscribers”: [ “ALL” ]

}

}

}

/etc/sensu/conf.d/client.json

{

“client”: {

“name”: “Client”,

“address”: “127.0.0.1”,

“subscriptions”: [

“memory_usage”]

}

}

  1.  In the server side, the following is our handler ( we are using Sensu Enterprise)
    
    **/etc/sensu/conf.d/eventstream.json**
    

{

“event_stream”: {

“host”: “127.0.0.1”,

“port”: 5514,

“filter_metrics”: false

}

}

  1. Attached is the screenshot of our monitoring application (Kibana). There are 4 check results/logs to every single execution of check-memory.rb (given they have same timestamp, command, etc.) - and this runs every 60 seconds which we intentionally set for testing purposes.
    Please advise. Thank you so much in advance!

Regards,

Keeshia