Sensu events are not sending to Logstash

Hello,

We are trying to send Sensu events to Logstash using tcp handler and we installed sensu-plugins-logstash to do the job but it doesn’t seem to be working and we are currently blocked in solving this as there are no significant logs that can be seen in both sensu-server and in logstash to identify if there are problems in sending, retrieval, or if there are issues in the configurations. We would truly appreciate if you can give us some advice. Thank you in advance!

  • /etc/sensu/conf.d/check.json

{

“checks”: {

“cpu_usage”: {

“command”: “check-ports.rb -h 192.168.30.187 -p 3000,15672 -t 30”,

“interval”: 60,

“handlers”: [“logstash”,“default”],

“ttl”: 90,

“subscribers”: [ “ALL” ]

}

}

}

  • /etc/sensu/conf.d/handlers.json
    {

“handlers”: {

“logstash”: {

“type”: “tcp”,

“command”: “/usr/local/bin/handler-logstash.rb”,

“socket”: {

“host”: “127.0.0.1”,

“port”: 5514

}

}

}

}

  • /etc/logstash/conf.d/logstash.conf
    input {

tcp {

port => 5514

codec => “json”

type => “sensu_log”

}

}

output {

elasticsearch {

hosts => [“127.0.0.1:9200”]

index => “%{[@metadata][sensu]}-%{+YYYY.MM.dd}”

}

}

What do you expect to happen?

a) every check result is written to logstash, regardless of exit code (aka. OK, Warn, etc) OR

b) every negative result is written to logstash (Warning, Critical, Unknown, but not OK)

from a cursory view of the snippets provided by you, that would be b).

Note that I have not sanity checked your config.

···

On Wednesday, March 8, 2017 at 1:41:43 PM UTC+1, Kaye wrote:

We are trying to send Sensu events to Logstash using tcp handler and we installed sensu-plugins-logstash to do the job but it doesn’t seem to be working and we are currently blocked in solving this as there are no significant logs that can be seen in both sensu-server and in logstash to identify if there are problems in sending, retrieval, or if there are issues in the configurations.

Really appreciate your response! :slight_smile: we actually wanted to monitor everything so that would be a). Can you advise what’s missing on our config? and would it be okay if you can run a sanity check? we are blocked and we really can’t figure out what’s wrong.
Thank you again.

···

On Thu, Mar 9, 2017 at 5:22 PM, Alexander Skiba ghostlyrics@gmail.com wrote:

What do you expect to happen?

a) every check result is written to logstash, regardless of exit code (aka. OK, Warn, etc) OR

b) every negative result is written to logstash (Warning, Critical, Unknown, but not OK)

from a cursory view of the snippets provided by you, that would be b).

Note that I have not sanity checked your config.

On Wednesday, March 8, 2017 at 1:41:43 PM UTC+1, Kaye wrote:

We are trying to send Sensu events to Logstash using tcp handler and we installed sensu-plugins-logstash to do the job but it doesn’t seem to be working and we are currently blocked in solving this as there are no significant logs that can be seen in both sensu-server and in logstash to identify if there are problems in sending, retrieval, or if there are issues in the configurations.

I would recommend to read the logs using logstash or filebeat instead of using a CPU heavy handler.

It sounds like you may have mixed the configuration for a command (pipe) handler and tcp handlers. With TCP handlers sensu is just going to send events to something else via tcp, it is not going to run ‘/usr/local/bin/handler-logstash.rb’ so the ‘command’ option in the handler configuration is not correct. What documentation are you looking at? I believe the ‘logstash.conf’ and ‘command’ in the tcp handler are misleading in your configuration and don’t represent what you are trying to do. (probably represent your first attempt using a pipe handler)

But the thing that listens for tcp must be running (logstash/beats). You must get that up and running fully first, otherwise sensu won’t be able to talk to it over tcp.

···

On Mon, Mar 13, 2017 at 12:21 AM, Keeshia San Pascual sanpascual.keeshia@gmail.com wrote:

Thank you for your response! we modified the check to “metric” but still, we cannot see the check results(Sensu) to kibana and the logs( logstash-plain.log) only displays info about filebeat (service is not running):

[2017-03-07T09:38:33,368][INFO ][logstash.pipeline ] Starting pipeline {“id”=>“main”, “pipeline.workers”=>2, “pipeline.batch.size”=>125, “pipeline.batch.delay”=>5, “pipeline.max_inflight”=>250}

[2017-03-07T09:38:33,809][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>“0.0.0.0:5044”}

[2017-03-07T09:38:33,842][INFO ][logstash.pipeline ] Pipeline main started

[2017-03-07T09:38:33,904][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

[2017-03-07T10:42:51,794][WARN ][logstash.runner ] SIGTERM received. Shutting down the agent.

[2017-03-07T10:42:51,801][WARN ][logstash.agent ] stopping pipeline {:id=>“main”}

[2017-03-07T10:42:57,591][WARN ][logstash.shutdownwatcher ] {“inflight_count”=>0, “stalling_thread_info”=>{“other”=>[{“thread_id”=>23, “name”=>"[main]<beats", “current_call”=>"[…]/vendor/bundle/jruby/1.9/gems/logstash-input-beats-3.1.12-java/lib/logstash/inputs/beats.rb:213:in run'"}, {"thread_id"=>20, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:127:insynchronize’"}, {“thread_id”=>21, “name”=>"[main]>worker1", “current_call”=>"[…]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:127:in `synchronize’"}]}}

[2017-03-07T10:42:57,593][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.

floadmin@flosensu1:/var/log/logstash$

so we’re not sure how to verify if the Sensu checks are sending to logstash and if our configurations are working. Kindly advise.

Thanks again.

On Sat, Mar 11, 2017 at 1:03 PM, Kyle Anderson kyle@xkyle.com wrote:

If you want a) (all events sent to logstash) then the checks need to be “metric” checks
https://sensuapp.org/docs/latest/reference/checks.html#check-attributes

I agree that using a handler like this is expensive cpu-wise.

On Thu, Mar 9, 2017 at 3:06 AM, Keeshia San Pascual sanpascual.keeshia@gmail.com wrote:

Really appreciate your response! :slight_smile: we actually wanted to monitor everything so that would be a). Can you advise what’s missing on our config? and would it be okay if you can run a sanity check? we are blocked and we really can’t figure out what’s wrong.
Thank you again.

On Thu, Mar 9, 2017 at 5:22 PM, Alexander Skiba ghostlyrics@gmail.com wrote:

What do you expect to happen?

a) every check result is written to logstash, regardless of exit code (aka. OK, Warn, etc) OR

b) every negative result is written to logstash (Warning, Critical, Unknown, but not OK)

from a cursory view of the snippets provided by you, that would be b).

Note that I have not sanity checked your config.

On Wednesday, March 8, 2017 at 1:41:43 PM UTC+1, Kaye wrote:

We are trying to send Sensu events to Logstash using tcp handler and we installed sensu-plugins-logstash to do the job but it doesn’t seem to be working and we are currently blocked in solving this as there are no significant logs that can be seen in both sensu-server and in logstash to identify if there are problems in sending, retrieval, or if there are issues in the configurations.

Thank you again for your advice :slight_smile: We are using these sites as references:

The first link seems to be outdated and we’re not particularly sure on the configuration of Sensu to Logstash.

We also intentionally stopped the beats service since we want to use Sensu as our metrics collection engine.

We updated our configurations but we still can’t get the Sensu checks to ELK stack:

# /etc/sensu/conf.d/check.json

{“checks”: {

      "CHECK": {

         "type": "metric",

         "command": "check-ports.rb -h 192.168.30.187 -p 3000,15672 -t 30",

         "subscribers": ["ALL"],

       "interval": 60,

         "handlers": ["logstash"]

}

}

}

# /etc/sensu/conf.d/handler.json

{

“handlers”: {

  "logstash": {

    "type": "tcp",

    "socket": {

      "host": "localhost",

      "port": 5514

    }

  }

}

}

# /etc/logstash/conf.d/logstash.conf

input {

      tcp {port => 5514

         codec => json

}

}

filter {

}

output {

elasticsearch {

  hosts => ["localhost:9200"]

  index => "%{[@metadata][sensu]}-%{+YYYY.MM.dd}"

     }

stdout {

  codec => rubydebug

     }

}

···

On Mon, Mar 13, 2017 at 10:54 PM, Kyle Anderson kyle@xkyle.com wrote:

But the thing that listens for tcp must be running (logstash/beats). You must get that up and running fully first, otherwise sensu won’t be able to talk to it over tcp.

It sounds like you may have mixed the configuration for a command (pipe) handler and tcp handlers. With TCP handlers sensu is just going to send events to something else via tcp, it is not going to run ‘/usr/local/bin/handler-logstash.rb’ so the ‘command’ option in the handler configuration is not correct. What documentation are you looking at? I believe the ‘logstash.conf’ and ‘command’ in the tcp handler are misleading in your configuration and don’t represent what you are trying to do. (probably represent your first attempt using a pipe handler)

On Mon, Mar 13, 2017 at 12:21 AM, Keeshia San Pascual sanpascual.keeshia@gmail.com wrote:

Thank you for your response! we modified the check to “metric” but still, we cannot see the check results(Sensu) to kibana and the logs( logstash-plain.log) only displays info about filebeat (service is not running):

[2017-03-07T09:38:33,368][INFO ][logstash.pipeline ] Starting pipeline {“id”=>“main”, “pipeline.workers”=>2, “pipeline.batch.size”=>125, “pipeline.batch.delay”=>5, “pipeline.max_inflight”=>250}

[2017-03-07T09:38:33,809][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>“0.0.0.0:5044”}

[2017-03-07T09:38:33,842][INFO ][logstash.pipeline ] Pipeline main started

[2017-03-07T09:38:33,904][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

[2017-03-07T10:42:51,794][WARN ][logstash.runner ] SIGTERM received. Shutting down the agent.

[2017-03-07T10:42:51,801][WARN ][logstash.agent ] stopping pipeline {:id=>“main”}

[2017-03-07T10:42:57,591][WARN ][logstash.shutdownwatcher ] {“inflight_count”=>0, “stalling_thread_info”=>{“other”=>[{“thread_id”=>23, “name”=>"[main]<beats", “current_call”=>"[…]/vendor/bundle/jruby/1.9/gems/logstash-input-beats-3.1.12-java/lib/logstash/inputs/beats.rb:213:in run'"}, {"thread_id"=>20, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:127:insynchronize’"}, {“thread_id”=>21, “name”=>"[main]>worker1", “current_call”=>"[…]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:127:in `synchronize’"}]}}

[2017-03-07T10:42:57,593][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.

floadmin@flosensu1:/var/log/logstash$

so we’re not sure how to verify if the Sensu checks are sending to logstash and if our configurations are working. Kindly advise.

Thanks again.

On Sat, Mar 11, 2017 at 1:03 PM, Kyle Anderson kyle@xkyle.com wrote:

If you want a) (all events sent to logstash) then the checks need to be “metric” checks
https://sensuapp.org/docs/latest/reference/checks.html#check-attributes

I agree that using a handler like this is expensive cpu-wise.

On Thu, Mar 9, 2017 at 3:06 AM, Keeshia San Pascual sanpascual.keeshia@gmail.com wrote:

Really appreciate your response! :slight_smile: we actually wanted to monitor everything so that would be a). Can you advise what’s missing on our config? and would it be okay if you can run a sanity check? we are blocked and we really can’t figure out what’s wrong.
Thank you again.

On Thu, Mar 9, 2017 at 5:22 PM, Alexander Skiba ghostlyrics@gmail.com wrote:

What do you expect to happen?

a) every check result is written to logstash, regardless of exit code (aka. OK, Warn, etc) OR

b) every negative result is written to logstash (Warning, Critical, Unknown, but not OK)

from a cursory view of the snippets provided by you, that would be b).

Note that I have not sanity checked your config.

On Wednesday, March 8, 2017 at 1:41:43 PM UTC+1, Kaye wrote:

We are trying to send Sensu events to Logstash using tcp handler and we installed sensu-plugins-logstash to do the job but it doesn’t seem to be working and we are currently blocked in solving this as there are no significant logs that can be seen in both sensu-server and in logstash to identify if there are problems in sending, retrieval, or if there are issues in the configurations.

I think a TCP handler is the right idea here, but note that each event will require a new TCP socket, with associated handshake overhead.

Sensu Enterprise is available with a free 14 day trial, and offers an event stream integration for sending event data to external services via long-lived TCP socket.

This integration works without setting checks type to metric, etc.

See https://sensuapp.org/docs/latest/enterprise/integrations/event_stream.html for details.

Cameron

···

On Monday, March 13, 2017 at 8:54:03 AM UTC-6, Kyle Anderson wrote:

But the thing that listens for tcp must be running (logstash/beats). You must get that up and running fully first, otherwise sensu won’t be able to talk to it over tcp.

It sounds like you may have mixed the configuration for a command (pipe) handler and tcp handlers. With TCP handlers sensu is just going to send events to something else via tcp, it is not going to run ‘/usr/local/bin/handler-logstash.rb’ so the ‘command’ option in the handler configuration is not correct. What documentation are you looking at? I believe the ‘logstash.conf’ and ‘command’ in the tcp handler are misleading in your configuration and don’t represent what you are trying to do. (probably represent your first attempt using a pipe handler)

On Mon, Mar 13, 2017 at 12:21 AM, Keeshia San Pascual sanpascua...@gmail.com wrote:

Thank you for your response! we modified the check to “metric” but still, we cannot see the check results(Sensu) to kibana and the logs( logstash-plain.log) only displays info about filebeat (service is not running):

[2017-03-07T09:38:33,368][INFO ][logstash.pipeline ] Starting pipeline {“id”=>“main”, “pipeline.workers”=>2, “pipeline.batch.size”=>125, “pipeline.batch.delay”=>5, “pipeline.max_inflight”=>250}

[2017-03-07T09:38:33,809][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>“0.0.0.0:5044”}

[2017-03-07T09:38:33,842][INFO ][logstash.pipeline ] Pipeline main started

[2017-03-07T09:38:33,904][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

[2017-03-07T10:42:51,794][WARN ][logstash.runner ] SIGTERM received. Shutting down the agent.

[2017-03-07T10:42:51,801][WARN ][logstash.agent ] stopping pipeline {:id=>“main”}

[2017-03-07T10:42:57,591][WARN ][logstash.shutdownwatcher ] {“inflight_count”=>0, “stalling_thread_info”=>{“other”=>[{“thread_id”=>23, “name”=>"[main]<beats", “current_call”=>"[…]/vendor/bundle/jruby/1.9/gems/logstash-input-beats-3.1.12-java/lib/logstash/inputs/beats.rb:213:in run'"}, {"thread_id"=>20, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:127:insynchronize’"}, {“thread_id”=>21, “name”=>"[main]>worker1", “current_call”=>"[…]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:127:in `synchronize’"}]}}

[2017-03-07T10:42:57,593][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.

floadmin@flosensu1:/var/log/logstash$

so we’re not sure how to verify if the Sensu checks are sending to logstash and if our configurations are working. Kindly advise.

Thanks again.

On Sat, Mar 11, 2017 at 1:03 PM, Kyle Anderson ky...@xkyle.com wrote:

If you want a) (all events sent to logstash) then the checks need to be “metric” checks
https://sensuapp.org/docs/latest/reference/checks.html#check-attributes

I agree that using a handler like this is expensive cpu-wise.

On Thu, Mar 9, 2017 at 3:06 AM, Keeshia San Pascual sanpascua...@gmail.com wrote:

Really appreciate your response! :slight_smile: we actually wanted to monitor everything so that would be a). Can you advise what’s missing on our config? and would it be okay if you can run a sanity check? we are blocked and we really can’t figure out what’s wrong.
Thank you again.

On Thu, Mar 9, 2017 at 5:22 PM, Alexander Skiba ghost...@gmail.com wrote:

What do you expect to happen?

a) every check result is written to logstash, regardless of exit code (aka. OK, Warn, etc) OR

b) every negative result is written to logstash (Warning, Critical, Unknown, but not OK)

from a cursory view of the snippets provided by you, that would be b).

Note that I have not sanity checked your config.

On Wednesday, March 8, 2017 at 1:41:43 PM UTC+1, Kaye wrote:

We are trying to send Sensu events to Logstash using tcp handler and we installed sensu-plugins-logstash to do the job but it doesn’t seem to be working and we are currently blocked in solving this as there are no significant logs that can be seen in both sensu-server and in logstash to identify if there are problems in sending, retrieval, or if there are issues in the configurations.

I suggest handlers aren’t the way to get your event data into logstash. if there’s any variation in the handlers for different types of checks you have to make sure none of them forget to set the logging handler or they don’t get logged. if you label your check a metric in order to get non failing events handled, then all your handlers handle all results not just logstash, and they have to implement filtering themselves.

the sensu-server log has every event that’s handled by any handler. and filebeat or something else can be made to filter on “processing event” to get only the event lines, and on occurrences = 1 or action resolve to get just the state changes. (just the state changes of failing events is what I think you want in logstash)

However this doesn’t log all events only those that are failing, or labeled metric. If you want all, without the caveats of handlers you have to look in the code and make use of the event_bridge extension capability. [ thanks vijay ]

some example filebeat for selecting just the state changes of processed event lines from the log file.

filebeat.prospectors:

  • input_type: stdin

    json.keys_under_root: true

    document_type: sensu-event-logs-v2

    fields:

    foo: “%{[message]}”

output.console:

pretty: true

processors:

  • drop_event:

    when:

    and:
    
      - contains: 
    
          type: sensu-event
    
      - not:
    
          and:
    
            - equals:
    
                message: "processing event"
    
            - or: 
    
              - equals:
    
                  event.occurrences: 1
    
              - equals:
    
                  event.action: resolve
    
···

On Tuesday, March 14, 2017 at 10:17:32 AM UTC-7, Cameron Johnston wrote:

I think a TCP handler is the right idea here, but note that each event will require a new TCP socket, with associated handshake overhead.

Sensu Enterprise is available with a free 14 day trial, and offers an event stream integration for sending event data to external services via long-lived TCP socket.

This integration works without setting checks type to metric, etc.

See https://sensuapp.org/docs/latest/enterprise/integrations/event_stream.html for details.

Cameron

On Monday, March 13, 2017 at 8:54:03 AM UTC-6, Kyle Anderson wrote:

But the thing that listens for tcp must be running (logstash/beats). You must get that up and running fully first, otherwise sensu won’t be able to talk to it over tcp.

It sounds like you may have mixed the configuration for a command (pipe) handler and tcp handlers. With TCP handlers sensu is just going to send events to something else via tcp, it is not going to run ‘/usr/local/bin/handler-logstash.rb’ so the ‘command’ option in the handler configuration is not correct. What documentation are you looking at? I believe the ‘logstash.conf’ and ‘command’ in the tcp handler are misleading in your configuration and don’t represent what you are trying to do. (probably represent your first attempt using a pipe handler)

On Mon, Mar 13, 2017 at 12:21 AM, Keeshia San Pascual sanpascua...@gmail.com wrote:

Thank you for your response! we modified the check to “metric” but still, we cannot see the check results(Sensu) to kibana and the logs( logstash-plain.log) only displays info about filebeat (service is not running):

[2017-03-07T09:38:33,368][INFO ][logstash.pipeline ] Starting pipeline {“id”=>“main”, “pipeline.workers”=>2, “pipeline.batch.size”=>125, “pipeline.batch.delay”=>5, “pipeline.max_inflight”=>250}

[2017-03-07T09:38:33,809][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>“0.0.0.0:5044”}

[2017-03-07T09:38:33,842][INFO ][logstash.pipeline ] Pipeline main started

[2017-03-07T09:38:33,904][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

[2017-03-07T10:42:51,794][WARN ][logstash.runner ] SIGTERM received. Shutting down the agent.

[2017-03-07T10:42:51,801][WARN ][logstash.agent ] stopping pipeline {:id=>“main”}

[2017-03-07T10:42:57,591][WARN ][logstash.shutdownwatcher ] {“inflight_count”=>0, “stalling_thread_info”=>{“other”=>[{“thread_id”=>23, “name”=>"[main]<beats", “current_call”=>"[…]/vendor/bundle/jruby/1.9/gems/logstash-input-beats-3.1.12-java/lib/logstash/inputs/beats.rb:213:in run'"}, {"thread_id"=>20, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:127:insynchronize’"}, {“thread_id”=>21, “name”=>"[main]>worker1", “current_call”=>"[…]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:127:in `synchronize’"}]}}

[2017-03-07T10:42:57,593][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.

floadmin@flosensu1:/var/log/logstash$

so we’re not sure how to verify if the Sensu checks are sending to logstash and if our configurations are working. Kindly advise.

Thanks again.

On Sat, Mar 11, 2017 at 1:03 PM, Kyle Anderson ky...@xkyle.com wrote:

If you want a) (all events sent to logstash) then the checks need to be “metric” checks
https://sensuapp.org/docs/latest/reference/checks.html#check-attributes

I agree that using a handler like this is expensive cpu-wise.

On Thu, Mar 9, 2017 at 3:06 AM, Keeshia San Pascual sanpascua...@gmail.com wrote:

Really appreciate your response! :slight_smile: we actually wanted to monitor everything so that would be a). Can you advise what’s missing on our config? and would it be okay if you can run a sanity check? we are blocked and we really can’t figure out what’s wrong.
Thank you again.

On Thu, Mar 9, 2017 at 5:22 PM, Alexander Skiba ghost...@gmail.com wrote:

What do you expect to happen?

a) every check result is written to logstash, regardless of exit code (aka. OK, Warn, etc) OR

b) every negative result is written to logstash (Warning, Critical, Unknown, but not OK)

from a cursory view of the snippets provided by you, that would be b).

Note that I have not sanity checked your config.

On Wednesday, March 8, 2017 at 1:41:43 PM UTC+1, Kaye wrote:

We are trying to send Sensu events to Logstash using tcp handler and we installed sensu-plugins-logstash to do the job but it doesn’t seem to be working and we are currently blocked in solving this as there are no significant logs that can be seen in both sensu-server and in logstash to identify if there are problems in sending, retrieval, or if there are issues in the configurations.

I little of-topic - I suggest to avoid TCP for events delivery to Logstash and look for other method for integration. Or at least double check that Sensu works stable when Logstash is down.
A few years ago we encountered several outage when used TCP stream for log delivery.

···

On Wednesday, March 8, 2017 at 2:41:43 PM UTC+2, Kaye wrote:

Hello,

We are trying to send Sensu events to Logstash using tcp handler and we installed sensu-plugins-logstash to do the job but it doesn’t seem to be working and we are currently blocked in solving this as there are no significant logs that can be seen in both sensu-server and in logstash to identify if there are problems in sending, retrieval, or if there are issues in the configurations. We would truly appreciate if you can give us some advice. Thank you in advance!

  • /etc/sensu/conf.d/check.json

{

“checks”: {

“cpu_usage”: {

“command”: “check-ports.rb -h 192.168.30.187 -p 3000,15672 -t 30”,

“interval”: 60,

“handlers”: [“logstash”,“default”],

“ttl”: 90,

“subscribers”: [ “ALL” ]

}

}

}

  • /etc/sensu/conf.d/handlers.json
    {

“handlers”: {

“logstash”: {

“type”: “tcp”,

“command”: “/usr/local/bin/handler-logstash.rb”,

“socket”: {

“host”: “127.0.0.1”,

“port”: 5514

}

}

}

}

  • /etc/logstash/conf.d/logstash.conf
    input {

tcp {

port => 5514

codec => “json”

type => “sensu_log”

}

}

output {

elasticsearch {

hosts => [“127.0.0.1:9200”]

index => “%{[@metadata][sensu]}-%{+YYYY.MM.dd}”

}

}

Hi,

Thanks for your response! :slight_smile: This has been resolved by installing Sensu Enterprise. We configured the Event Stream to forward the events to Logstash.

We are still reviewing the products and the possible methods of integration.

···

On Sun, Mar 19, 2017 at 8:41 PM, Vitaly Karasik linux.il@gmail.com wrote:

I little of-topic - I suggest to avoid TCP for events delivery to Logstash and look for other method for integration. Or at least double check that Sensu works stable when Logstash is down.
A few years ago we encountered several outage when used TCP stream for log delivery.

On Wednesday, March 8, 2017 at 2:41:43 PM UTC+2, Kaye wrote:

Hello,

We are trying to send Sensu events to Logstash using tcp handler and we installed sensu-plugins-logstash to do the job but it doesn’t seem to be working and we are currently blocked in solving this as there are no significant logs that can be seen in both sensu-server and in logstash to identify if there are problems in sending, retrieval, or if there are issues in the configurations. We would truly appreciate if you can give us some advice. Thank you in advance!

  • /etc/sensu/conf.d/check.json

{

“checks”: {

“cpu_usage”: {

“command”: “check-ports.rb -h 192.168.30.187 -p 3000,15672 -t 30”,

“interval”: 60,

“handlers”: [“logstash”,“default”],

“ttl”: 90,

“subscribers”: [ “ALL” ]

}

}

}

  • /etc/sensu/conf.d/handlers.json
    {

“handlers”: {

“logstash”: {

“type”: “tcp”,

“command”: “/usr/local/bin/handler-logstash.rb”,

“socket”: {

“host”: “127.0.0.1”,

“port”: 5514

}

}

}

}

  • /etc/logstash/conf.d/logstash.conf
    input {

tcp {

port => 5514

codec => “json”

type => “sensu_log”

}

}

output {

elasticsearch {

hosts => [“127.0.0.1:9200”]

index => “%{[@metadata][sensu]}-%{+YYYY.MM.dd}”

}

}