bulk check result submission into Sensu

Nagios has a spool directory where check results are written into text
files and queued for processing.

It is possible for results from multiple checks to be placed in a single
file, all of those results are then processed more efficiently by the
Nagios code.

ganglia-nagios-bridge[1] extracts all the metrics from a Ganglia gmetad
server in a single poll and dumps them all out to a single Nagios check
results file for bulk processing.

Can Sensu accept these Nagios bulk checkresult files?

If not, is there any alternative way to submit bulk check results from
tools like ganglia-nagios-bridge into Sensu? It has been observed
processing thousands of check results in one go, doing each of those
individually through the REST API[2] would appear to be very inefficient.

syslog-nagios-bridge[3] also uses this mechanism to submit results to
Nagios, but usually not with the same intensity as ganglia-nagios-bridge.

Regards,

Daniel

1. https://danielpocock.com/ganglia-nagios-bridge
2. https://sensuapp.org/docs/latest/api/results-api.html#results-post
3. https://github.com/dpocock/syslog-nagios-bridge

https://sensuapp.org/docs/0.26/reference/clients.html#example-client-socket-usage

Is the socket you want to send bulk check results, but it consumes json of its own specification.

···

On Mon, Oct 17, 2016 at 7:14 AM, Daniel Pocock daniel@pocock.pro wrote:

Nagios has a spool directory where check results are written into text

files and queued for processing.

It is possible for results from multiple checks to be placed in a single

file, all of those results are then processed more efficiently by the

Nagios code.

ganglia-nagios-bridge[1] extracts all the metrics from a Ganglia gmetad

server in a single poll and dumps them all out to a single Nagios check

results file for bulk processing.

Can Sensu accept these Nagios bulk checkresult files?

If not, is there any alternative way to submit bulk check results from

tools like ganglia-nagios-bridge into Sensu? It has been observed

processing thousands of check results in one go, doing each of those

individually through the REST API[2] would appear to be very inefficient.

syslog-nagios-bridge[3] also uses this mechanism to submit results to

Nagios, but usually not with the same intensity as ganglia-nagios-bridge.

Regards,

Daniel

  1. https://danielpocock.com/ganglia-nagios-bridge

  2. https://sensuapp.org/docs/latest/api/results-api.html#results-post

  3. https://github.com/dpocock/syslog-nagios-bridge

Can multiple results be submitted in a single TCP connection?

Or is it necessary to reconnect each time a result is submitted?

Doesn't the client then have to send them over RabbitMQ to the server,
is there no way to get thousands of check results directly into the
server process, bypassing the client and RabbitMQ?

···

On 17/10/16 16:43, Kyle Anderson wrote:

https://sensuapp.org/docs/0.26/reference/clients.html#example-client-socket-usage

Is the socket you want to send bulk check results, but it consumes json
of its own specification.

Hmm. I’ve never tried submitting multiple requests to that socket at a time, but aparently you can, and even keep a consistent connection (but this doesn’t look like it is documented):
https://github.com/sensu/sensu/blob/899c298bca5e2c5dfa79a3622cb1b2ee59424f10/lib/sensu/client/socket.rb#L235-L238

But even via the API, it is still submitting to rabbitmq for the server to process.

I’m not aware of a method to bypass the transport. Maybe you could write an Extension. (Maybe the server could have a socket?

But if it did, how would you be sure you were talking to the current leader, etc?)

···

On Mon, Oct 17, 2016 at 7:46 AM, Daniel Pocock daniel@pocock.pro wrote:

On 17/10/16 16:43, Kyle Anderson wrote:

https://sensuapp.org/docs/0.26/reference/clients.html#example-client-socket-usage

Is the socket you want to send bulk check results, but it consumes json

of its own specification.

Can multiple results be submitted in a single TCP connection?

Or is it necessary to reconnect each time a result is submitted?

Doesn’t the client then have to send them over RabbitMQ to the server,

is there no way to get thousands of check results directly into the

server process, bypassing the client and RabbitMQ?

Las time I checked, you can't. The tcp socket is read as long as the data
received fails to parse as a json string or a timeout occurrs (500msec
iirc). If you try to submit multiple check results, how would you append
one check result to another? json parsing would fail unless you sent a json
array of check results, in which case code will fail cuz is not expecting
an array, but just a json object (hash/dictionary).

···

On Mon, Oct 17, 2016 at 10:59 AM, Kyle Anderson <kyle@xkyle.com> wrote:

Hmm. I've never tried submitting multiple requests to that socket at a
time, but aparently you can, and even keep a consistent connection (but
this doesn't look like it is documented):
https://github.com/sensu/sensu/blob/899c298bca5e2c5dfa79a3622cb1b2
ee59424f10/lib/sensu/client/socket.rb#L235-L238

aside: transport (rabbitmq usually) is the servers work queue. you don’t want to bypass it.

···

On Oct 17, 2016, at 7:46 AM, Daniel Pocock <daniel@pocock.pro> wrote:

is there no way to get thousands of check results directly into the
server process, bypassing the client and RabbitMQ?

Would you consider supporting some bulk imports through a dedicated
queue in either RabbitMQ or directly into Redis perhaps? I notice you
also support Redis as a transport now, can both Redis and RabbitMQ
transports be enabled concurrently for different clients?

Something like ganglia-nagios-bridge (or ganglia-sensu-bridge) would
probably run on the same host as the Sensu server itself.

Regards,

Daniel

···

On 17/10/16 19:25, John Fessenden wrote:

aside: transport (rabbitmq usually) is the servers work queue. you don’t want to bypass it.