sensu remediation not quite working ..not triggering client

Hi,
I am a newbee to Sensu remediation i tried to restart script if link goes down but sensu server is not triggering to client below are my configuration … please let me know what exactly is the problem.

this is the server log

{“timestamp”:“2016-05-16T09:44:52.768622+0000”,“level”:“info”,“message”:“processing event”,“event”:{“id”:“9a9f66c2-e70e-45fb-87fb-c9e9085c8e05”,“client”:{“name”:“zubron”,“address”:“10.0.0.110”,“subscriptions”:[“zubron”],“version”:“0.20.3”,“timestamp”:1463391880},“check”:{“command”:"/etc/sensu/plugins/check_http -H 10.0.0.110 -p 7077",“interval”:60,“occurrences”:2,“handlers”:[“remediator”],“subscribers”:[“zubron”],“standalone”:false,“remediation”:{“remediate-zubron”:{“occurrences”:[1,3],“severities”:[2]},“trigger_on”:[“zubron”]},“name”:“check-zubron-port”,“issued”:1463391892,“executed”:1463391892,“duration”:0.002,“output”:“connect to address 10.0.0.110 and port 7077: Connection refused\nHTTP CRITICAL - Unable to open TCP socket\n”,“status”:2,“history”:[“0”,“0”,“0”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:4},“occurrences”:18,“action”:“create”,“timestamp”:1463391892}}

{“timestamp”:“2016-05-16T09:44:52.864908+0000”,“level”:“info”,“message”:“handler output”,“handler”:{“command”:"/etc/sensu/handlers/sensu.rb",“type”:“pipe”,“severities”:[“critical”],“name”:“remediator”},“output”:["/etc/sensu/handlers/sensu.rb:108:in []': can't convert String into Integer (TypeError)\n","\tfrom /etc/sensu/handlers/sensu.rb:108:inblock in parse_remediations’\n","\tfrom /etc/sensu/handlers/sensu.rb:106:in each'\n","\tfrom /etc/sensu/handlers/sensu.rb:106:inparse_remediations’\n","\tfrom /etc/sensu/handlers/sensu.rb:90:in handle'\n","\tfrom /var/lib/gems/1.9.1/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:55:inblock in class:Handler’\n",“REMEDIATION: Evaluating remediation: zubron {“remediate-zubron”=>{“occurrences”=>[1, 3], “severities”=>[2]}, “trigger_on”=>[“zubron”]} #=18 sev=2\n”]}

this is check-http.json check file…

{

“checks”: {

“check-zubron-port”: {

“command”: “/etc/sensu/plugins/check_http -H 10.0.0.110 -p 7077”,

“interval”: 60,

“occurrences”: 2,

“handlers”: [

“remediator”

],

“subscribers”: [

“zubron”

],

“standalone”: false,

“remediation”: {

“remediate-zubron”: {

“occurrences”: [

1,

3

],

“severities”: [

2

]

},

“trigger_on”: [

“zubron”

]

}

}

}

}

this is remediation file…

{

“remediate-zubron”: {

“command”: “sudo /bin/bash ~/zubron/home/moofwd-zubron-server/bin/start-moofwd.sh”,

“handlers”: ,

“subscribers”: [

“zubron”

],

“standalone”: false,

“publish”: false

}

}

Is there anything m missing pls tell me i need to update fast

Please read the last line of the log that you pasted carefully.

It reports
can't convert String into Integer (TypeError)\n","

Are you sure that is pasted in the email is what is on disk? In that
error log it says:
{\"remediate-zubron\"=>{\"occurrences\"=>[1, 3], \"severities\"=>[2]},
\"trigger_on\"=>[\"zubron\"]} #=18 sev=2\n"]}

Which hints that there is some sort of comment ("#=18 sev=2") in the
json file, which is not valid.

···

On Mon, May 16, 2016 at 2:47 AM, vinod katuwa <vinod.katuwa12@gmail.com> wrote:

Hi,
I am a newbee to Sensu remediation i tried to restart script if link goes
down but sensu server is not triggering to client below are my configuration
.. please let me know what exactly is the problem.

this is the server log
{"timestamp":"2016-05-16T09:44:52.768622+0000","level":"info","message":"processing
event","event":{"id":"9a9f66c2-e70e-45fb-87fb-c9e9085c8e05","client":{"name":"zubron","address":"10.0.0.110","subscriptions":["zubron"],"version":"0.20.3","timestamp":1463391880},"check":{"command":"/etc/sensu/plugins/check_http
-H 10.0.0.110 -p
7077","interval":60,"occurrences":2,"handlers":["remediator"],"subscribers":["zubron"],"standalone":false,"remediation":{"remediate-zubron":{"occurrences":[1,3],"severities":[2]},"trigger_on":["zubron"]},"name":"check-zubron-port","issued":1463391892,"executed":1463391892,"duration":0.002,"output":"connect
to address 10.0.0.110 and port 7077: Connection refused\nHTTP CRITICAL -
Unable to open TCP
socket\n","status":2,"history":["0","0","0","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2"],"total_state_change":4},"occurrences":18,"action":"create","timestamp":1463391892}}
{"timestamp":"2016-05-16T09:44:52.864908+0000","level":"info","message":"handler
output","handler":{"command":"/etc/sensu/handlers/sensu.rb","type":"pipe","severities":["critical"],"name":"remediator"},"output":["/etc/sensu/handlers/sensu.rb:108:in
`': can't convert String into Integer (TypeError)\n","\tfrom
/etc/sensu/handlers/sensu.rb:108:in `block in parse_remediations'\n","\tfrom
/etc/sensu/handlers/sensu.rb:106:in `each'\n","\tfrom
/etc/sensu/handlers/sensu.rb:106:in `parse_remediations'\n","\tfrom
/etc/sensu/handlers/sensu.rb:90:in `handle'\n","\tfrom
/var/lib/gems/1.9.1/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:55:in
`block in <class:Handler>'\n","REMEDIATION: Evaluating remediation: zubron
{\"remediate-zubron\"=>{\"occurrences\"=>[1, 3], \"severities\"=>[2]},
\"trigger_on\"=>[\"zubron\"]} #=18 sev=2\n"]}

this is check-http.json check file..

{
        "checks": {
                "check-zubron-port": {
                        "command": "/etc/sensu/plugins/check_http -H
10.0.0.110 -p 7077",
                        "interval": 60,
                        "occurrences": 2,
                        "handlers": [
                                "remediator"
                        ],
                        "subscribers": [
                                "zubron"
                        ],
                        "standalone": false,
                        "remediation": {
                                "remediate-zubron": {
                                        "occurrences": [
                                                1,
                                                3
                                        ],
                                        "severities": [
                                                2
                                        ]
                                },
                                "trigger_on": [
                                        "zubron"
                                ]
                        }
                }
        }
}

this is remediation file...
{
        "remediate-zubron": {
            "command": "sudo /bin/bash
~/zubron/home/moofwd-zubron-server/bin/start-moofwd.sh",
            "handlers": ,
            "subscribers": [
                "zubron"
            ],
            "standalone": false,
            "publish": false
   }
}

Is there anything m missing pls tell me i need to update fast

sir its a valid json file , can u pls tell me m i following proper steps , is there anything else or can u pls help me any other blog for remediation

···

On Tuesday, May 17, 2016 at 10:26:00 AM UTC+5:30, Kyle Anderson wrote:

Please read the last line of the log that you pasted carefully.

It reports
can’t convert String into Integer (TypeError)\n","

Are you sure that is pasted in the email is what is on disk? In that
error log it says:
{“remediate-zubron”=>{“occurrences”=>[1, 3], “severities”=>[2]},
“trigger_on”=>[“zubron”]} #=18 sev=2\n"]}

Which hints that there is some sort of comment ("#=18 sev=2") in the
json file, which is not valid.

On Mon, May 16, 2016 at 2:47 AM, vinod katuwa vinod.k...@gmail.com wrote:

Hi,
I am a newbee to Sensu remediation i tried to restart script if link goes
down but sensu server is not triggering to client below are my configuration
… please let me know what exactly is the problem.

this is the server log
{“timestamp”:“2016-05-16T09:44:52.768622+0000”,“level”:“info”,“message”:“processing
event”,“event”:{“id”:“9a9f66c2-e70e-45fb-87fb-c9e9085c8e05”,“client”:{“name”:“zubron”,“address”:“10.0.0.110”,“subscriptions”:[“zubron”],“version”:“0.20.3”,“timestamp”:1463391880},“check”:{“command”:"/etc/sensu/plugins/check_http
-H 10.0.0.110 -p
7077",“interval”:60,“occurrences”:2,“handlers”:[“remediator”],“subscribers”:[“zubron”],“standalone”:false,“remediation”:{“remediate-zubron”:{“occurrences”:[1,3],“severities”:[2]},“trigger_on”:[“zubron”]},“name”:“check-zubron-port”,“issued”:1463391892,“executed”:1463391892,“duration”:0.002,“output”:“connect
to address 10.0.0.110 and port 7077: Connection refused\nHTTP CRITICAL -
Unable to open TCP
socket\n”,“status”:2,“history”:[“0”,“0”,“0”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:4},“occurrences”:18,“action”:“create”,“timestamp”:1463391892}}
{“timestamp”:“2016-05-16T09:44:52.864908+0000”,“level”:“info”,“message”:“handler
output”,“handler”:{“command”:"/etc/sensu/handlers/sensu.rb",“type”:“pipe”,“severities”:[“critical”],“name”:“remediator”},“output”:["/etc/sensu/handlers/sensu.rb:108:in
[]': can't convert String into Integer (TypeError)\n","\tfrom /etc/sensu/handlers/sensu.rb:108:inblock in parse_remediations’\n","\tfrom
/etc/sensu/handlers/sensu.rb:106:in each'\n","\tfrom /etc/sensu/handlers/sensu.rb:106:inparse_remediations’\n","\tfrom
/etc/sensu/handlers/sensu.rb:90:in handle'\n","\tfrom /var/lib/gems/1.9.1/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:55:inblock in class:Handler’\n",“REMEDIATION: Evaluating remediation: zubron
{“remediate-zubron”=>{“occurrences”=>[1, 3], “severities”=>[2]},
“trigger_on”=>[“zubron”]} #=18 sev=2\n”]}

this is check-http.json check file…

{
“checks”: {
“check-zubron-port”: {
“command”: “/etc/sensu/plugins/check_http -H
10.0.0.110 -p 7077”,
“interval”: 60,
“occurrences”: 2,
“handlers”: [
“remediator”
],
“subscribers”: [
“zubron”
],
“standalone”: false,
“remediation”: {
“remediate-zubron”: {
“occurrences”: [
1,
3
],
“severities”: [
2
]
},
“trigger_on”: [
“zubron”
]
}
}
}
}

this is remediation file…
{
“remediate-zubron”: {
“command”: “sudo /bin/bash
~/zubron/home/moofwd-zubron-server/bin/start-moofwd.sh”,
“handlers”: ,
“subscribers”: [
“zubron”
],
“standalone”: false,
“publish”: false
}
}

Is there anything m missing pls tell me i need to update fast

Hi Kyle sir,

i m now getting proper connection with client its triggering that command aswell but its not getting executed, pls can u tell me what shd be done on client side to execute that command from sensu server

I am also facing the same issue, can some help in solving it, API is triggering the post request but its not getting executed on client side

can u share what exactly you are facing the problem with means which service you are trying to restarted or trigger. did u gave sensu user permission with no-passowrd . better look in to my post i just created for Sensu Remidiation.

http://linuxgeekstuffs.com/sensu-remediation-configuration/

···


Thanks…

Vinod N Katuwa (Maheshwari)
vinodk@moofwd.com | BigData****DevOps Engineer
P: +91-22-61270221 | C: +91-8898441221

On 14-Feb-2017, at 6:22 PM, IMRAN SHAIK imranrehman5155@gmail.com wrote:

I am also facing the same issue, can some help in solving it, API is triggering the post request but its not getting executed on client side

202 means remediation is reaching and executing file …

Check running that script manually there … check with permission & executing permission.

Add sensu user in Sudoer file.

If still you are facing prob then you can call me …

Are all other

···

On Tue, 14 Feb 2017 at 7:15 PM, IMRAN SHAIK imranrehman5155@gmail.com wrote:

This is my check

{

“checks”: {

“check-python-script”: {

“command”: “check-process.rb -p oood.py -C 1”,

“interval”: 10,

“occurrences”: 3,

“notification”: “Converter python script is not running!”,

“refresh”: 10,

“subscribers”: [“CONVERTER”],

“handlers”: [“remediator”],

“remediation”: {

“trigger-python-script”: {

“occurrences”: [“1+”],

“severities”: [2]

}

}

},

“trigger-python-script”: {

“command”: “echo hai >> /mnt/test.txt”,

“publish”: false

}

}

}

This is what I am getting in sensu-api.log

{

“checks”: {

“check-python-script”: {

“command”: “check-process.rb -p oood.py -C 1”,

“interval”: 10,

“occurrences”: 3,

“notification”: “Converter python script is not running!”,

“refresh”: 10,

“subscribers”: [“CONVERTER”],

“handlers”: [“remediator”],

“remediation”: {

“trigger-python-script”: {

“occurrences”: [“1+”],

“severities”: [2]

}

}

},

“trigger-python-script”: {

“command”: “echo hai >> /mnt/test.txt”,

“publish”: false

}

}

}

below is for seneu-server.log

{

“timestamp”: “2017-02-14T13:14:43.575709+0000”,

“level”: “info”,

“message”: “handler output”,

“handler”: {

  "type": "pipe",
  "command": "/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-plugins-sensu-1.0.0/bin/handler-sensu.rb",
  "name": "remediator"

},

“output”: [“REMEDIATION: Evaluating remediation: servername {“trigger-python-script”=>{“occurrences”=>[“1+”], “severities”=>[2]}} #=570 sev=2\nREMEDIATION: Triggering remediation check ‘trigger-python-script’ for [“servername”]\nREMEDIATION: Received API Response (202): {“issued”:1487078083}, exiting.\n”]

}

POST API IS TRIGGERING THE CALL BUT ITS NOT BEEN EXECUTING ON CLIEN

I have already added this in sudoers file

%sensu ALL=(ALL) NOPASSWD: ALL

I have executed triggering script manually its working, but when I gets executed through handler its not working

Remove that % that’s for particular group

Read abt sudoer file

···

On Tue, 14 Feb 2017 at 7:37 PM, IMRAN SHAIK imranrehman5155@gmail.com wrote:

I have already added this in sudoers file

%sensu ALL=(ALL) NOPASSWD: ALL

I have executed triggering script manually its working, but when I gets executed through handler its not working

removed the % still its not working, get the same logs as mentioned

Hi Imran,
You can call me on +918898441221 i ll help u wth what exactly is the prob.

···


Thanks…

Vinod N Katuwa (Maheshwari)
vinodk@moofwd.com | BigData****DevOps Engineer
P: +91-22-61270221 | C: +91-8898441221

On 14-Feb-2017, at 8:23 PM, IMRAN SHAIK imranrehman5155@gmail.com wrote:

removed the % still its not working, get the same logs as mentioned

Helo Imran,

Comments inline.

···

On 14-Feb-2017, at 7:37 PM, IMRAN SHAIK imranrehman5155@gmail.com wrote:

I have already added this in sudoers file

%sensu ALL=(ALL) NOPASSWD: ALL

I have executed triggering script manually its working, but when I gets executed through handler its not working

This is what my sudoers looks like

[root@XXX ~]# cat /etc/sudoers.d/sensu

Defaults:sensu !requiretty

sensu ALL=(root) NOPASSWD: ALL

@shankerbalan

@shankerbalan

still no use, I see that Sensu API is triggering the command, but its not getting executed at client and giving result as 202.

{

"timestamp": "2017-02-15T09:15:19.845553+0000",

"level": "info",

"message": "publishing check request",

"payload": {

	"command": "sudo echo hai >> /mnt/test.txt",

	"subscribers": ["webserver"],

	"publish": false,

	"name": "trigger-python-script",

	"issued": 1487150119

},

"subscribers": ["webserver"]

},

{

"timestamp": "2017-02-15T09:15:19.845949+0000",

"level": "info",

"message": "api response",

"request": {

	"remote_address": "127.0.0.1",

	"user_agent": "Ruby",

	"method": "POST",

	"uri": "/request",

	"query_string": null,

	"body": "{\"check\":\"trigger-python-script\",\"subscribers\":[\"webserver\"]}"

},

"status": 202,

"content_length": 21

}

Hi Imran,

See commen inline…

···

On 15-Feb-2017, at 2:48 PM, IMRAN SHAIK <imranrehman5155@gmail.com> wrote:

@shankerbalan

still no use, I see that Sensu API is triggering the command, but its not getting executed at client and giving result as 202.

{
  "timestamp": "2017-02-15T09:15:19.845553+0000",
  "level": "info",
  "message": "publishing check request",
  "payload": {
    "command": "sudo echo hai >> /mnt/test.txt”,

Can you run a simple command like “ls” and see if that works for you?

Hi Imran,

I m not sure y you sensu user agent is saying ruby . With you details I really can’t help like this . Logs here which you are showing r very small part of it when ever is remi triggered it shows in uchiwa dashboard can u send snapshot of that check . And if it’s possible better let me take a look into ur demo machine then only I can check.

Thanks

Vinod NK

···

On Wed, 15 Feb 2017 at 3:28 PM, mail@shankerbalan.net mail@shankerbalan.net wrote:

Hi Imran,

See commen inline…

On 15-Feb-2017, at 2:48 PM, IMRAN SHAIK imranrehman5155@gmail.com wrote:

@shankerbalan

still no use, I see that Sensu API is triggering the command, but its not getting executed at client and giving result as 202.

{

  "timestamp": "2017-02-15T09:15:19.845553+0000",
  "level": "info",
  "message": "publishing check request",
  "payload": {
          "command": "sudo echo hai >> /mnt/test.txt”,

Can you run a simple command like “ls” and see if that works for you?

Sorry, I have missed adding clientname in trigger subscribers, Now I have added it and I seee that check is executing on client

But I am getting permission denied issue, how to solve this,

touch: cannot touch `/mnt/hai.txt’: Permission denied

Just give that dir /mnt permission with below command

chown -R sensu /mnt

this ll resolve your prob…

···


Thanks…

Vinod N Katuwa (Maheshwari)
vinodk@moofwd.com | BigData****DevOps Engineer
P: +91-22-61270221 | C: +91-8898441221

On 15-Feb-2017, at 4:36 PM, IMRAN SHAIK imranrehman5155@gmail.com wrote:

Sorry, I have missed adding clientname in trigger subscribers, Now I have added it and I seee that check is executing on client

But I am getting permission denied issue, how to solve this,

touch: cannot touch `/mnt/hai.txt’: Permission denied

thanks, it resolved the problem,

but after the remediator had triggered the script, and when I try to restart the sensu-client

Its not restarting

service sensu-client restart

configuration is valid

Timed out waiting for sensu-client to stop [FAILED]

                                                       [FAILED]

and I see the sensu-client.log is got stuck by below msg

{

"timestamp": "2017-02-16T09:42:13.179222+0000",

"level": "info",

"message": "publishing check result",

"payload": {

	"client": "webserver",

	"check": {

		"name": "check-python-script",

		"issued": 1487238132,

		"command": "check-process.rb -p oood.py -C 1",

		"executed": 1487238132,

		"duration": 0.305,

		"output": "CheckProcess OK: Found 1 matching processes; cmd /oood.py/\n",

		"status": 0

	}

}

} {

"timestamp": "2017-02-16T09:42:14.453940+0000",

"level": "info",

"message": "completing checks in progress",

"checks_in_progress": ["trigger-python-script"]

}