Proxy check is unstable

Sensu Go
1

Proxy check is not get collect entity label information and sometimes check is not interval 1 minuite.
(Like execute at 18:00, 18:01, 18:02, 18:08, 18:09, …)

Detail
[proxy check request]

{
  "command": "check-mysql hang --password=<PASSWORD> --user=<USER> --host={{ .name }} --readwrite={{ .labels.adminhangcheckreadwrite | default false }} --retry={{ .labels.adminhangcheckretry | default 5 }} --interval={{ .labels.adminhangcheckinterval | default 10}} --subdue {{ .annotations.subdue | default \"nil\" }} --check-name \"check-mysql-hang-admin\"",
  "handlers": [
    "admin-email"
  ],
  "high_flap_threshold": 9,
  "interval": 60,
  "low_flap_threshold": 0,
  "publish": true,
  "runtime_assets": null,
  "subscriptions": [
    "sensu-backend"
  ],
  "proxy_entity_name": "",
  "check_hooks": null,
  "stdin": false,
  "subdue": null,
  "ttl": 0,
  "timeout": 0,
  "proxy_requests": {
    "entity_attributes": [
      "entity.subscriptions.indexOf('server:DBaaS') >= 0|| entity.subscriptions.indexOf('entity-type:lbvip') >= 0"
    ],
    "splay": true,
    "splay_coverage": 90
  },
  "round_robin": true,
  "output_metric_format": "",
  "output_metric_handlers": null,
  "env_vars": null,
  "metadata": {
    "name": "check-mysql-hang-admin",
    "namespace": "default"
  }
}

[update agent labels value]
Update agent.yml adding labels.AdminHangcheckReadWrite=true .
Restart agent.

[wait check will run using new labels value]
Before scheduler is shutting down, this check is not use adminhangcheckreadwrite=true .
After scheduler restart, this check use adminhangcheckreadwrite=true .
(systemctl reload sensu-backend.service or automatic restart after logging " shutting down scheduler")

Do you have any knowledge about this phenomenon?

  1. What have you already tried? Please include links to gists and/or code blocks (if relatively small)
    agent/backend restart/reload.
    backend version up 5.13.0 to 5.13.2

  2. Tell us about your setup, this should include OS, version of Sensu, version of Sensu components (redis, rabbitmq), plugin versions (if applicable), anything special about your setup such as an airgapped network or strict ACLs
    install rpm and make agent.yml and backend.yml

  3. Is there a Github issue related to your issue?
    May be these issue is related.
    https://github.com/sensu/sensu-go/issues/3156
    https://github.com/sensu/sensu-go/issues/2926
    https://github.com/sensu/sensu-go/issues/3168

  4. Is there anything else that can help us effectively help you?

2

These are backends logs.
backend1

Oct 07 18:37:31 BACKEND1 sensu-backend[20521]: {"component":"keepalived","entity":"ENTITY1","is_leader":"true","level":"warning","msg":"keepalive timed out","namespace":"default","previous_status":"dead","status":"dead","time":"2019-10-07T18:37:31+09:00"}
Oct 07 18:41:09 BACKEND1 sensu-backend[20521]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T18:41:09+09:00"}
Oct 07 18:44:54 BACKEND1 sensu-backend[20521]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T18:44:54+09:00"}
Oct 07 18:47:36 BACKEND1 sensu-backend[20521]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T18:47:36+09:00"}
Oct 07 18:47:55 BACKEND1 sensu-backend[20521]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T18:47:55+09:00"}
Oct 07 18:48:22 BACKEND1 sensu-backend[20521]: {"component":"keepalived","entity":"ENTITY2","is_leader":"true","level":"warning","msg":"keepalive timed out","namespace":"default","previous_status":"alive","status":"dead","time":"2019-10-07T18:48:22+09:00"}
Oct 07 18:48:22 BACKEND1 sensu-backend[20521]: {"component":"keepalived","entity":"ENTITY3","is_leader":"true","level":"warning","msg":"keepalive timed out","namespace":"default","previous_status":"alive","status":"dead","time":"2019-10-07T18:48:22+09:00"}
Oct 07 18:51:01 BACKEND1 sensu-backend[20521]: {"addr":"IPADDR:44264","agent":"ENTITY4","component":"agentd","level":"warning","msg":"stopping session","recv error":"Connection closed: websocket: close 1001 (going away): bye","time":"2019-10-07T18:51:01+09:00"}
Oct 07 18:53:18 BACKEND1 sensu-backend[20521]: {"component":"eventd","error":"rpc error: code = ResourceExhausted desc = trying to send message larger than max (9216318 vs. 2097152)","level":"error","msg":"eventd - error handling event","time":"2019-10-07T18:53:18+09:00"}
Oct 07 18:54:16 BACKEND1 sensu-backend[20521]: {"component":"eventd","error":"rpc error: code = ResourceExhausted desc = trying to send message larger than max (4781845 vs. 2097152)","level":"error","msg":"eventd - error handling event","time":"2019-10-07T18:54:16+09:00"}
Oct 07 19:02:02 BACKEND1 sensu-backend[20521]: {"addr":"IPADDR:36748","agent":"ENTITY5","component":"agentd","level":"warning","msg":"stopping session","recv error":"Connection closed: websocket: close 1001 (going away): bye","time":"2019-10-07T19:02:02+09:00"}
Oct 07 19:02:03 BACKEND1 sensu-backend[20521]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T19:02:03+09:00"}
Oct 07 19:02:30 BACKEND1 sensu-backend[20521]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T19:02:30+09:00"}
Oct 07 19:02:31 BACKEND1 sensu-backend[20521]: {"addr":"IPADDR:42676","agent":"ENTITY6","component":"agentd","level":"warning","msg":"stopping session","recv error":"Connection error: read tcp IPADDR:8081-\u003eIPADDR:42676: read: connection timed out","time":"2019-10-07T19:02:31+09:00"}

backend2

Oct 07 18:35:53 BACKEND2p sensu-backend[32606]: {"component":"keepalived","entity":"ENTITY9","is_leader":"true","level":"warning","msg":"keepalive timed out","namespace":"default","previous_status":"dead","status":"dead","time":"2019-10-07T18:35:53+09:00"}
Oct 07 18:41:09 BACKEND2p sensu-backend[32606]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T18:41:09+09:00"}
Oct 07 18:44:54 BACKEND2p sensu-backend[32606]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T18:44:54+09:00"}
Oct 07 18:47:36 BACKEND2p sensu-backend[32606]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T18:47:36+09:00"}
Oct 07 18:47:55 BACKEND2p sensu-backend[32606]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T18:47:55+09:00"}
Oct 07 18:53:15 BACKEND2p sensu-backend[32606]: {"component":"eventd","error":"rpc error: code = ResourceExhausted desc = trying to send message larger than max (9188825 vs. 2097152)","level":"error","msg":"eventd - error handling event","time":"2019-10-07T18:53:15+09:00"}
Oct 07 18:53:19 BACKEND2p sensu-backend[32606]: {"component":"eventd","error":"rpc error: code = ResourceExhausted desc = trying to send message larger than max (9257158 vs. 2097152)","level":"error","msg":"eventd - error handling event","time":"2019-10-07T18:53:19+09:00"}
Oct 07 18:54:13 BACKEND2p sensu-backend[32606]: {"component":"eventd","error":"rpc error: code = ResourceExhausted desc = trying to send message larger than max (4791060 vs. 2097152)","level":"error","msg":"eventd - error handling event","time":"2019-10-07T18:54:13+09:00"}
Oct 07 18:54:14 BACKEND2p sensu-backend[32606]: {"component":"eventd","error":"rpc error: code = ResourceExhausted desc = trying to send message larger than max (4723359 vs. 2097152)","level":"error","msg":"eventd - error handling event","time":"2019-10-07T18:54:14+09:00"}
Oct 07 19:02:01 BACKEND2p sensu-backend[32606]: {"addr":"IPADDR:55924","agent":"ENTITY7","component":"agentd","level":"warning","msg":"stopping session","recv error":"Connection closed: websocket: close 1001 (going away): bye","time":"2019-10-07T19:02:01+09:00"}
Oct 07 19:02:03 BACKEND2p sensu-backend[32606]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T19:02:03+09:00"}
Oct 07 19:02:28 BACKEND2p sensu-backend[32606]: {"addr":"IPADDR:42206","agent":"ENTITY2","component":"agentd","level":"warning","msg":"stopping session","recv error":"Connection error: read tcp IPADDR:8081-\u003eIPADDR:42206: read: connection timed out","time":"2019-10-07T19:02:28+09:00"}
Oct 07 19:02:30 BACKEND2p sensu-backend[32606]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T19:02:30+09:00"}
Oct 07 19:02:47 BACKEND2p sensu-backend[32606]: {"addr":"IPADDR:33904","agent":"ENTITY3","component":"agentd","level":"warning","msg":"stopping session","recv error":"Connection error: read tcp IPADDR:8081-\u003eIPADDR:33904: read: connection timed out","time":"2019-10-07T19:02:47+09:00"}
Oct 07 19:05:01 BACKEND2p sensu-backend[32606]: {"addr":"IPADDR:36928","agent":"ENTITY5","component":"agentd","level":"warning","msg":"stopping session","recv error":"Connection closed: websocket: close 1001 (going away): bye","time":"2019-10-07T19:05:01+09:00"}
Oct 07 19:05:02 BACKEND2p sensu-backend[32606]: {"addr":"IPADDR:39296","agent":"ENTITY8","component":"agentd","level":"warning","msg":"stopping session","recv error":"Connection closed: websocket: close 1001 (going away): bye","time":"2019-10-07T19:05:02+09:00"}

backend3

Oct 07 18:41:09 BACKEND3 sensu-backend[23287]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T18:41:09+09:00"}
Oct 07 18:44:54 BACKEND3 sensu-backend[23287]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T18:44:54+09:00"}
Oct 07 18:47:36 BACKEND3 sensu-backend[23287]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T18:47:36+09:00"}
Oct 07 18:47:55 BACKEND3 sensu-backend[23287]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T18:47:55+09:00"}
Oct 07 18:48:22 BACKEND3 sensu-backend[23287]: {"component":"keepalived","entity":"ENTITY6","is_leader":"true","level":"warning","msg":"keepalive timed out","namespace":"default","previous_status":"alive","status":"dead","time":"2019-10-07T18:48:22+09:00"}
Oct 07 19:02:01 BACKEND3 sensu-backend[23287]: {"addr":"IPADDR:39118","agent":"ENTITY8","component":"agentd","level":"warning","msg":"stopping session","recv error":"Connection closed: websocket: close 1001 (going away): bye","time":"2019-10-07T19:02:01+09:00"}
Oct 07 19:02:03 BACKEND3 sensu-backend[23287]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T19:02:03+09:00"}
Oct 07 19:02:30 BACKEND3 sensu-backend[23287]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"check-mysql-hang-admin","namespace":"default","scheduler_type":"round-robin interval","time":"2019-10-07T19:02:30+09:00"}
Oct 07 19:05:01 BACKEND3 sensu-backend[23287]: {"addr":"IPADDR:56104","agent":"ENTITY7","component":"agentd","level":"warning","msg":"stopping session","recv error":"Connection closed: websocket: close 1001 (going away): bye","time":"2019-10-07T19:05:01+09:00"}
3

This is event information with one entity.

{
  "timestamp": 1570443619,
  "entity": {
    "entity_class": "agent",
    "system": {
      "hostname": "ENTITYNAME",
      "os": "****",
      "platform": "****",
      "platform_family": "****",
      "platform_version": "****",
      "network": {
        "interfaces": [
          {
            "name": "lo",
            "addresses": [
              "127.0.0.1/8"
            ]
          },
          {
            "name": "eth0",
            "mac": "****",
            "addresses": [
              "****"
            ]
          },
          {
            "name": "eth1",
            "mac": "****",
            "addresses": [
              "****"
            ]
          }
        ]
      },
      "arch": "****"
    },
    "subscriptions": [
      "entity-type:agent",
      "dbaas-cluster:ENTITYNAME",
      "pxc-node-number:1",
      "server:DBaaS",
      "hypervisor:HVHOSTNAME",
      "availability-zone:AZNAME",
      "os-image:OSIMAGE",
      "cluster-task-status:TASKSTATUS",
      "entity:ENTITYNAME",
      "entity:ENTITYNAME"
    ],
    "last_seen": 1570443601,
    "deregister": false,
    "deregistration": {},
    "user": "agent",
    "redact": [
      "password",
      "passwd",
      "pass",
      "api_key",
      "api_token",
      "access_key",
      "secret_key",
      "private_key",
      "secret"
    ],
    "metadata": {
      "name": "ENTITYNAME",
      "namespace": "default",
      "labels": {
        "adminhangcheckreadwrite": "true"
      }
    }
  },
  "check": {
    "command": "/opt/mysqlpf-sensu-check-scripts/bin/check-mysql hang --password=PASSWORD --user=USER --host=ENTITYNAME --readwrite=true --retry=5 --interval=10 --subdue nil --check-name \"check-mysql-hang-admin\"",
    "handlers": [
      "admin-email"
    ],
    "high_flap_threshold": 9,
    "interval": 60,
    "low_flap_threshold": 0,
    "publish": true,
    "runtime_assets": null,
    "subscriptions": [
      "sensu-backend"
    ],
    "proxy_entity_name": "ENTITYNAME",
    "check_hooks": null,
    "stdin": false,
    "subdue": null,
    "ttl": 0,
    "timeout": 0,
    "proxy_requests": {
      "entity_attributes": [
        "entity.subscriptions.indexOf('server:DBaaS') >= 0|| entity.subscriptions.indexOf('entity-type:lbvip') >= 0"
      ],
      "splay": true,
      "splay_coverage": 90
    },
    "round_robin": true,
    "duration": 0.151803777,
    "executed": 1570443619,
    "history": [
      {
        "status": 0,
        "executed": 1570442320
      },
      {
        "status": 0,
        "executed": 1570442380
      },
      {
        "status": 0,
        "executed": 1570442440
      },
      {
        "status": 0,
        "executed": 1570442500
      },
      {
        "status": 0,
        "executed": 1570442586
      },
      {
        "status": 0,
        "executed": 1570442646
      },
      {
        "status": 0,
        "executed": 1570442706
      },
      {
        "status": 0,
        "executed": 1570442766
      },
      {
        "status": 0,
        "executed": 1570442826
      },
      {
        "status": 0,
        "executed": 1570442886
      },
      {
        "status": 0,
        "executed": 1570442946
      },
      {
        "status": 0,
        "executed": 1570443006
      },
      {
        "status": 0,
        "executed": 1570443125
      },
      {
        "status": 0,
        "executed": 1570443185
      },
      {
        "status": 0,
        "executed": 1570443245
      },
      {
        "status": 0,
        "executed": 1570443306
      },
      {
        "status": 0,
        "executed": 1570443366
      },
      {
        "status": 0,
        "executed": 1570443426
      },
      {
        "status": 0,
        "executed": 1570443498
      },
      {
        "status": 0,
        "executed": 1570443558
      },
      {
        "status": 0,
        "executed": 1570443619
      }
    ],
    "issued": 1570443619,
    "output": "MySQL Hang OK: Connect is OK.\n---\nSelect is OK.\n---\nInsert is OK.\n---\nUpdate is OK.\n---\nDelete is OK.\n---\n",
    "state": "passing",
    "status": 0,
    "total_state_change": 0,
    "last_ok": 1570443619,
    "occurrences": 3296,
    "occurrences_watermark": 3296,
    "output_metric_format": "",
    "output_metric_handlers": null,
    "env_vars": null,
    "metadata": {
      "name": "check-mysql-hang-admin",
      "namespace": "default"
    }
  },
  "metadata": {
    "namespace": "default"
  }
}

The check executed 1570443006 and next check executed 1570443125.
It is not interval 1min and some other histories show interval is not 1min.

4

hey!
We just released 3.14. Are you able to retest with the new version?

5

@jspaleta I tried 5.14, and these problems were cleared.
Thank you for you suggestion.