I’m having a strange latency issue with the 3 items above.
Sensu - 0.16
Uchiwa - 0.32
Redis - 2.8.17
I’m using Sensu with Graphite and am calculating about 7K metrics in a 3 minute period and feeding the alarms directly to the Sensu UDP socket. This generates about 100 Alarms every cycle. What I’m seeing is that the API gradually slows down until times out in uchiwa, and while Uchiwa is trying to read the API, no other requests are serviced.
The strange thing is that a redis flushall resolves the issue, but it will reoccur within 10 cycles or so.
Any insight is appreciated.
I’ve not nailed down the exact cause but it definitely has something to do with the Uchiwa->sensu-api->redis pathway. I’ve mitigated with the below solution. Querying events can still be slow, but they don’t time out.
Reduced the default Uchiwa refresh from 5s to 30s, and adding a Redis slave instance (on the same server).
···
On Monday, November 17, 2014 10:01:31 AM UTC-6, N. Robison wrote:
I’m having a strange latency issue with the 3 items above.
Sensu - 0.16
Uchiwa - 0.32
Redis - 2.8.17
I’m using Sensu with Graphite and am calculating about 7K metrics in a 3 minute period and feeding the alarms directly to the Sensu UDP socket. This generates about 100 Alarms every cycle. What I’m seeing is that the API gradually slows down until times out in uchiwa, and while Uchiwa is trying to read the API, no other requests are serviced.
The strange thing is that a redis flushall resolves the issue, but it will reoccur within 10 cycles or so.
Any insight is appreciated.