Error starting etcd: wal: max entry size limit exceeded

Description of your problem

I was trying to connect my sensuGo instance with SumoLogic Following this link
I had a couple problems since I don’t have a license for sensu-enterprise. At the end, the integration seemed to be working but now the sensu-backend service is down.
When I check the logs I can see an error under the “sensu-enterprise” component saying that the etcd value is not enough. Or at least that’s what I understand. The problem is that I’m not sure where to modify the value. If I do it over the backend.yml file, nothing really happen.

Description of steps you’ve taken to attempt to solve the issue

Environmental information

Operating system information

  • Ubuntu 20.04.5 LTS

Package versions

  • Sensu Go:
root@sensu2:~# sensu-backend version
sensu-backend version 6.9.1+ee, enterprise edition, build 3a2641136a1e0fbf0bc0f237e5abff203c106a8d, built 2022-12-01T16:53:21Z, built with go1.17.6
  • Etcd:
troot@sensu2:~# etcd version

Command 'etcd' not found, but can be installed with:

apt install etcd-server

root@sensu2:~#
  • PostgreSQL:
root@sensu2:~# postgres version

Command 'postgres' not found, did you mean:

  command 'postgrey' from deb postgrey (1.36-5.1)

Try: apt install <deb name>

root@sensu2:~#

Plugin information

Service logs, configuration, and environment variables

Log File:

root@sensu2:~# sensu-backend start --debug -c /etc/sensu/backend.yml
{"component":"etcd","level":"info","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["http://127.0.0.1:2380"],"time":"2023-01-10T13:47:45-05:00"}
{"component":"etcd","level":"info","caller":"embed/etcd.go:132","msg":"configuring client listeners","listen-client-urls":["http://127.0.0.1:2379"],"time":"2023-01-10T13:47:45-05:00"}
{"component":"etcd","level":"info","caller":"embed/etcd.go:306","msg":"starting an etcd server","etcd-version":"3.5.5","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.17.6","go-os":"linux","go-arch":"amd64","max-cpu-set":1,"max-cpu-available":1,"member-initialized":true,"name":"default","data-dir":"/var/lib/sensu/sensu-backend/etcd/data","wal-dir":"/var/lib/sensu/sensu-backend/etcd/wal","wal-dir-dedicated":"/var/lib/sensu/sensu-backend/etcd/wal","member-dir":"/var/lib/sensu/sensu-backend/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"3s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://127.0.0.1:2380"],"listen-peer-urls":["http://127.0.0.1:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://127.0.0.1:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":4294967296,"max-request-bytes":64000000,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"revision","auto-compaction-retention":"2ns","auto-compaction-interval":"2ns","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s","time":"2023-01-10T13:47:45-05:00"}
{"component":"etcd","level":"warning","caller":"etcdserver/server.go:343","msg":"exceeded recommended request limit","max-request-bytes":64000000,"max-request-size":"64 MB","recommended-request-bytes":10485760,"recommended-request-size":"10 MB","time":"2023-01-10T13:47:45-05:00"}
{"component":"etcd","level":"info","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/sensu/sensu-backend/etcd/data/member/snap/db","took":"4.671785ms","time":"2023-01-10T13:47:45-05:00"}
{"component":"etcd","level":"info","caller":"embed/etcd.go:371","msg":"closing etcd server","name":"default","data-dir":"/var/lib/sensu/sensu-backend/etcd/data","advertise-peer-urls":["http://127.0.0.1:2380"],"advertise-client-urls":["http://localhost:2379"],"time":"2023-01-10T13:47:46-05:00"}
{"component":"etcd","level":"info","caller":"embed/etcd.go:373","msg":"closed etcd server","name":"default","data-dir":"/var/lib/sensu/sensu-backend/etcd/data","advertise-peer-urls":["http://127.0.0.1:2380"],"advertise-client-urls":["http://localhost:2379"],"time":"2023-01-10T13:47:46-05:00"}
{"component":"sensu-enterprise","error":"error starting etcd: wal: max entry size limit exceeded, recBytes: 10376, fileSize(64000000) - offset(63991808) - padBytes(0) = entryLimit(8192)","level":"fatal","msg":"error executing sensu-backend","time":"2023-01-10T13:47:46-05:00"}
root@sensu2:~#

Configuration File:

##
# backend configuration
##
#labels:
#  example_key: "example value"
#annotations:
#  example/key: "example value"
#assets-burst-limit: 100
#assets-rate-limit: 1.39
#cache-dir: "/var/cache/sensu/sensu-backend"
#config-file: "/etc/sensu/backend.yml"
#debug: false
#deregistration-handler: "example_handler"
log-level: "debug" #available log levels: panic, fatal, error, warn, info, debug, trace
state-dir: "/var/lib/sensu/sensu-backend"
#require-fips: false
#require-openssl: false
#eventd-buffer-size: 100
#eventd-workers: 100
#keepalived-buffer-size: 100
#keepalived-workers: 100
#pipelined-buffer-size: 100
#pipelined-workers: 100

##
# api configuration
##
api-listen-address: "[::]:8080" #listen on all IPv4 and IPv6 addresses
api-request-limit: 512000
api-url: "http://localhost:8080"


##
# tls configuration
##
#agent-auth-cert-file: /path/to/tls/backend-1.pem
#agent-auth-crl-urls: http://localhost/CARoot.crl
#agent-auth-key-file: /path/to/tls/backend-1-key.pem
#agent-auth-trusted-ca-file: /path/to/tls/ca.pem
#agent-burst-limit: null
#agent-host: "[::]" #listen on all IPv4 and IPv6 addresses
#agent-port: 8081
#agent-rate-limit: null
#cert-file: "/etc/sensu/tls/letsencrypt/fullchain.pem"
#insecure-skip-tls-verify: false
#jwt-private-key-file: /path/to/key/private.pem
#jwt-public-key-file: /path/to/key/public.pem
#key-file: "/etc/sensu/tls/letsencrypt/privkey.pem"
#trusted-ca-file: "/path/to/tls/ca.pem"
#dashboard-cert-file: "/etc/sensu/tls/fullchain.pem"
#dashboard-host: "[::]"
#dashboard-key-file:"/etc/sensu/tls/privkey.pem"
#dashboard-port: 3000


##
# etcd datastore configuration
##
#etcd-advertise-client-urls:
#  - http://localhost:2378
#  - http://localhost:2379
#etcd-cert-file: "/path/to/tls/backend-1.pem"
#etcd-cipher-suites:
#  - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
#  - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
#  - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
#  - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
#  - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
#  - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
#etcd-client-cert-auth: false
#etcd-client-urls:
#  - https://10.0.0.1:2379
#  - https://10.1.0.1:2379
#etcd-discovery:
#  - https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
#etcd-discovery-srv:
#  - example.org
#etcd-initial-advertise-peer-urls:
#  - https://10.0.0.1:2380
#  - https://10.1.0.1:2380
#etcd-initial-cluster: "backend-0=https://10.0.0.1:2380,backend-1=https://10.1.0.1:2380,backend-2=https://10.2.0.1:2380"
#etcd-initial-cluster-state: "new"
#etcd-initial-cluster-token: "unique_token_for_this_cluster"
#etcd-key-file: "/path/to/tls/backend-1-key.pem"
#etcd-listen-client-urls:
#  - https://10.0.0.1:2379
#  - https://10.1.0.1:2379
#etcd-listen-peer-urls:
#  - https://10.0.0.1:2380
#  - https://10.1.0.1:2380
#etcd-name: "backend-0"
#etcd-peer-cert-file: "/path/to/tls/backend-1.pem"
#etcd-peer-client-cert-auth: false
#etcd-peer-key-file: "/path/to/tls/backend-1-key.pem"
#etcd-peer-trusted-ca-file: "./ca.pem"
#etcd-trusted-ca-file: "./ca.pem"
#no-embed-etcd: false
#etcd-election-timeout: 1000
etcd-heartbeat-interval: 100
etcd-max-request-bytes: 64000000 #1572864
etcd-quota-backend-bytes: 4294967296

@jeff-fongo I’ve been looking over the logs you provided, but I’m a bit confused–you’re using the sensu-backend start command instead of your system’s process manager (i.e., systemctl start sensu-backend). I’m assuming that the system’s journal would provide a bit more information when you attempt to start the backend.

To clarify here, the error you’re showing in your post isn’t specific to Sensu Go Enterprise, but rather the etcd component.

I will point out that adjusting the max request bytes in your backend config is something that should be done with caution (see the warning here: Backend reference - Sensu Docs and here: System limits | etcd)

Hey there, one thing to note: By adjusting this value:

etcd-max-request-bytes: 64000000

You’ve set your request size to the size of the entire Etcd write ahead log (WAL). That should be set to the default value, which should allow the backend to start back up.

HI aaronsachs,
Thanks for your comment. I tried doing this and got an error:

root@sensu2:~# sensu-backend init --etcd-max-request-bytes 64000000
{"component":"sensu-enterprise","error":"both cluster-admin-username and cluster-admin-password are required to be set (or an API key)","level":"fatal","msg":"error executing sensu-backend","time":"2023-01-12T20:26:41-05:00"}
root@sensu2:~#

and I do try to start the service via systemctl but it starts and then fail:

toot@sensu2:~# sudo systemctl status sensu-backend
● sensu-backend.service - The Sensu Backend service.
     Loaded: loaded (/lib/systemd/system/sensu-backend.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-01-12 20:26:59 EST; 484ms ago
   Main PID: 7890 (sensu-backend)
      Tasks: 6 (limit: 2339)
     Memory: 52.1M
     CGroup: /system.slice/sensu-backend.service
             └─7890 /usr/sbin/sensu-backend start -c /etc/sensu/backend.yml

Jan 12 20:26:59 sensu2 systemd[1]: sensu-backend.service: Scheduled restart job, restart counter is at 4.
Jan 12 20:26:59 sensu2 systemd[1]: Stopped The Sensu Backend service..
Jan 12 20:26:59 sensu2 systemd[1]: Started The Sensu Backend service..
Jan 12 20:26:59 sensu2 sensu-backend[7890]: {"component":"etcd","level":"info","caller":"embed/etcd.go:124","msg":"configuring peer listeners",">
Jan 12 20:26:59 sensu2 sensu-backend[7890]: {"component":"etcd","level":"info","caller":"embed/etcd.go:132","msg":"configuring client listeners">
Jan 12 20:26:59 sensu2 sensu-backend[7890]: {"component":"etcd","level":"info","caller":"embed/etcd.go:306","msg":"starting an etcd server","etc>
Jan 12 20:26:59 sensu2 sensu-backend[7890]: {"component":"etcd","level":"warning","caller":"etcdserver/server.go:343","msg":"exceeded recommende>
Jan 12 20:26:59 sensu2 sensu-backend[7890]: {"component":"etcd","level":"info","caller":"etcdserver/backend.go:81","msg":"opened backend db","pa>
Jan 12 20:27:00 sensu2 sensu-backend[7890]: {"component":"etcd","level":"info","caller":"embed/etcd.go:371","msg":"closing etcd server","name":">
Jan 12 20:27:00 sensu2 sensu-backend[7890]: {"component":"etcd","level":"info","caller":"embed/etcd.go:373","msg":"closed etcd server","name":"d>
Jan 12 20:27:00 sensu2 sensu-backend[7890]: {"component":"sensu-enterprise","error":"error starting etcd: wal: max entry size limit exceeded, re>
Jan 12 20:27:00 sensu2 systemd[1]: sensu-backend.service: Main process exited, code=exited, status=1/FAILURE
Jan 12 20:27:00 sensu2 systemd[1]: sensu-backend.service: Failed with result 'exit-code'.
root@sensu2:~# sudo systemctl status sensu-backend
● sensu-backend.service - The Sensu Backend service.
     Loaded: loaded (/lib/systemd/system/sensu-backend.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2023-01-12 20:27:00 EST; 13s ago
    Process: 7890 ExecStart=/usr/sbin/sensu-backend start -c /etc/sensu/backend.yml (code=exited, status=1/FAILURE)
   Main PID: 7890 (code=exited, status=1/FAILURE)

Jan 12 20:27:00 sensu2 systemd[1]: sensu-backend.service: Main process exited, code=exited, status=1/FAILURE
Jan 12 20:27:00 sensu2 systemd[1]: sensu-backend.service: Failed with result 'exit-code'.
Jan 12 20:27:00 sensu2 systemd[1]: sensu-backend.service: Scheduled restart job, restart counter is at 5.
Jan 12 20:27:00 sensu2 systemd[1]: Stopped The Sensu Backend service..
Jan 12 20:27:00 sensu2 systemd[1]: sensu-backend.service: Start request repeated too quickly.
Jan 12 20:27:00 sensu2 systemd[1]: sensu-backend.service: Failed with result 'exit-code'.
Jan 12 20:27:00 sensu2 systemd[1]: Failed to start The Sensu Backend service..
root@sensu2:~#

And actually now that I check again the sense version, it does says enterprise edition which I don’t have or pay for. I installed the Sensu GO version. I’m not sure if by trying the integration with Sumo Logic the license change to enterprise.

I’m not sure how to go from here, since I don’t have a cluster username or password.

Hey there,

So to clarify on:

it does says enterprise edition which I don’t have or pay for.

“Enterprise” is our commercially packaged version–eg, what you download from sensu.io or docs.sensu.io and is free for under 100 entities (see Sensu | Standalone Sensu Pricing).

“OSS” is the version that you compile yourself from GitHub - sensu/sensu-go: Simple. Scalable. Multi-cloud monitoring..

Trying Sensu Plus/the Sumo Logic integration doesn’t change the license or what version you have.

To try and get back at the original issue, was the instance running just fine for a while, or did this happen right away? Also, what’s the full output from journalctl -u sensu-backend? ← That command would be the recommended way of grabbing all the logs (please make sure you redact any sensitive info from those logs).

Hi aaron, here is part of the output when I run that command.
Since I was not able to figure it out, I used a backup image to restore the server. Then, I used the catalog to enable the Disk and Network monitoring. The next day the service was down once again.
At first, the log file inside /var/log/sensu/ was huge and the server run out of space, I deleted that file and tried to restart the service but didn’t work. I’m still getting the same error as before. Not sure what to do.
Maybe the output of those tasks from the catalog are too much?

Jan 24 09:21:25 sensu2 systemd[1]: Started The Sensu Backend service..
Jan 24 09:21:25 sensu2 sensu-backend[7075]: {"component":"etcd","level":"info","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["http://127.0.0.1:2380"],"time":"2023-01-24T09:21:25-05:00"}
Jan 24 09:21:25 sensu2 sensu-backend[7075]: {"component":"etcd","level":"info","caller":"embed/etcd.go:132","msg":"configuring client listeners","listen-client-urls":["http://127.0.0.1:2379"],"time":"2023-01-24T09:21:25-05:00"}
Jan 24 09:21:25 sensu2 sensu-backend[7075]: {"component":"etcd","level":"info","caller":"embed/etcd.go:306","msg":"starting an etcd server","etcd-version":"3.5.5","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.17.6","go-os":"linux","go-arch":"amd64","max-cpu-set":1,"max-cpu-available":1,"member-initialized":true,"name":"default","data-dir":"/var/lib/sensu/sensu-backend/etcd/data","wal-dir":"/var/lib/sensu/sensu-backend/etcd/wal","wal-dir-dedicated":"/var/lib/sensu/sensu-backend/etcd/wal","member-dir":"/var/lib/sensu/sensu-backend/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"300ms","election-timeout":"3s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://127.0.0.1:2380"],"listen-peer-urls":["http://127.0.0.1:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://127.0.0.1:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":4294967296,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"revision","auto-compaction-retention":"2ns","auto-compaction-interval":"2ns","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s","time":"2023-01-24T09:21:25-05:00"}
Jan 24 09:21:25 sensu2 sensu-backend[7075]: {"component":"etcd","level":"info","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/sensu/sensu-backend/etcd/data/member/snap/db","took":"44.027964ms","time":"2023-01-24T09:21:25-05:00"}
Jan 24 09:21:27 sensu2 sensu-backend[7075]: {"component":"etcd","level":"info","caller":"embed/etcd.go:371","msg":"closing etcd server","name":"default","data-dir":"/var/lib/sensu/sensu-backend/etcd/data","advertise-peer-urls":["http://127.0.0.1:2380"],"advertise-client-urls":["http://localhost:2379"],"time":"2023-01-24T09:21:27-05:00"}
Jan 24 09:21:27 sensu2 sensu-backend[7075]: {"component":"etcd","level":"info","caller":"embed/etcd.go:373","msg":"closed etcd server","name":"default","data-dir":"/var/lib/sensu/sensu-backend/etcd/data","advertise-peer-urls":["http://127.0.0.1:2380"],"advertise-client-urls":["http://localhost:2379"],"time":"2023-01-24T09:21:27-05:00"}
Jan 24 09:21:27 sensu2 sensu-backend[7075]: {"component":"sensu-enterprise","error":"error starting etcd: wal: max entry size limit exceeded, recBytes: 4416, fileSize(64000000) - offset(63997472) - padBytes(0) = entryLimit(2528)","level":"fatal","msg":"error executing sensu-backend","time":"2023-01-24T09:21:27-05:00"}
Jan 24 09:21:27 sensu2 systemd[1]: sensu-backend.service: Main process exited, code=exited, status=1/FAILURE
Jan 24 09:21:27 sensu2 systemd[1]: sensu-backend.service: Failed with result 'exit-code'.
Jan 24 09:21:28 sensu2 systemd[1]: sensu-backend.service: Scheduled restart job, restart counter is at 1.
Jan 24 09:21:28 sensu2 systemd[1]: Stopped The Sensu Backend service..
Jan 24 09:21:28 sensu2 systemd[1]: Started The Sensu Backend service..
Jan 24 09:21:28 sensu2 sensu-backend[7100]: {"component":"etcd","level":"info","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["http://127.0.0.1:2380"],"time":"2023-01-24T09:21:28-05:00"}
Jan 24 09:21:28 sensu2 sensu-backend[7100]: {"component":"etcd","level":"info","caller":"embed/etcd.go:132","msg":"configuring client listeners","listen-client-urls":["http://127.0.0.1:2379"],"time":"2023-01-24T09:21:28-05:00"}
Jan 24 09:21:28 sensu2 sensu-backend[7100]: {"component":"etcd","level":"info","caller":"embed/etcd.go:306","msg":"starting an etcd server","etcd-version":"3.5.5","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.17.6","go-os":"linux","go-arch":"amd64","max-cpu-set":1,"max-cpu-available":1,"member-initialized":true,"name":"default","data-dir":"/var/lib/sensu/sensu-backend/etcd/data","wal-dir":"/var/lib/sensu/sensu-backend/etcd/wal","wal-dir-dedicated":"/var/lib/sensu/sensu-backend/etcd/wal","member-dir":"/var/lib/sensu/sensu-backend/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"300ms","election-timeout":"3s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://127.0.0.1:2380"],"listen-peer-urls":["http://127.0.0.1:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://127.0.0.1:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":4294967296,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"revision","auto-compaction-retention":"2ns","auto-compaction-interval":"2ns","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s","time":"2023-01-24T09:21:28-05:00"}
Jan 24 09:21:28 sensu2 sensu-backend[7100]: {"component":"etcd","level":"info","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/sensu/sensu-backend/etcd/data/member/snap/db","took":"4.076141ms","time":"2023-01-24T09:21:28-05:00"}
Jan 24 09:21:29 sensu2 sensu-backend[7100]: {"component":"etcd","level":"info","caller":"embed/etcd.go:371","msg":"closing etcd server","name":"default","data-dir":"/var/lib/sensu/sensu-backend/etcd/data","advertise-peer-urls":["http://127.0.0.1:2380"],"advertise-client-urls":["http://localhost:2379"],"time":"2023-01-24T09:21:29-05:00"}
Jan 24 09:21:29 sensu2 sensu-backend[7100]: {"component":"etcd","level":"info","caller":"embed/etcd.go:373","msg":"closed etcd server","name":"default","data-dir":"/var/lib/sensu/sensu-backend/etcd/data","advertise-peer-urls":["http://127.0.0.1:2380"],"advertise-client-urls":["http://localhost:2379"],"time":"2023-01-24T09:21:29-05:00"}
Jan 24 09:21:29 sensu2 sensu-backend[7100]: {"component":"sensu-enterprise","error":"error starting etcd: wal: max entry size limit exceeded, recBytes: 4416, fileSize(64000000) - offset(63997472) - padBytes(0) = entryLimit(2528)","level":"fatal","msg":"error executing sensu-backend","time":"2023-01-24T09:21:29-05:00"}
Jan 24 09:21:29 sensu2 systemd[1]: sensu-backend.service: Main process exited, code=exited, status=1/FAILURE
Jan 24 09:21:29 sensu2 systemd[1]: sensu-backend.service: Failed with result 'exit-code'.
Jan 24 09:21:29 sensu2 systemd[1]: sensu-backend.service: Scheduled restart job, restart counter is at 2.
Jan 24 09:21:29 sensu2 systemd[1]: Stopped The Sensu Backend service..
Jan 24 09:21:29 sensu2 systemd[1]: Started The Sensu Backend service..

How can I adjust that value? I can’t find the place to do it.

Hey there, to answer this:

That’s in your backend.yml. I’m seeing that value set in your initial post:

As to the log file, my assumption is that the log file was the platform metrics, which is enabled by default (you can disable it by adding disable-platform-metrics: true to /etc/sensu/backend.yml). It could very well be that with the disk filling up, there may have been some etcd corruption.

The catalog would have 0 impact on the backend’s ability to run, or rather, adding an integration to your deployment wouldn’t lead to this sort of behavior. I think the platform metrics in this case would be the cause of the problem.