Very interesting, I’d love some help getting better logging into this. I’ll check my file system. I wonder if the fact that this entity runs out of a docker container is material to the issue.
Is this the official sensu-agent container image?
Gah…
If this was any week but right before summit I’d whip you up an agent binary real quick with a modified error message to include additional logging info to at least point you to the tmpfile on disk that is throwing the error.
But failing that, you should be able to log into the running container using docker exec or equivalent and look in /tmp/ to see what sensu-asset files are sitting there. The tmp files are only cleaned up if they successful unpack. If you see any sensu-asset* files in /tmp they are most likely left over from previous extraction errors. If you see multiple ones they could be the same asset repeatedly downloaded in fact.
How I would diagnose this…
If the container has access to sha512sum command you can get the sha512 of the tmpfile and compare it to your current asset catalog to determine which file it is without inspecting the contents. That’s probably the quickest. And if you have multiple sensu-asset* tmpfiles take the sha512 of all of them, to quickly see if they are really the same file downloaded and aborted repeatedly. if the container doesn’t have the sha512sum, I’d copy those files out of the container to where i could take the sha512sums.
If the sha512sums don’t match anything in your asset catelog (sensuctl asset list) that would suggest to me that maybe your running out of space on disk or there’s a network disconnect and we probably need to enhance our error logging to catch that. If the sha512sum’s match something in your asset catalog…then we need to go look at that specific file to make sure it its a supported archive type…regardless of what the file extension would have us believe…file extensions can lie.
-jef
It’s not the official container, didn’t know there was one. I could definitely switch to that and see if it makes any difference.
That aside, i went into the container and there’s absolutely nothing in /tmp. The asset does exist in /var/cache/sensu/sensu-agent//bin/apppool_monitor
The asset does work in that location and returns proper exit codes. We’ve tried destroying the container and recreating it with similar results. For now i’ve created a container volume to the docker host and i’ve put the asset manually in the mapped volume. I’ve changed the check to point to that mapped location in the container and it’s working fine this way. I’ll keep it this way until we can work on the debugging further. I’d love spending some time with you on adding debugging and solving this after the conference. I’m going to hashicon myself next week so i’ll be out of pocket until that’s concluded.
wow, stranger and stranger.
so /tmp is the default as per golang os.TmpDir but if you have TMPDIR set in the environment for some reason the agent will use that location instead.
yeah I think we need to instrument this with some diagnostic debug messages. If I spun up a static linux amd64 build of agent for you to grab with additional debugging output next week, could you wedge it into a test container image?
-jef
yup i could definitely do that.
Was this ever resolved? I see the GitHub issue is still open…
I don’t think this was ever resolved
Ok, thanks. I’ll see what we can do about prioritizing this one and getting it resolved.
Cheers!