- What have you already tried? See below.
- Sensu 5.11 on ubuntu 18.04, using the built-in etcd interface
- Is there a Github issue related to your issue? No
- Is there anything else that can help us effectively help you?
Error message: error getting assets for event: given file of format 'application/zip' does not appear valid
Check config: apppool_monitor --f5poool my.examplesite.com -appPool example -minNum 2 --suser {{.labels.salt_username}} --spass {{.labels.salt_password}} --iuser {{.labels.influx_username}} --ipass {{.labels.influx_password}}
Okay, here’s what I know so far. The asset does successfully pull to the entity and unpacks. This is true whether it’s a .tar or a .tar.gz file.
The asset can be run manually from /var/cache/sensu/sensu-agent//bin
The asset works if I change the check to point to a static location and not use runtime assets.
The entity is running 5.11 and so is the backend.
I have 5 other assets that are .tar.gz packages and they all pull-down and work properly.
This asset was packaged and deployed using a concourse pipeline however we’ve manually uploaded it to our repo server (Artifactory) with the same result.
We are not uploading to Artifactory with any MIME types. I did try to apply application/gzip to a tar.gz package of the asset and I’ve tried application/x-tar to a tar package of the asset.
The asset is a go asset built using amd64 arch type and GOOS = linux. (But since it works after compiled, this probably isn’t important)
I’ve seen in github that the expander.go section that likely is responsible for this error is expecting a mime type of application/gzip or application/x-tar but I haven’t actively set any properties on my assets and the only one that’s failing is this one.
1 Like
Can you run the file command on the file in question? Is there any chance it’s actually a zip archive with a .tar.gz extension?
Here’s an example:
$ file assets/filename.tar.gz
assets/filename.tar.gz: gzip compressed data, last modified: Thu Apr 4 15:40:35 2019, from Unix
…vs when I run file on a Zip archive that I have renamed to .tar.gz extension:
$ file filename.zip.tar.gz
filename.zip.tar.gz: Zip archive data, at least v2.0 to extract
I’m sure you’ve tried this already but it would help to rule this out. 
Actually, on unix systems, you could use the following command: file --mime-type -b asset.tar.gz, which will give you the exact mime type. E.g.
$ file --mime-type -b asset.zip
application/zip
1 Like
I could, but the file in the cache directory is not a tar.gz or a zip. it’s a .go
let me pull the file from the repo and see what it says.
1 Like
file --mime-type -b apppool_monitor_1.tar.gz
application/x-gzip
When i created the .tar file:
file --mime-type -b apppool_monitor.tar
application/x-tar
This didn’t seem to work either.
Thanks @Darth.Scrumlord
So the problem is that application/x-gzip & application/x-tar are not official media types.
In RFC 6713 - The 'application/zlib' and 'application/gzip' Media Types, it’s mentioned that:
Some applications have informally used media types such as
[…] application/x-gzip […] to describe data compressed with gzip.
Indeed, according to JFrog Help Center, the archives produced by Artifactory use these informal media types.
I’ve created this issue to support these informal media types. In the meantime, you might want to look at Artifactory to understand if these media types are configurable or maybe create those archives in some other way.
Would you recommend i create these archives just using the linux package zip? Because it seems, at least with my distro, tar -xcvf is creating the x-gzip media type.
If i create it with the zip package they come back as application/zip. Can I assume expander.go will handle this file properly?
Well, I tried zip. When I create the archive using zip -r archive.zip /path/to/files and run file --mime-type -b archive.zip I get a return that the archive is application/zip.
I push that up to Artifactory and set the URL and SHA to the asset in sensuctl. I get the same message. I’m at a loss here. The binary gets unpacked and I can use it on the file system in /var/cache/sensu/sensu-agent. So the expander is working in some capacity…
I found where you can configure MIME Types in Artifactory and we’re looking into that but I honestly don’t have much hope that it will help; Especially when I have multiple assets pulling from Artifactory that are tar.gz, that show a MIME Type of x-gzip. Makes me think there’s something else wrong here.
Artifactory is configured to set the content-type for tar to x-tar, tar.gz to x-gzip and zip to zip.
Hi @Darth.Scrumlord,
I’ve never used Artifactory so I’m not sure if it does anything specific behind the scenes or whatever.
So does zip archives work for you? If so I’d recommend sticking to those until https://github.com/sensu/sensu-go/issues/3157 is released.
1 Like
Zip archive didn’t work for this. But I can’t test anymore right now.
Hey,
I believe if you configured our Artifactory to explicitly use the application/gzip for the gzip compressed tarballs assets you have uploaded this should work as a work around for now. Artifactory according to my reading of the available docs lets you explicitly set mimetypes to override the defaults it uses.
Beyond that… I think the x- mimetypes are in general problematic as they are not part of the agreed on RFC standards. I think we’ll need to expose a tunable that allows people to map which x- mimetypes should be treated as tar versus gzip compressed tar to cover all possible corner cases. In reality I could have something like application/x-tarball-of-doom and have it be equally as valid as x-targz from an RFC sense. Sensu agent and backend will probably both need a tunable or the asset definition extended to allow for user defined mimetypes to support.
1 Like
If that’s the case, shouldn’t application/zip work? It’s RFC compliant. I have artifactory mapping .zip to application/zip and the file was created as application/zip. It downloads and unpacks manually or via the asset engine as application/zip. However, I still get the error.
Well, this gets a bit technical… while RFC complaint…zip is a different format than gzip…and as of now assets are documented as being tar with optional gzip.
zips would potentially need a different unpacker codebase than what is currently in use, might not be as simple as just conditionally allowing the zip type to be processed with the existing unpacker.
As of now we don’t have documented support for assets compressed with zip (just tar… optionally compressed with gzip). So as of now I have to assume based on the documentation that the asset unpacker code in use can only interact with tar and gzip compressed tar.
Adding support for zip might be an enhancement request with some merit as zip is pretty popular especially on windows systems. Just need to figure out how hard this would be. If the underlying unpacking golang code can unpack zips…that’s should be an easy change, else have to identify new code to integrate.
I’ve started to look at this again. After re-reading through this thread It still doesn’t make sense to me that if the expander.go can’t deal with the informal type x-gzip why is the file actually being unpacked into /var/cache/sensu/sensu-agent ?
Hey!
Hmm that definitely seems odd. My reading the relevant code, the file should not extract if you see that error message.
Were you at some point in your diagnose testing a different mime type or manual extraction? The unpacked directory could be left over from a diagnostic experimental change.
-jef
I’ve tried to override the property from the download side and that hasn’t worked. Going to try upload next. If this fails we’ll change the mimetype xml for artifactory but none of it makes sense. At least 6 other assets are delivered this same way, are compiled go binaries and they work fine. It’s like something is wrong with the asset db but I’ve destroyed it and recreated.
Sensu version 5.11.1
hmm…
you know what we need here, is some slightly better logging information so we can get the name of the tmpFile throwing the error so we can be sure we know which file is the problem. I think the error your are seeing is not from the asset you think it is from.
So the general order of operations is the agent will download the asset into the temp directory… by default i believe that is /tmp with a filename prefix of sensu-asset
followed by some random numbers.
The asset manager then infers its mimetype from reading the file header on disk and will attempt to expand or throw an error.
If it successfully expands it will clean up the tempFile, if it fails to expand the tempFile should still be on the system for you to inspect.
Here’s the relevant matching logic for the packaging being used:
https://github.com/h2non/filetype/blob/v1.0.10/matchers/archive.go
@Darth.Scrumlord to be as clear as I can… looking at the codebase right now and tracking the logic i’m seeing… the Mimetype is inferred from the first 262 bytes of the tmp file after the asset is downloaded. Whatever you are doing on your server to muck with mime types i believe is immaterial. I think i led you astray a little bit earlier, assuming the mimetype was being picked up from the url.