Sensu asset file format is invalid error

  1. What have you already tried? See below.
  2. Sensu 5.11 on ubuntu 18.04, using the built-in etcd interface
  3. Is there a Github issue related to your issue? No
  4. Is there anything else that can help us effectively help you?

Error message: error getting assets for event: given file of format 'application/zip' does not appear valid

Check config: apppool_monitor --f5poool my.examplesite.com -appPool example -minNum 2 --suser {{.labels.salt_username}} --spass {{.labels.salt_password}} --iuser {{.labels.influx_username}} --ipass {{.labels.influx_password}}

Okay, here’s what I know so far. The asset does successfully pull to the entity and unpacks. This is true whether it’s a .tar or a .tar.gz file.

The asset can be run manually from /var/cache/sensu/sensu-agent//bin

The asset works if I change the check to point to a static location and not use runtime assets.

The entity is running 5.11 and so is the backend.

I have 5 other assets that are .tar.gz packages and they all pull-down and work properly.

This asset was packaged and deployed using a concourse pipeline however we’ve manually uploaded it to our repo server (Artifactory) with the same result.

We are not uploading to Artifactory with any MIME types. I did try to apply application/gzip to a tar.gz package of the asset and I’ve tried application/x-tar to a tar package of the asset.

The asset is a go asset built using amd64 arch type and GOOS = linux. (But since it works after compiled, this probably isn’t important)

I’ve seen in github that the expander.go section that likely is responsible for this error is expecting a mime type of application/gzip or application/x-tar but I haven’t actively set any properties on my assets and the only one that’s failing is this one.

1 Like

Can you run the file command on the file in question? Is there any chance it’s actually a zip archive with a .tar.gz extension?

Here’s an example:

$ file assets/filename.tar.gz
assets/filename.tar.gz: gzip compressed data, last modified: Thu Apr  4 15:40:35 2019, from Unix

…vs when I run file on a Zip archive that I have renamed to .tar.gz extension:

$ file filename.zip.tar.gz
filename.zip.tar.gz: Zip archive data, at least v2.0 to extract

I’m sure you’ve tried this already but it would help to rule this out. :slightly_smiling_face:

Actually, on unix systems, you could use the following command: file --mime-type -b asset.tar.gz, which will give you the exact mime type. E.g.

$ file --mime-type -b asset.zip
application/zip
1 Like

I could, but the file in the cache directory is not a tar.gz or a zip. it’s a .go
let me pull the file from the repo and see what it says.

1 Like
file --mime-type -b apppool_monitor_1.tar.gz
application/x-gzip

When i created the .tar file:

file --mime-type -b apppool_monitor.tar
application/x-tar

This didn’t seem to work either.

Thanks @Darth.Scrumlord

So the problem is that application/x-gzip & application/x-tar are not official media types.

In https://tools.ietf.org/html/rfc6713, it’s mentioned that:

Some applications have informally used media types such as
[…] application/x-gzip […] to describe data compressed with gzip.

Indeed, according to https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API, the archives produced by Artifactory use these informal media types.

I’ve created this issue to support these informal media types. In the meantime, you might want to look at Artifactory to understand if these media types are configurable or maybe create those archives in some other way.

Would you recommend i create these archives just using the linux package zip? Because it seems, at least with my distro, tar -xcvf is creating the x-gzip media type.

If i create it with the zip package they come back as application/zip. Can I assume expander.go will handle this file properly?

Well, I tried zip. When I create the archive using zip -r archive.zip /path/to/files and run file --mime-type -b archive.zip I get a return that the archive is application/zip.

I push that up to Artifactory and set the URL and SHA to the asset in sensuctl. I get the same message. I’m at a loss here. The binary gets unpacked and I can use it on the file system in /var/cache/sensu/sensu-agent. So the expander is working in some capacity…

I found where you can configure MIME Types in Artifactory and we’re looking into that but I honestly don’t have much hope that it will help; Especially when I have multiple assets pulling from Artifactory that are tar.gz, that show a MIME Type of x-gzip. Makes me think there’s something else wrong here.

Artifactory is configured to set the content-type for tar to x-tar, tar.gz to x-gzip and zip to zip.

Hi @Darth.Scrumlord,

I’ve never used Artifactory so I’m not sure if it does anything specific behind the scenes or whatever.

So does zip archives work for you? If so I’d recommend sticking to those until https://github.com/sensu/sensu-go/issues/3157 is released.

1 Like

Zip archive didn’t work for this. But I can’t test anymore right now.

Hey,
I believe if you configured our Artifactory to explicitly use the application/gzip for the gzip compressed tarballs assets you have uploaded this should work as a work around for now. Artifactory according to my reading of the available docs lets you explicitly set mimetypes to override the defaults it uses.

Beyond that… I think the x- mimetypes are in general problematic as they are not part of the agreed on RFC standards. I think we’ll need to expose a tunable that allows people to map which x- mimetypes should be treated as tar versus gzip compressed tar to cover all possible corner cases. In reality I could have something like application/x-tarball-of-doom and have it be equally as valid as x-targz from an RFC sense. Sensu agent and backend will probably both need a tunable or the asset definition extended to allow for user defined mimetypes to support.

1 Like

If that’s the case, shouldn’t application/zip work? It’s RFC compliant. I have artifactory mapping .zip to application/zip and the file was created as application/zip. It downloads and unpacks manually or via the asset engine as application/zip. However, I still get the error.

Well, this gets a bit technical… while RFC complaint…zip is a different format than gzip…and as of now assets are documented as being tar with optional gzip.
zips would potentially need a different unpacker codebase than what is currently in use, might not be as simple as just conditionally allowing the zip type to be processed with the existing unpacker.

As of now we don’t have documented support for assets compressed with zip (just tar… optionally compressed with gzip). So as of now I have to assume based on the documentation that the asset unpacker code in use can only interact with tar and gzip compressed tar.

Adding support for zip might be an enhancement request with some merit as zip is pretty popular especially on windows systems. Just need to figure out how hard this would be. If the underlying unpacking golang code can unpack zips…that’s should be an easy change, else have to identify new code to integrate.