Moving forward on asset processing

[viridia]

"Assets V2" was released as part of Bevy 0.12. It brought a number of welcome improvements, however it also came with a large shopping list of future aspirations which, for the most part, have not been realized. In fact, if you take a look at the git history for bevy_assets, most of the changes in the last 6 months are either small bug fixes or knock-on effects from global refactorings.

I feel like the work on assets has stalled, and I have a couple theories why:

• @cart has been focused primarily on other important issues like BSN and the Bevy Foundation.
• Other than @cart, there's no clear owner or "keeper of the vision" for assets.
• Optional features such as asset processing are not being widely utilized.

From what I have heard on Discord, developers are leery of enabling the "asset_processing" feature flag because they don't like the automatic generation of .meta files in the asset source directory. This makes sense - as a general rule, developers dislike checking in automatically-generated files unless there is a compelling reason to do so, as it violates the "hermetic builds" principle. For developers dealing with thousands of asset files, doubling the number of files to manage can also have an appreciable negative impact.

I think that the general idea of .meta files is sound, but needs some tweaks. The rest of this post is going to be a set of suggestions for possible next steps which hopefully will unblock progress and get things moving forward. Some of this has already been discussed on Discord.

Opting-out of automatic .meta generation

This first step is very simple: provide a configuration option that allows enabling background processing of assets without the automatic creation of .meta files. Presumably this would be an option that is passed to the AssetPlugin at startup.

Opting-in to manual .meta generation

However, we need to address the original reason that .meta files were generated. The format of a .meta file contains a serialized loader settings object, which will be different for every asset loader type. This means that you won't always be able to look up a documented example on the web for how to write a .meta file, especially if the asset loader is a custom one with its own idiosyncratic settings options.

Thus, it can be helpful to have a way to generate a template .meta file for a given asset type. However, you should generally only need to do this once per project, as you can simply cut & paste the example file afterwards (which you are probably going to do anyway, since likely the custom options for one asset will be used for multiple assets).

At this point, one might ask "OK, so what kind of user experience should creating a .meta file entail?". Will it be a command-line tool? A debug command? A feature of the Bevy editor?

I think the right answer here is to punt on the user experience, and let the Bevy community fill in the blanks. Instead, Bevy should provide a simple API that says "generate a .meta file for this asset path", and it will do the work of looking up the correct loader, serializing the default settings, and writing out the file. Third-party developers can then make tools of various kinds which call this API, some of which we can upstream once they have proved themselves.

There is one subtle nuance which I think will be important, which is that it should be possible to generate a .meta file for an asset that doesn't exist yet. This will be important for the creation of wildcard metas (next section).

Folder meta files

In the Assets V2 feature list, there's an important item for "folder metas" - that is, being able to specify a .meta file for an entire folder. However, there's not a lot of detail on this, and the feature idea has languished.

I think having a single .meta for an entire directory is somewhat problematic. Meta files contain serialized loader settings, and the meta file for a folder could potentially contain settings from multiple loaders, which would make deserialization tricky. Yes, you could have a serialized map whose keys uniquely identify a loader, but this would either have to be the reflected name of the loader, or the file extension, since those are the only type ids we have available. And each entry in the map would have a different type.

Instead, I propose that we allow multiple folder meta files, one for each file extension. Each "file type meta" would contain settings for only one type of file. To distinguish these from regular .meta files, a naming convention will be used. Some suggestions from the Discord are:

• .png.meta
• _.png.meta
• mod.png.meta
• <directoryname>.png.meta

Of these, I like the second option, _.png.meta for the following reasons:

• Because it doesn't start with a period, it won't be treated as a hidden file.
• Because it's unlikely to have an asset named _.png, there is little chance of a naming conflict with regular meta files.
• The name is constant, meaning that you can test for the existence of the file without knowing the name of the directory.
• According to the Wikipedia article on filesystems, _ is not a reserved character in any modern filesystem.
• _ sorts before alphanumeric, so the folder metas will appear at the top of the directory listing.

To keep things simple, I propose that there will be no inheritance or merging of meta files. If an asset has a regular .meta file, we use that, end of story; otherwise we check for the existence of a folder meta for that file extension. (A future option might be to check the parent directory and so on, all the way up to the asset root).

Performance considerations: A filesystem operation to test for the existence of a file is not free, and we want to avoid testing for the folder meta every time we load an asset. This can be accomplished by caching the metas:

• If a meta file exists, we store the contents in a cache the first time we access it.
• If we check for a meta file and it doesn't exist, we cache the fact of its non-existence so that subsequent checks will be fast.
• Future improvements can invalidate the cache using directory watching.
  This means that in a directory of a dozen .pngs, the .meta file will only be accessed once, even if it's not there.

How this caching is to be done is TBD.

@alice-i-cecile

[alice-i-cecile]

Opting-out of automatic .meta generation

Simple and relatively uncontroversial. This should be an easy PR.

Opting-in to manual .meta generation

I like your design here, and I think it's the right building block.

Folder meta files

This is the critical thing needed for me to consider using meta files. I want to be able to change settings at a folder / project level. Definitely agree with splitting these out by file type. I like _png.meta best.