Contents
Introduction
Systems and Components are not allowed to possess game assets such as textures. Instead, assets are referred to in Components using handles like numbers or strings. When processing the Components, Systems retrieve the assets from Asset Manager, which is a Runtime Service. If the requested asset is not loaded in memory, this Service is responsible for streaming it from appropriate sources such as packages files on the disk or network. Depending on the loading policy specified by the System, a fallback may be provided for a currently unavailable asset. During the loading of assets, the raw binary on the disk may need conversion or decoding. Multiple types of conversions may be requested for one file type. If the assigned memory is exhausted while new assets being requested, the manager is also responsible for evicting those assets not needed anymore to create space for the new resident. This post discusses these processes involved in processing the game assets.
Use Cases
Consider several use cases in Systems:
- A GUI system that loads a font file in order to build a font atlas. The font atlas is a texture whose content is dynamically incremented while encountering new code points. Initially, the atlas is built with a predefined range of characters. If the font size is changed, the atlas may be discarded and rebuilt. The mapping of texture regions to code points is managed by the System. Meanwhile, the System uses custom icons and images to render user-defined buttons with styles. Those images may be only temporarily used at some moment in the game or frequently referred to during the game.
- A collision detection System which requires the collision models of game entities to be loaded to operate correctly.
- A dynamic texture System that plays movies.
- Generally speaking, most Systems compiling render command lists will be using shaders.
- A map streaming System that loads/unloads map content according to player position.
In the following subsections, these use cases will be dissected into different issues that have to be handled by the Asset Manager.
Disk Access
No matter what do we do with the loaded assets, they come from somewhere on the disk. There are two cases of the organization of the file content, however.
One File Per Asset
The simplest asset storage pattern is that one asset corresponds to a file. This is very inefficient in terms of disk access since files may be scattered on the hard drive. It will lead to many small files that are slow to read, find, and manage. However, this pattern is very useful for development purposes. Before the content of asset packages was even decided, game developers will import a lot of assets to try them out in the game engine. The assets may be located anywhere on the disk or even shared across multiple game projects. They may just come from a large asset repository for development purposes and will be compacted into packages before game release – maybe after some additional processing.
Asset Packages
Few games would deliver individual asset files with the release. One famous exception maybe Minecraft: assets are distributed just in .png and .ogg files. The reasons behind packing game assets are usually reading performance, sometimes content protection. Sometimes both. Nevertheless, using encrypted packages to protect game assets is nonsense since players will eventually crack the format and release convenient tools for noob players to modify the game content, as long as it worths modifying.
An asset package format usually consists of a header, a file table, and blobs of asset content. It’s basically a simplified filesystem.
Loading Policy and Residence
When an asset is requested, the first problem is to decide which parts of the file should be read.
The decisions for it depends on the characteristics of the asset. For most of the game assets, the content has to be completely loaded before use, such as textures, models, etc.
A common exception is multimedia assets such as music and movies. Movies are usually huge in size but the game rarely needs the whole content of it to be resident in memory: the comparatively slow playback speed only requires up to several megabytes to be in the memory for every second. For a good container format, the data are divided into chunks so that we only have to go forward to load future frames. The only occasion when it’s better to preserve the whole movie in the memory is when it’s looped in the game.
For the loaded assets, the next question is how long they should resident in the memory. Most assets tend to have a one-time use only for uploading them to the video memory. The management of video memory is another problem and probably requires some other strategies since we have less control over there thanks to the lack of transparency of how GPU works. Some assets are only used on the CPU side such as font files. However, they may also be used to produce output that meant to be uploaded to the GPU.
Memory-Mapped Files
What we need are explicit control over disk access and memory usage. The virtual memory again provides a convenient mechanism by decoupling the consumption of address space from the usage of physical memory. When an asset is requested, a memory mapping backed by the file is first created. The mapping consumes no physical memory before the first read to any part of it, which causes a page fault and the operating system handles it by loading the corresponding page from the disk. Depending on the access pattern of the asset, the whole or part of the content of the file may be prefetched from the disk, such as using VirtualLock() on Windows. When the content is not needed anymore, it can be explicitly unloaded by notifying the operating system that a certain range of pages is not in need anymore and may be released using VirtualUnlock() on Windows.
Our memory management mechanism would be thus monitoring how much memory has been committed and how much quota is left, which may be a configurable parameter or a limit set by the available physical memory.
Even though the files are accessed through memory mapping, we cannot rely on the operating system to evict pages when we run out of quota. The operating system is simply not informed of what kind of data is important and may start to getting rid of pages containing program code when the situation comes. Instead, we manage the memory cache of assets with the frequency it is being used and the importance of it. Some assets may not be accessed frequently but are supposed to be resident in the memory instantly when used, such as audio effects. Some assets have variable levels of details and may be replaced with the version with lower quality when the memory is scarce.
Asset Decoding and Streaming
Not all assets may be immediately useful after loading into memory. Actually, very few assets are able to be so. After the corresponding pages are loaded into the memory-mapped heap managed by the asset service, post-processing is usually needed for decoding the asset content which is then used to create device-dependent resources such as GPU textures.
The decoding is needed for several reasons. During the development phase of a game, most of the assets will be in various formats created by arbitrary content creation tools. Because of the need for rapid prototyping, they might not go through the complete data conversion pipeline which is used for game release to save the development time.
Ideally, we should be able to read data in an immediately useable format from the disk so that as long as the file is memory-mapped it can be used as assets. However, these formats usually mean plain blobs of uncompressed bytes, which would consume a lot of disk storage and potentially slow to read. So most popular binary formats will have some forms of compression. Choosing the compression method is not our job. Instead, the asset service treats the decompression as a decoding phase which may be chained.
The asset service should provide a flexible and type-safe way of handling the decoding of assets. Because the Systems obtain asset names from Components, they have no idea about how the data is represented on the disk but only what kind of runtime resource they finally aim to get.
For example, a rendering system could be using some textures. Since it simply gets the texture name from the Component, it does not know whether the texture is simply a png file on the disk or some compressed texture stored in some asset package. It does not even know what kind of rendering API it is dealing with so it can’t decide how to convert a raw image buffer to a GPU resource. It should actually never try to do so as that would seriously complicate the management of GPU resources.
The responsibility of managing GPU heaps is delegated to the rendering subsystem. The rendering System informs what asset is needed for a particular rendering command and its work is done. In this sense, the System has nothing to do with Asset Service. However, the System should inform the rendering subsystem about the importance of the asset and what kind of compromises could be made in case of insufficient memory, such as using fallback assets or discarding the render command list.
The following diagram illustrates the handling of asset requests.
Asset Decoder Interface
The role of asset decoder is somehow involved. One obvious problem is the decoded asset could be anything: a pixel buffer, video chunk, audio chunk, compiled script, etc. The type of decoded asset highly depends on the client of it.
Let’s consider some potential clients of the asset service:
- Graphics Subsystem
- Request decoded textures. The raw asset is some common image format such as png or jpg, which should be decoded into a pixel buffer with a specific number of channels. One raw image may be decoded with different parameters. After the decoding, the graphics subsystem streams the pixels by copying it to a host-accessible GPU memory location, then to a device-local location. Only the first copy is necessary if the machine has a unified memory. A missing texture generally doesn’t forbid the rendering process.
- Request shaders. There are two issues around loading shaders. First is shader code may need cross-compiling depending on the graphics API in use. Second is compiling shader takes time. The rendering generally cannot proceed before the shader is properly loaded.
- Request graphics pipelines. Graphics pipelines are generally a set of parameters plus the references to shaders. The new APIs such as Vulkan tend to provide methods for caching compiled graphics pipelines since it is a time-consuming task, so the asset manager should be able to return the cached pipeline object when possible. However, it should be aware of the graphics API requesting the cache and return the correct version. Besides, the asset manager should also take care of the references to shaders.
- Game Scripts
- The script may be stored as source codes or compiled lua bytecodes. The object to obtain is a lua state machine.
To summarize, the asset service is basically an interface to the memory and filesystem. The basic service it provides it mentioned above, which is loading assets
KInd of heaps? Kind of allocation /cache / usages?
reference management
read only asset
read write asset
cache
