Many build systems across software development rely on ephemeral, or 'autoscaled', agents that can be spun up on demand. This is a great way to scale your build system and reduce costs. When a new build is requested (either explicitly or because of a commit), the system can create a new build agent to handle the build. When fewer (or no) builds are needed, the system can shut down the agent. This has the added benefit of ensuring that each build has a clean environment and is as reproducible as possible. In game development, this approach is often not feasible.
In game development, build agents are often stateful and persistent. Why? The most common reason is the sheer size of game repositories and artifacts. A game repository can easily exceed 2tb in size, and a game artifact can exceed 100gb. With these sizes, even the simplest of actions in your CI/CD pipeline can take tens of minutes, if not hours.
Imagine having to wait 2 hours just to find out you missed a semicolon in your code. This can happen if every build must pull down the entire repository and build from scratch. This can be extremely costly to a studio's velocity and jeopardize its ability to iterate and find the fun.
To combat this, many game companies instead rely on incremental build and stateful build agents. They prefer to keep the source and build artifacts cached on the agents. While this approach can lower determinism and open the door for subtle bugs, it also allows for significantly faster builds. The same 2 hours you spent waiting to find a simple syntax error could instead be found in the first 60 seconds. Is it any wonder that game companies might choose to prioritize build speed over determinism, when the difference is so stark?
In addition to speed and determinism, there's a third point on this Iron Triangle: complexity (representing cost). Some studios choose to invest heavily on intelligent caching systems, enabling them to use ephemeral agents while still maintaining fast builds. Using modern tools and cloud infrastructure, studios can dynamically scale build agents on demand that use sophisticated caching (of both source code and build artifacts) to achieve fast builds without sacrificing determinism. This approach can be complex and costly to implement and maintain, especially for smaller studios, but for those with the resources to invest in it, it can be a game-changer for their build system.