Amazon recently published a case study describing how Second Dinner, the studio behind Marvel Snap, used BespokeCI to accelerate their game development. I am an employee of Gamebreaking Studios working on BespokeCI. In this blog post, I'll go into more detail about the challenges we faced and what we did to overcome them. If you're interested in learning more about BespokeCI, you can check out our website or reach out to us directly.
Problem 1: Repo size
The first problem we faced was the sheer size of the repo. As we've discussed before, video game repos can be huge, and cloning them can take a very long time. For this project, the initial clone of the repo took over an hour, and the size of the workspace could sometimes exceed 200gb. This situation would cause timeouts in our CI system. To solve this problem, we opted not to do "clean checkouts" of the repo for every build. Instead, subsequent builds would run against a pre-populated workspace and only pull the latest changes.
When setting this up, we had to be very careful when running git clean. We wanted to establish some level of determinism so that our builds would be consistent, but we also didn't want the next build to take an hour for a fresh clone! We ended up using a combination of a targeted git clean and git reset to remove previously built artifacts and a handful of known temporary files, while leaving the rest of the repo and derived data intact.
As an effort to "reset entropy", we implemented a weekly "Clean Build" that would do the costly rebuild of the entire workspace. This clean build was scheduled during off-hours (Friday evening) to minimize impact for developers and ensuring that the system stayed healthy week over week. While it would be desirable for every build to be a clean build, this compromise gave us reasonable confidence in the quality of our builds while greatly improving their performance.
Problem 2: Scaling builds for multiple lanes
Another request from the customer was that new development branches (referred to as 'Lanes' for this project) be easy to onboard and reasonably performant to build. The traditional practice here is to create cloned build jobs for each lane, each using an isolated workspace. This practice would help guard Mainline or a Release branch from any disruptive development that may be occurring in a lane.
However, this approach would have required us to maintain potentially dozens of workspaces, each one weighing in at hundreds of gigabytes. Many would say disk space is cheap, but it's not that cheap! Instead, we opted to have all lanes share the same workspace. We trusted git checkout's ability to reliably switch the codebase between lanes (reinforced by our careful cleaning process). Each build agent was only required to hold a single copy of the repo, which enabled excellent scaling no matter how many active lanes were in development.
In order to create the build jobs to handle a variable number of lanes, we could have created a single job that accepted the Lane (really, a git branch plus a set of configurations) as parameters. Instead, we opted to create automation that scanned the repo and automatically created new jobs for any lanes found that matched a specific set of criteria. This approach allowed developers to create a new lane for their project and begin building it with BespokeCI almost instantly, without needing manual intervention or a costly initial checkout.
Problem 3: Building on Mac. Yes, it's really that hard.
Provisioning Linux or even Windows servers is relatively easy. BespokeCI uses a combination of technologies like Terraform, Docker, and Ansible to automate provisioning and configuration of build agents. Macs, however, are a different story.
Many of the same tools we use for provisioning also work on MacOS, but some things are harder to automate than others. While we were able to use Ansible to install tools like git, Unity, or clang, we had to rely on manual configuration for some of the more complex tools like Xcode or Keychain Access. There has been some headway in this area (e.g. https://github.com/XcodesOrg/xcodes) but learning and setting up these tools required an initial investment of time and effort.
Aside from software and configuration, even finding where to host a Mac has different options. AWS/EC2 does offer Mac instances, but they are expensive and have some limitations (e.g. minimum 24 hour rental and dedicated host requirements). Other colocation vendors like MacStadium or Mac Mini Vault offer more flexible options, but they require more manual setup and maintenance. It's no wonder that many game studios choose to simply buy a Mac Mini and set it up in their office!
For this project, we opted to use AWS-hosted Macs as build agents. The added flexibility and scaling capabilities outweighed the costs and limitations of the pricier Mac instances. By using AWS, we were able to automate the provisioning of the builders using Terraform and Ansible. The manual portions of the provisioning (mentioned above) were handled by a runbook that we developed to guide the operator through the necessary steps.
Problem 4: Publishing builds to Apple's ecosystem. Yes, this is hard too.
Publishing builds to certain robotically-named mobile platforms can be fairly straightforward. Most game engines support a diversity of platforms, and can generate builds as a single .apk file. After changing a single setting on the phone, a user can download this .apk file (from, say, a web server or Slack) and install it directly. This makes build distribution to play testers a breeze.
Apple's ecosystem, however, is a bit more complex. Apple supports what are called 'Ad Hoc builds', which can be single .ipa files distributed outside of the App Store. These builds still require code signing and provisioning profiles, which means you'll need to register any device you want to play it on in advance of distribution. Apple also provides a tool called 'TestFlight' to ease build distribution, but this tool must be managed manually via a Web interface and has other limitations around the number of devices that can be registered and the length of time a build can be available for testing.
To help mitigate these challenges, we chose to use a 3rd party tool called Fastlane. Fastlane is a series of tools that help automate many of these processes, including certificate management, code signing, and uploads to the TestFlight service. By integrating Fastlane into our build pipeline, we were able to ensure that all builds were published to a location that was easily accessible to play testers, even for new Lanes created for development purposes.
Conclusion
After overcoming these challenges, we were able to create a build system for Second Dinner that streamlined the development and deployment process, greatly increasing their team velocity. Focusing on developer experience and build speed lowered the friction inherent to developing a Live Service game, and allowed the team to focus on what they do best: making great games.
By sharing some of the lessons we learned along the way, we hope to help other game developers who may be facing similar challenges in their own projects.