Remapped Unreal Engine Binaries
ℹ️ This article is based on my experience working at Grimlore Games.
⚠️ I’m assuming high-level familiarity with Epic’s suggested UE5 deployment workflows, particularly Unreal Game Sync, and Perforce (P4).
For every Unreal Engine project, the team needs to decide how they want to deploy the engine source code and binaries. Depending on the project needs this might vary a lot. For smaller projects it might by sufficient to download a prebuilt engine release from the Epic Games Launcher, but most studios will opt for compiling the engine from source.
In the past (up until UE 4.27), Epic Games recommended to pick one of the following three deployment workflows:
- Using an Installed Build
- Submitting Binaries to Perforce
- Unreal Game Sync (UGS)
With the change to UE5 they added a fourth option to run UE5 in cloud based containers and revoked their recommendation to submit binaries to Perforce, mostly because they moved away from this deployment workflow in favor of UGS.
I want to offer my own take of submitting binaries to Perforce that takes some core ideas of UGS, while adopting it to the requirements I’ve encountered with my latest project at Grimlore Games.
Early on we specified the following requirements for hosting the engine in Perforce:
- Allow frequent changes of engine and project source code
- Check-in precompiled binaries for the engine and project, so artists do not have to compile from source
- Allow programmers not to sync binaries and compile directly from source
- Keep data transfer for precompiled binary sync as low as possible (e.g. only build and submit incremental changes) - especially important for remote developers
- Single step syncs (no manual follow-up steps after Perforce sync to unpack/move binaries)
- One source control tool for submitting and syncing
From the above requirements we concluded that Unreal Game Sync would not suit our needs because of the following points:
- UGS does not allow submitting, only syncing
- Uploaded binaries (at the time) were always archived in their entirety, i.e. no incremental builds
- Syncing with UGS to a changelist with matching source files was not detected and would would needlessly download all binaries again
The modified pseudo-UGS workflow we came up with boils down to this:
- Engine and project source code are hosted in the same repository
- Projects are hosted as “native” projects (i.e. they are part of the engine folder structure)
- Binaries for both projects and the engine are submitted into a
PrecompiledBinariesdirectory outside of the regular engine directories in P4 (on the same stream as source code). These files are…
- …excluded from workspaces of coders who compile from source
- …locally remapped to the regular engine directories using symlinks for everyone else
- Only exception: The buildserver that actually compiles and submits the binaries has a checkout without symlinks
Specifically, this means that…
Precompiled binaries are built on a build server and copied to the precompiled binaries directory. Those build agent workspaces do NOT have the symlink setup to allow simple reconcile/revert of the
There is a special
precompile.p4ignorefile that gets remapped to a regular
.p4ignorefile in workspaces that do use the precompiled binaries which prevents any files being checked in via the symlink paths:
Precompiled binaries are only stored for the last 3 revisions as per P4 typemap settings
//ProjectDepot/.../PrecompiledBinaries/....dll binary+S3w //ProjectDepot/.../PrecompiledBinaries/....exe binary+S3w //ProjectDepot/.../PrecompiledBinaries/....pdb binary+S3w //ProjectDepot/.../PrecompiledBinaries/....target text+S3w //ProjectDepot/.../PrecompiledBinaries/....modules text+S3w
Changes to file-type / ignores / workspace-mapping of precompiled binaries is easy:
# This line excludes all precompiled binaries from the workspace, when added to the stream paths exclude PrecompiledBinaries/...
The perforce stream layout for the project looked like this:
mainsees the bulk of commits with content creators working with precompiled binaries. Their workspaces usually point to virtual streams like
integration-animationthat have department specific changes to the directory mappings.
codeis used as a HUB for code changes. Editor binaries are precompiled and submitted here before they are merged to
- Coders work is done on child streams of
code-no-binaries. This is a single virtual stream that removes all precompiled binary files from their workspace mappings.
The other streams depicted are not really part to the precompiled binary workflow, but to give a more complete picture:
- The engine changes were merged into the project via an engine develop and release stream below
- The release stream (here
ue-epic) contains a clean copy of UE taken from Epic’s Perforce.
- The develop stream (here
ue) contains a full copy of project and engine to integrate engine changes into the project before releaseing them to the team.
- The release stream (here
- Per-person developer streams for isolated code changes
- Release streams branched from
mainfor project milestones & hardening
The perforce configuration described above does all of the heavy lifting. Of course, we have some automation tooling on-top that actually builds the editor binaries, updates the engine version files, etc and merges changes between streams, but that’s pretty independent of the setup described here.