Remapped Unreal Engine Binaries

Introduction

For every Unreal Engine project, the team needs to decide how they want to deploy the engine source code and binaries. Depending on the project needs this might vary a lot. For smaller projects it might by sufficient to download a prebuilt engine release from the Epic Games Launcher, but most studios will opt for compiling the engine from source.

In the past (up until UE 4.27), Epic Games recommended to pick one of the following three deployment workflows:

• Using an Installed Build
• Submitting Binaries to Perforce
• Unreal Game Sync (UGS)

With the change to UE5 they added a fourth option to run UE5 in cloud based containers and revoked their recommendation to submit binaries to Perforce, mostly because they moved away from this deployment workflow in favor of UGS.

I want to offer my own take of submitting binaries to Perforce that takes some core ideas of UGS, while adopting it to the requirements I’ve encountered with my latest project at Grimlore Games.

Requirements

Early on we specified the following requirements for hosting the engine in Perforce:

• Allow frequent changes of engine and project source code
• Check-in precompiled binaries for the engine and project, so artists do not have to compile from source
• Allow programmers not to sync binaries and compile directly from source
• Keep data transfer for precompiled binary sync as low as possible (e.g. only build and submit incremental changes) - especially important for remote developers
• Single step syncs (no manual follow-up steps after Perforce sync to unpack/move binaries)
• One source control tool for submitting and syncing

From the above requirements we concluded that Unreal Game Sync would not suit our needs because of the following points:

• UGS does not allow submitting, only syncing
• Uploaded binaries (at the time) were always archived in their entirety, i.e. no incremental builds
• Syncing with UGS to a changelist with matching source files was not detected and would would needlessly download all binaries again

Core Idea

The modified pseudo-UGS workflow we came up with boils down to this:

• Engine and project source code are hosted in the same repository
• Projects are hosted as “native” projects (i.e. they are part of the engine folder structure)
• Binaries for both projects and the engine are submitted into a PrecompiledBinaries directory outside of the regular engine directories in P4 (on the same stream as source code). These files are…
  • …excluded from workspaces of coders who compile from source
  • …locally remapped to the regular engine directories using symlinks for everyone else
  • Only exception: The buildserver that actually compiles and submits the binaries has a checkout without symlinks

Implementation Details

Specifically, this means that…

• Precompiled binaries are built on a build server and copied to the precompiled binaries directory. Those build agent workspaces do NOT have the symlink setup to allow simple reconcile/revert of the PrecompiledBinaries folder

• There is a special precompile.p4ignore file that gets remapped to a regular .p4ignore file in workspaces that do use the precompiled binaries which prevents any files being checked in via the symlink paths:

  PrecompiledBinaries/integration.p4ignore PrecompiledBinaries/.p4ignore

• Precompiled binaries are only stored for the last 3 revisions as per P4 typemap settings

  //ProjectDepot/.../PrecompiledBinaries/....dll      binary+S3w
  //ProjectDepot/.../PrecompiledBinaries/....exe      binary+S3w
  //ProjectDepot/.../PrecompiledBinaries/....pdb      binary+S3w
  //ProjectDepot/.../PrecompiledBinaries/....target   text+S3w
  //ProjectDepot/.../PrecompiledBinaries/....modules  text+S3w

• Changes to file-type / ignores / workspace-mapping of precompiled binaries is easy:

  # This line excludes all precompiled binaries from the workspace, when added to the stream paths
  exclude PrecompiledBinaries/...

The perforce stream layout for the project looked like this:

• main sees the bulk of commits with content creators working with precompiled binaries. Their workspaces usually point to virtual streams like integration-animation that have department specific changes to the directory mappings.
• code is used as a HUB for code changes. Editor binaries are precompiled and submitted here before they are merged to main.
• Coders work is done on child streams of code-no-binaries. This is a single virtual stream that removes all precompiled binary files from their workspace mappings.

The other streams depicted are not really part to the precompiled binary workflow, but to give a more complete picture:

• The engine changes were merged into the project via an engine develop and release stream below code
  • The release stream (here ue-epic) contains a clean copy of UE taken from Epic’s Perforce.
  • The develop stream (here ue) contains a full copy of project and engine to integrate engine changes into the project before releaseing them to the team.
• Per-person developer streams for isolated code changes
• Release streams branched from main for project milestones & hardening

The perforce configuration described above does all of the heavy lifting. Of course, we have some automation tooling on-top that actually builds the editor binaries, updates the engine version files, etc and merges changes between streams, but that’s pretty independent of the setup described here.