Unreal Text Localization

Last Thursday, we released the new Wild Lands chapter for Titan Quest II.

In addition to the huge new chapter and various gameplay improvements, we added a new work-in-progress set of text translations for Ukrainian and Russian languages. I figured this might be a good time to write a short blog post about some of the localization tools we use to make this possible, especially all the things that don’t come out of the Unreal box.

CSV String Tables for Source Texts

I strongly recommend, disallow any in-asset text literals and only using string tables. Either the binary asset or a even better a text backed source.

Biggest advantages: - all strings in one place - no gather commandlet / asset loading needed to find all strings and export them for translators - better version control support than binary string tablse (text based format)

For Titan Quest II, we’re using the CSLocTools plugin / engine modifications provided by our friends at Coffee Stain and co-developed by Deep Silver Fishlabs. This plugin provide easy access to CSV backed string tables and patching tools that build on top of the old CSV string table system built by Epic. In contrast to Epic’s old system, this plugin allows loading string tables dynamically and not having to declare/load them from code, which makes working with them a breeze.

I added some modification for that plugin that were since merged which add UI to allow adding new string table entires directly from the FText property window, as well as providing a basic content browser source and in-engine editor for string tables, so devs don’t need to switch tools for simple string edits.

More information: CSLocTools on GitHub (you need to log-in with a GitHub account that has access to the Epic Games user group as described here)

Another alternative: My former colleague Philipp Brasin joined one of Epic’s Inside Unreal sessions to show off the framework he built at Aesir Interactive. They are using a json backed format which allowed them to attach more structured per-context data.

To actually enfore that texts are never left in assets as literals, I created an asset validator that uses FPropertyLocalizationDataGatherer to search for gatherable text properties and asserts that all found texts originate from string tables.

CSV String Tables for Runtime

Almost as bad as the source editing/export process are build and patch times for small string adjustments. We knew we needed support for translation files that live outside of the cooked UFS packages to patch them quickly.

For our game this is as simple as calling UOUUTextLibrary::LoadLocalizedTextsFromCSV() whenver the language is changed. This function - available as part of my OpenUnrealUtilities here loads csv files with a name pattern matching the currently active culture code and loads them into a PolyglotTextData source.

With this approach we could completely ditch the compilation of texts into locres files.

A nice side effect of this is that we could support arbitrary community translations - as long as the culture code is known to Unreal, we would pick up the matching file and display it as an additional entry in the settings menu with “community translation” label.

This greatly helped one of our players who was providing a Russian translation mod before our official Russian translation had shipped:

One exception to the rule: For cutscenes, we used timed subrip files, which are suffixed with the culture code. This works for a game of our scale since we have very few cutscenes, but a game with a lot more cutscenes / time based content may want to use different formats.

Import/Export

Most of the automation scripts we use at Grimlore are Python based. For CSV loc tables I published some utilities for CSV/po conversion and validation in localization.py which is part of my free automation tools.

There’s a bunch of stuff for versioning and additional files relevant for our specific translation process, but essentially the import/export script we use for TQ2 looks somewhat like this:

import_dir = loc.get_csv_dir(f"{root_dir}/TQ2/", "LastReceivedByTranslators")
export_dir = loc.get_csv_dir(f"{root_dir}/TQ2/", "ExportForTranslation")

# Gather source texts from Unreal source CSVs
new_lines = loc.collect_source_strings(root_dir, source_language, ["TQ2"], source_csvs=list(glob.glob(f"{root_dir}/TQ2/Content/Text/*.csv")))
for target_language in ["de", "zh-Hans", "fr", "ru", "uk"]:
    # Import last translations based on new lines (can optionally discard outdated strings, etc)
    last_translated_lines = loc.import_csv_translations(target_language, "TQ2", new_lines, translation_csvs=[f"{import_dir}/TQ2_{target_language}.csv"])

    # Export for translators (with as much metadata as needed)
    meta_data_keys = ["Source", "DevComment"]
    loc.write_translation_csv(os.path.normpath(os.path.join(
        export_dir, f"TQ2_{target_language}.csv")), last_translated_lines, combine_key=True, meta_data_keys=meta_data_keys)

    # Combine current source texts with translations for game (no metadata)
    loc.export_csv_for_game_ouu(f"{root_dir}/TQ2/", "TQ2", target_language, last_translated_lines)

This is a procedure that takes less than a minute, where a traditional Unreal gather/export/import/compile procedure took us min 10-15 minutes.

Format Validation

FText formatting can get tricky, especially with Epic’s custom CSV flavor that doesn’t stick to the RFC 4180 (Excel doesn’t either, but with its own quirks).

Then on top you have format placeholders, argument modifiers and multiple possible layers of escape sequences required to render special characters or make conditions in quoted strings evaluate correctly.

To make this testable from outside of the Unreal Editor, I created a browser based Unreal Text Format Validator that can convert between in-engine and Epic style CSV formats AND breaks down and highlights all the syntactic things going on.

This was strongly inspired by regexr and regex101 - both beautiful tools that help make sense of cryptic Regex expressions.

The validator still has some quirks in edge cases with backtick escaped special characters, but it’s already proved really useful internally when reviewing texts without having to resort to launching the Unreal editor.