[Tooling] Add localization automation #765

iangmaia · 2025-06-14T00:53:10Z

Related: AINFRA-765: Integrate new Fluent conversion tool into wordpress-rs

Note: The fluent-tools Gem used in the PR is currently using the Gem directly from the branch Add Ruby Gem wrapper / CLI interface and CI setup. 🎗️ We need to update it once the Rust tool is fully merged.

This PR supersedes #756.

What does it do?

This PR integrates the new Fluent conversion tool Gem into wordpress-rs to enable a complete translation workflow between the project's Fluent localization files (.ftl) and GlotPress, allowing for:

Source generation: Converts the source English Fluent file to PO format (.pot) for uploading to GlotPress
Translation sync: Downloads translated PO files from GlotPress and converts them back to Fluent format, adding them to the project
Locale management: Supports MAG16 locales with mapping between GlotPress and project locale codes; this was implemented just to limit the languages we know we support, but this mapping can be easily updated in the future

New lanes

generate_source_po_file lane - Converts en-US/main.ftl to en-US.pot for GlotPress upload
download_translations lane - Downloads all available translations from GlotPress following the supported locales list and updates local Fluent files
generate_fluent_file_from_po lane - Converts individual PO files back to Fluent format
download_po_files_from_glotpress lane - Downloads PO files for all supported locales

Proposed localization workflow

A day-to-day workflow can look like this:

New features are implemented, new localization strings need to be created (or updated).
Developers add strings to the source language file wp_localization/localization/en-US/main.ftl.
With a certain frequency or based on a trigger (for example, new commits to trunk or a daily scheduled build), we run generate_source_po_file and commit the updated wp_localization/glotpress/en-US.pot.
A wpcom job uploads the updated en-US source file to GlotPress (see 184627-ghe-Automattic/wpcom )
Again with a given frequency, we fetch the translations, generate the Fluent files and commit them to the project (or create a PR for review) by running download_translations.

Update: The PR #770 integrates the automation using the Nightly schedule as the trigger mentioned above.

…wnloading

crazytonyli · 2025-06-17T10:13:38Z

fastlane/Fastfile

+  { glotpress: 'sv',    project: 'sv'     },
+  { glotpress: 'tr',    project: 'tr-TR'  },
+  { glotpress: 'zh-cn', project: 'zh-CN'  },
+  { glotpress: 'zh-tw', project: 'zh-TW'  }


This is shorter than I thought it'd be. The translations in this library should be a combination of the locales supported in the iOS and Android apps. Here is the list that I gathered a while ago:

ar az bg cs cy da de el en-AU en-CA en-GB en-US es es-CL es-CO es-MX es-VE eu fr fr-CA gd gl he hi hr hu id is it ja kmr ko lv mk ms nb nl pl pt pt-BR ro ru sk sq sr sv th tr-TR uz vi zh-CN zh-HK zh-TW

I've updated the list of supported locales on 51af12c (for the record, I also had to add fr-CA in GlotPress). Thanks @crazytonyli!

crazytonyli · 2025-06-17T10:14:44Z

Thanks for working on this! The result looks great!

AliSoftware

LGTM 👍

Left a couple of nitpicks but just mostly form and code adjustments, nothing major.
Approving to unblock (though reminder not to merge this until the fluent-tools gem is finalized and published as a tagged version so we can point to it instead of the branch)

AliSoftware · 2025-06-17T10:07:12Z

Gemfile

 gem 'fastlane-plugin-wpmreleasetoolkit', '~> 13.2'
+gem 'fluent-tools', '~> 0.1', git: 'https://github.com/Automattic/fluent-rust-tools.git', branch: 'iangmaia/ruby-gem-support'


🎗️ As you mentioned in the PR description, reminder to update this once the gem is published before merging this PR

AliSoftware · 2025-06-17T10:15:24Z

wp_localization/glotpress/en-US.pot

💄 I wonder if it would be worth adding a README.md in that wp_localization/glotpress/ folder, even a very simple one, to explain what it's for?

### What is this folder? We use GlotPress as a platform/service to have translators translate our strings from English to other languages. But GlotPress does not support Rust's Fluent `.ftl` format as an input/output file format. To circumvent that, during the localization automation process, we convert the `localization/en-US/main.ftl` file to the PO format in `glopress/en-US.pot` so that a cron job can then later pick up that `.pot` file[^1] and upload it to GlotPress and send it to translators. This transformation is handled by `bundle exec fastlane generate_source_po_file`. Later we then download the translations from GlotPress (which are exported in the `.po` format), then regenerate the `localization/*/main.ftl` files for each language based on those downloaded translated `.po` files. This is handled by `bundle exec fastlane download_translations` [^1]: Note that we have to commit the `glotpress/en-US.pot` file here because that file is picked up by a cron job in our systems on a regular basis (as opposed to the `.pot` being sent via API on demand), given how those imports are integrated in the company's larger localization system (with import logs and packages and pings sent to translators etc. that a simple API upload to GlotPress wouldn't cover otherwise)

Good point, thanks!
I've added it to the sync job PR as it refers to the automated stuff, but I tried to add the process and lanes description to the README as well: d69ebf3

AliSoftware · 2025-06-17T10:22:10Z

fastlane/Fastfile

+# Converts a PO file for a given locale back to Fluent format
+#
+# This lane takes a PO file and converts it to the corresponding Fluent format file.
+# The locale is extracted from the PO filename (e.g., 'fr-FR.po' becomes 'fr-FR').
+# The resulting Fluent file is saved in the appropriate locale directory.
+#
+# @param file_path [String] The PO file path to convert (e.g., 'path/to/fr-FR.po')
+# @return [String] The path to the generated Fluent file
+#
+lane :generate_fluent_file_from_po do |file_path:|
+  locale = File.basename(file_path, '.po')
+  fluent_file_path = File.join(LOCALIZATION_FLUENT_FILES_DIR, locale, MAIN_FLUENT_FILE_NAME)
+
+  UI.user_error!("❌ PO file not found: #{file_path}") unless File.exist?(file_path)
+
+  FileUtils.mkdir_p(File.dirname(fluent_file_path))
+
+  FluentTools.po_to_fluent(
+    file_path,
+    fluent_file_path
+  )
+
+  next if !File.exist?(fluent_file_path) || File.empty?(fluent_file_path)
+
+  UI.message("✅ #{File.basename(file_path)} → #{fluent_file_path}")
+
+  fluent_file_path
+end


💄 / 💡 Opinionated: I'd find it easier to read if that whole helper lane was moved to after the download_translations and download_po_files_from_glotpress lanes in the Fastfile.

This simple move would allow the reader of the Fastfile to better understand that generate_source_po_file and download_translations lanes, that would then appear first, are the main lanes one is supposed to call, while download_po_files_from_glotpress and generate_fluent_file_from_po are helper lanes called by the one above it.

This also helps organizing the lanes in the Fastfile by the order in which they'd typically be called by the automation, with download_translations calling download_po_files_from_glotpress first then generate_fluent_file_from_po next on (each locale).

Agreed! Updated in 1d98854

AliSoftware · 2025-06-17T10:22:44Z

fastlane/Fastfile

+  locale = File.basename(file_path, '.po')
+  fluent_file_path = File.join(LOCALIZATION_FLUENT_FILES_DIR, locale, MAIN_FLUENT_FILE_NAME)
+
+  UI.user_error!("❌ PO file not found: #{file_path}") unless File.exist?(file_path)
+


💄 Might make more sense to do the guard statement earlier, since it's checking the input param?

Suggested change

locale = File.basename(file_path, '.po')

fluent_file_path = File.join(LOCALIZATION_FLUENT_FILES_DIR, locale, MAIN_FLUENT_FILE_NAME)

UI.user_error!("❌ PO file not found: #{file_path}") unless File.exist?(file_path)

UI.user_error!("❌ PO file not found: #{file_path}") unless File.exist?(file_path)

locale = File.basename(file_path, '.po')

fluent_file_path = File.join(LOCALIZATION_FLUENT_FILES_DIR, locale, MAIN_FLUENT_FILE_NAME)

👍 Updated on 98947d6

AliSoftware · 2025-06-17T10:28:58Z

fastlane/Fastfile

+    updated_fluent_files = []
+
+    # Convert PO files back to Fluent format
+    if downloaded_files.any?


💄 I'd suggest to instead use a guard statement to reduce indentation and exit early:

if downloaded_files.empty? UI.message("No .po files were downloaded from GlotPress") next end UI.header('🔄 Converting PO files to Fluent format') # … and the rest of the code, without the need for the extra indentation

👍 Updated in 55c1118

AliSoftware · 2025-06-17T10:33:31Z

fastlane/Fastfile

+  Dir.mktmpdir do |temp_download_dir|
+    downloaded_files = download_po_files_from_glotpress(download_dir: temp_download_dir)


❤️ Nice touch using a temp dir for the download and processing here, will not only prevent any intermediate file from being in the repo (even if we could have gitignored them) and also prevent any previous runs and translations downloads from impacting future ones (e.g. a fr.po file downloaded on first run but not anymore on the 2nd run for any reason, which would lead to the previous fr.po file to not be replaced and its old content to be used if you were not using a mktmpdir) 👍

…_po` lane

…o downloaded files

@crazytonyli

h/t to @crazytonyli for highlighting that the supported locales should be a combination of the locales supported in the iOS and Android apps #765 (comment)

…the latest version

iangmaia self-assigned this Jun 14, 2025

iangmaia added the Tooling label Jun 14, 2025

iangmaia mentioned this pull request Jun 14, 2025

[Tooling] Add GlotPress integration for Fluent localization files #756

Closed

iangmaia added 11 commits June 14, 2025 03:14

Use latest release-toolkit Gem

a93770d

Add Fluent / PO localization lanes

aa1c116

Update source PO file (en-US.pot) to be synced to GlotPress

e3ae230

Update generated Fluent files based on latest GlotPress translations

3b35f97

Update release-toolkit

7506225

Fix Rubocop violations

6f24ef2

Run bundle update

2aa5535

Create temporary dir for downloading PO files and use open-uri for do…

40c5c25

…wnloading

Update code to use fluent-tools Gem

7e0887d

Update generated source PO file

1d23b72

Update generated Fluent files based on latest GlotPress translations

b15d9ee

iangmaia force-pushed the iangmaia/fluent-to-glotpress-with-rust-tools branch from 6c68054 to b15d9ee Compare June 14, 2025 01:14

iangmaia marked this pull request as ready for review June 14, 2025 01:16

iangmaia requested a review from a team June 14, 2025 01:29

iangmaia mentioned this pull request Jun 16, 2025

[Tooling] Add Localization sync job #770

Open

crazytonyli reviewed Jun 17, 2025

View reviewed changes

AliSoftware approved these changes Jun 17, 2025

View reviewed changes

iangmaia added 6 commits June 17, 2025 19:19

Reorder helper lane

1d98854

Validate file_path at the beginning of the `generate_fluent_file_from…

98947d6

…_po` lane

Update code to exit early in download_translations when there are n…

55c1118

…o downloaded files

Add all locales supported by the iOS and Android apps

51af12c

h/t to @crazytonyli for highlighting that the supported locales should be a combination of the locales supported in the iOS and Android apps #765 (comment)

Add generated Fluent files with the updated list of supported locales

fe97f0c

Update fluent-tools to 0.2.0 and re-generate localization files with …

4c71ff2

…the latest version

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Tooling] Add localization automation #765

[Tooling] Add localization automation #765

Uh oh!

iangmaia commented Jun 14, 2025 •

edited

Loading

Uh oh!

crazytonyli Jun 17, 2025

Uh oh!

iangmaia Jun 18, 2025

Uh oh!

crazytonyli commented Jun 17, 2025

Uh oh!

AliSoftware left a comment

Uh oh!

AliSoftware Jun 17, 2025

Uh oh!

AliSoftware Jun 17, 2025 •

edited

Loading

Uh oh!

iangmaia Jun 17, 2025

Uh oh!

AliSoftware Jun 17, 2025 •

edited

Loading

Uh oh!

iangmaia Jun 17, 2025

Uh oh!

AliSoftware Jun 17, 2025

Uh oh!

iangmaia Jun 17, 2025

Uh oh!

AliSoftware Jun 17, 2025

Uh oh!

iangmaia Jun 17, 2025

Uh oh!

AliSoftware Jun 17, 2025

Uh oh!

Uh oh!

		gem 'fastlane-plugin-wpmreleasetoolkit', '~> 13.2'
		gem 'fluent-tools', '~> 0.1', git: 'https://github.com/Automattic/fluent-rust-tools.git', branch: 'iangmaia/ruby-gem-support'

		Dir.mktmpdir do \|temp_download_dir\|
		downloaded_files = download_po_files_from_glotpress(download_dir: temp_download_dir)

[Tooling] Add localization automation #765

Are you sure you want to change the base?

[Tooling] Add localization automation #765

Uh oh!

Conversation

iangmaia commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does it do?

New lanes

Proposed localization workflow

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

crazytonyli commented Jun 17, 2025

Uh oh!

AliSoftware left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AliSoftware Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AliSoftware Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

iangmaia commented Jun 14, 2025 •

edited

Loading

AliSoftware Jun 17, 2025 •

edited

Loading

AliSoftware Jun 17, 2025 •

edited

Loading