Internationalization/localization (i18n/l10n)

[Roardom]

Is there any possibility of adding internationalization support to rinja?

In the blog post https://blog.guillaume-gomez.fr/articles/2024-07-31+docs.rs+switching+jinja+template+framework+from+tera+to+rinja, I noticed:

Many things like i18n support will arrive in the future as well.

I'd be delighted in seeing such support added. Has there been any discussion yet as to the desired syntax and api for it to be implemented?

[GuillaumeGomez]

No discussion about it yet. I never used i18n/l10n so I don't know what API users expect, etc. Maybe @Kijewski has some ideas already?

In any case, if you have suggestions, it's very welcome.

[Kijewski]

The problem I have is that I don't quite like any of the existing i18n libraries. Fluent-bundle is very useful, but it's evaluated at runtime, which is not a good fit for rinja, IMO. Also, any solution should be "HTML-aware", so any variables get escaped, but only variables, not the whole string. And mentioning "string", the result should not be collected into a "String" first, but be rendered into its destination directly.

The crazy idea I had was to use fluent-syntax to parse a language file, convert it to a rinja_parser::Ast, to turn a language file into a Template, which then can be used in other templates. But I daresay implementing that solution would take ages.

[Roardom]

Sure, I can definitely expand on this.

In my case, I maintain a large foss Laravel application (framework docs are here: https://laravel.com/docs/localization) containing ~49k lines of html templates across 400 files. The way it works is the following:

1. Middleware extracts the desired language in the following order of priority:
   1. If user is signed in, use the language saved in their user settings from the database
   2. Use the language saved in cookies
   3. Use the language specified in the lang http header
   4. Fallback to en_US (configurable)
2. I maintain language files such as /lang/en_US/category.php which each contain a dict, with the translated string being the dict value and having a unique string as key to reference it.
3. In the template files, I call the function __('category.unique-string') which fetches the correct translated string from the extracted language. I can also do __('category.unique-string', ['variableName' => 'variableValue']) in case the translated strings have variables. There exist IDE plugins that autocomplete the unique-strings inside the templates.

There are a few things done here that aren't really idiomatic to rust:

1. I really enjoy not having to pass the extracted language through the entire flow of the http request and passing it into the __() function.
2. Message Format 2.0 should be used for the translation string syntax (it's currently still in draft though).
3. Not containing a matching unique string in /lang/ that's specified in __() will return an empty string instead of a compile time error

I have tried a few rust i18n libraries in the past, and haven't yet found one that stuck with me that offers a similar level of convenience, but that also offers the compile-time guarantee of rust.

[m4tx]

Askama-old had numerous i18n pull requests over time, none of which were ever merged. They might be some source of inspiration – they all are based on Fluent templates, I think, so not necessarily what rinja/new Askama is looking for.

• Fluent integration askama-old#211
• I18n runtime askama-old#237
• I18n askama-old#434
• I18n askama-old#700
• i18n askama-old#841
• Derive i18n support askama-old#844
• Askama base crate i18n support askama-old#845
• I18n testing askama-old#846

There's also a discussion here:

• Internationalization (i18n) askama-old#202

From my side, this feature would be great to have – I'm also willing to contribute when we know what we're looking for.

[pollend]

I ended up writing something like this.

// ISO 639 language codes
#[derive(Clone, Copy)]
pub enum Il8nShortCode {
    En,
    Ar,
}

pub const FALLBACK_IL8N: Il8nShortCode = Il8nShortCode::En;
pub const DEFAULT_IL8N: Il8nShortCode = Il8nShortCode::En;

impl Il8nShortCode {
    pub fn from_str(code: &str) -> Option<Self> {
        match code {
            "en" => Some(Il8nShortCode::En),
            "ar" => Some(Il8nShortCode::Ar),
            _ => None,
        }
    }
    pub fn as_str(&self) -> &'static str {
        match self {
            Il8nShortCode::En => "en",
            Il8nShortCode::Ar => "ar",
        }
    }
    pub fn supported_lang_iso_639() -> &'static [&'static str] {
        &["en", "ar"]
    }
}



macro_rules! tr_il8n {
    ($name: ident, { $($code: ident => $text: literal),* }) => {
        pub fn $name(code: Il8nShortCode) -> &'static str {
            let find_lang = |c| match c {
                $(Il8nShortCode::$code => Some($text),)*
                #[allow(unreachable_patterns)] 
                _ => None,
            };
            find_lang(code).unwrap_or_else(|| find_lang(FALLBACK_IL8N).expect("Invalid language code"))
        }
    }
}

macro_rules! tr_il8n_fast_fmt {
    ($name: ident, ($($prop: ident : $t: ty),*), { $($code: ident => ($text: literal, $($arg:ident),*) ),* } ) => {
        pub fn $name(code: Il8nShortCode, $($prop: $t,)* ) -> impl FastWritable {
            struct __LangFmt {
                code: Il8nShortCode,
                $($prop: $t,)*
            }
            impl FastWritable for __LangFmt {
                fn write_into<W: core::fmt::Write + ?Sized>(
                    &self,
                    dest: &mut W,
                    _values: &dyn askama::Values,
                ) -> askama::Result<()> {
                    let mut write_lang = |c| match c {
                        $(Il8nShortCode::$code => Some(dest.write_fmt(format_args!($text, $(self.$arg,)* ))),)*
                        #[allow(unreachable_patterns)] 
                        _ => None,
                    };
                    write_lang(self.code).unwrap_or_else(|| {
                        write_lang(FALLBACK_IL8N).expect("Invalid language code");
                        Ok(())
                    })?;
                    Ok(())
                }
            }
            return __LangFmt { code, $($prop,)* }; 
        }
    };
} 

macro_rules! tr_il8n_fmt {
    ($name: ident, ($($prop: ident : $t: ty),*), { $($code: ident => ($text: literal, $($arg:ident),*) ),* } ) => {
        pub fn $name(code: Il8nShortCode, $($prop: $t,)* ) -> String {
            let find_lang = |c| match c {
                $(Il8nShortCode::$code => Some(format!($text, $($arg,)* )),)*
                #[allow(unreachable_patterns)] 
                _ => None,
            };
            find_lang(code).unwrap_or_else(|| find_lang(FALLBACK_IL8N).expect("Invalid language code"))
        }
    };
}


tr_il8n_fast_fmt!(test, (value: u32), { 
    En => ("test {}", value), 
    Ar => ("اختبار {}", value) 
});

tr_il8n_fmt!(fmt_test, (name: &str), { En => ("test {}", name), Ar => ("اختبار {}", name) });
tr_il8n!(tr_filter, { En => "filter", Ar => "تصفية" });

fn test_filter(code: Il8nShortCode) {
    tr_filter(Il8nShortCode::En);
    fmt_test(Il8nShortCode::Ar, "hello world");
    test(Il8nShortCode::En, 10);
}

[seijikun]

Fluent-bundle is very useful, but it's evaluated at runtime, which is not a good fit for rinja, IMO.

@Kijewski I completely understand where you're coming from, since you can't validate that all required translations are available at compile-time when it's evaluated at runtime. But I would like to point out that it is a valid use case to want to also translate lines that are not statically available in the template file, but instead get pushed in via the Template struct's fields.
(For example, data downloaded from a server before intantiating the Template)

[MrSpock]

I was hit but this missing feature. Without i18n askama use in any serious project with mutliple languges support is a blocker.
I have realtively small but already production where request was made to make another language version. I choose askama for template engine and now it seems I have to rewrite whole templating engine because of i18n lack of support.

[GuillaumeGomez]

@MrSpock Please refrain commenting if it's not adding anything useful to the conversation. Neither you nor your company is paying for this code so trying to shame us for lack of features is not having helping in any way.

[MrSpock]

Please let me know how in any way one can express that this ticket is important other then a comment ?
it is quite popular on github to express importance of given ticket/feature by commenting on it since there is basically no other way.

[GuillaumeGomez]

The issue is open. Adding comment not bringing anything except "I need this feature" is not gonna make things faster in any way.

[Ryuno-Ki]

As I am experimenting with some shared code between Nunjucks (of Node.js land) and Rust I added i18n support as filter to Askama using i18next resp. rust-i18n:

#[macro_use]
extern crate rust_i18n;

use askama::Template;

i18n!("src/locales");

struct Logo {
    alt: String,
}

struct I18n {
    logo: Logo,
}

#[derive(Template)]
#[template(path = "atoms/Logo.njk")]
struct LogoAtomTemplate<'context> {
    logo: &'context str,
    i18n: &'context I18n,
}

mod filters {
    pub fn t<T: std::fmt::Display>(
        s: T,
        _: &dyn askama::Values,
    ) -> askama::Result<String> {
        let s = s.to_string();
        Ok(t!(s).to_string())
    }
}

fn render_template() -> () {
    let logo = String::from("/src/assets/logo.png");
    let i18n = I18n {
        logo: Logo {
            alt: String::from("page.welcome.logo.alt"),
        }
    };

    let template = LogoAtomTemplate {
        logo: &logo,
        i18n: &i18n,
    };

    assert_eq!(
        template.render().unwrap(),
        "<img src=\"/src/assets/logo.png\" alt=\"Logo\" />"
    );
}

fn main() {
    render_template();
}

With a JSON at src/locales/en.json using Version 1 (see rust-i18n docs) and a nested object.

The Nunjucks template src/components/atoms/Logo.njk looks like

<img src="{{ logo }}" alt="{{ i18n.logo.alt | t }}" />

askama.toml

[general]
# Directories to search for templates, relative to the crate root.
dirs = ["src/components"]
default_syntax = "njk"

[[escaper]]
path = "askama::filters::Html"
extensions = ["njk"]

# Same as Askama
[[syntax]]
name = "njk"

[martinetd]

As I am experimenting with some shared code between Nunjucks (of Node.js land) and Rust I added i18n support as filter to Askama using i18next resp. rust-i18n:

Thanks for sharing!

I don't see how the locale is determined (whether to use en or another language), that's the locale used when running the server?
An extractor can be made for axum that gets it either from the client request or whatever session info, so ideally the language would be another field of the template to be useful in the wild

[Ryuno-Ki]

In this specific example, the locale is determined at build-time: https://github.com/longbridge/rust-i18n/#usage

You can pass it along when running the t! macro: https://github.com/longbridge/rust-i18n/#get-localized-strings-in-rust