Proposed I18N Improvements for TDM 2.15

[Geep]

I am working on improvements I can make to all.lang. The potential scope of work is overwhelmning, so I'm focusing at this time on the Main Menu, particularly Settings. I'd appreciate your review of the workplan, which has 3 main components -
1) commenting-out #str phrases no longer useful
2) adding #str to replace phrases hard-coded in English, in the relevant GUI files.
   a) in [English]
   b) in other languages (using AI)
3) For existing Settings #str in the GUI files, that were translated for some languages: do the remaining languages (using AI).

I'm excluding this -
- strings in *.cpp files
- translations into Dutch and Swedish, where almost all strings are untranslated. Each of these is a project in itself, and I'm not up to it currently.

Reviewers... My base is the 2.14 all.lang. If you know of 2.15 changes that could affect individual #str, please mention it here

Details

I have a revised all.lang WIP with the following changes done so far:
- Group (1) work in both [English] and all other sections, as summarized and itemized here...

Spoiler

Strings Thought to Have No Current or Future Use
=================================
For now, these are given leading "//" and trailing reason in brackets. Change is made in {English] and in all other language sections of revised all.lang.
In the latter (if the #str_id pre-exists), the translated phrase is retained, but for brevity not any original comment (except tag "// stays the same"), instead replaced with the English trailing reason in brackets

Reviewers - do any of these need to NOT be commented out? Can the trailing remarks in brackets be improved?

In Settings
------------
//    "#str_02163"    "Bloom Intensity" [Separate slider circa 2014, now folded into "Bloom" setting]

//    "#str_02239"    "Gamma Standards"    [popup defined in mainmenu_settings_gamma.gui, but now dropped due to technical problems]

//    "#str_02301"    "Use-by-Frobbing"     [Superceded in 2023 by "#str_menu_holdfrob_delay_title" "Hold-Frob Interact"]
//    "#str_menu_useonfrob_tooltip"    "Automatically use currently selected inventory item\n on highlighted object if possible, e.g. keys on doors"    [Superceded in 2023 by "#str_menu_holdfrob_delay_tooltip"]

//    "#str_02321"    "Show Menu Tooltips"    [To free up setting row, now always on, unless disabled with DarkMod.cfg's "tdm_show_menu_tooltips"]
//    "#str_menu_show_menu_tooltips_tooltip"    "Show tooltips like this one"    [See also commented-out #str_02321 "Show Menu Tooltips"]

//    "#str_02932"    "Multi Core Enhancement"    [In 2.09, always on by default for cvar com_smp, so dropped from menu.]

//    "#str_07229"    "Player Gender"    [Commented-out in mainmenu_settings_audio.gui at request of Springheel and Greebo]
//     "#str_07218"    "Female;Male"    [Choice for commented-out "#str_07229"    "Player Gender"]

//     "#str_07230"    "Surround Sound"    [Commented out in mainmenu_settings_audio.gui, rows 1; row used by OpenAL instead.]
//    "#str_07226"    "Stereo/Surround 5.1/Surround 7.1"    [Choice for commented-out #str_07230 "Surround Sound". Also cvar s_numberOfSpeakers in snd_system.cpp marked "No longer works!"]

//    "#str_02161"    "Reverse Channels"    [This No/Yes item commented-out in mainmenu_settings_audio.gui, row 2; row used by OpenAL instead. Also, cvar "s_reverse" in snd_system.cpp, marked "No longer works!"]

//    "#str_07325"    "Soft Shadows" [presumably renamed to "#str_02943"    "Shadow Softness"  in 2.07]
//    "#str_07326"    "Screen-Space Ambient Occlusion"    [ssao, shortened to "Ambient Occlusion" in 2.08]
//    "#str_07327"    "Screen-Space Indirect Lighting" [ssil, possibly dropped as ineffective?]
//    "#str_07328"    "Enhanced" [possibly used with ssil? with Multi Core Enhancement?]
//    "#str_07329"    "Render Shadows" [no longer used, or just renamed?]
//    "#str_07330"    "Near;Medium;Far;Full" [Possibly was used with Render Shadows?]

In Mission Overview
-----------------------
//    "#str_02186"    "[Cleanup]"    [For pseudo-button, next to "Space used:" in FM overview, to erase FM from local disk. Commented-out in mainmenu_newgame.gui]
//    "#str_07204"    "Cleanup Mission Folder"    [See commented-out #str_02186 "[Cleanup]" ]
//    "#str_07205"    "Mission Folder Cleaned"    [See commented-out #str_02186 "[Cleanup]" ]
//    "#str_07206"    "Cleaning up Mod Folder..."    [See commented-out #str_02186 "[Cleanup]" ]

Strings Corrected in [English]
===================
Poor English capitalization:
     - setting_row_6:
     "#str_menu_sharpen_image"    "Sharpen image" --> "Sharpen Image" // Video/General/row 6 

- Group (2a), only in [English] so far, as summarized and itemized here...

Spoiler

Here is a list of Main Menu Settings strings that are hard-coded English in TDM 2.14, with suggested replacement using new #str names.
Indented lines with trailing comments is how they would appear in updated all.lang [English].
They are preceded by a non-indented supplemental comment lines, just for here 

// In tdm_gui01\guis\mainmenu_settings_audio.gui
    "#str_menu_reverb"    "Reverb Volume"    // Audio
// In main_menu_video.gui; appears as Settings/Video/General/row 12
    "#str_menu_texture_quality"    "Texture Quality"    // Video/General
 // Next 2, In main_menu_video.gui; appears as Settings/Video/Advanced/row 5
    "#str_menu_volumetric_fx"    "Volumetric Effects"    // Video/Advanced
    "#str_menu_volumetric_fx_tooltip"    "Disable Volumetric light-shaft effects for improved performance"
// Next 2, in main_menu_video.gui; appears as Settings/Video/Advanced/row 6
    "#str_menu_parallax_map"    "Parallax Mapping"    // Video/Advanced
    "#str_menu_parallax_map_tooltip"    "Disable Parallax Mapping for improved performance"
 // In mainmenu_settings_controls.gui; appears as Settings/Controls/General/row 2
    "#str_menu_toggle_creep"    "Toggle Creep"    // Controls/General
// Remainder are for the Mission List Settings popup, invocated by the gear icon on Mission List.
// This is already defined:    "#str_mission_list_settings_title" "Mission List Settings"
    "#str_mission_list_settings_title_style"    "Title Style"
    "#str_mission_list_settings_style_1"    "Style 1"
    "#str_mission_list_settings_style_2:    "Style 2"
    "#str_done"    "Done"    // Used for instance in Mission List Settings popup
// but leave hard-coded the popup's "Example Title, The" and "The Example Title", since FM names are seemingly always in English at this time.
// (While I18N.cpp has an ArticleDict to try to support this location-swap in other languages, there no doubt will be issues in some languages.)

It is anticipated that there will eventually be a bugtracker post to get help with corresponding GUI changes and testing.

Workplan for Group 3 is WIP.

[Geep]

To give you an idea of what I looking at for Group 3, much of what I want to cover are strings that will be more consistently translated if properly grouped, rather than the order they appear in all.lang [English]. I'm trying YAML as the light-weight structuring mechanism. A hand-built draft of Settings is mostly done. Here's a small example of that:

Fragment within the yaml hierarchy main_menu:/settings_page_layout:/body_settings:/

      video:
        # In main_menu_video.gui, unless indicated otherwise...
        general:
        - subheading:
          - line_1:
            - phrase_id: "#str_02298"
              phrase_en: "Changes to screen size & aspect ratio, or full screen require a restart."
          - line_2:
            - phrase_id: "#str_02299"
              phrase_en: "Fullscreen is Desktop Resolution, use Render Scale to adjust"
        - setting_row_1:
          - label_id: "#str_02154"
            label_en: "Fullscreen"
          - choice_id: "#str_04222"
            choice_en: "Windowed;Fullscreen;Borderless"
          - tooltip_id: "#str_menu_fullscreen_tooltip"
            tooltip_en: "Windowed: normal window of any size\nBorderless: covers whole screen, fast switching\nFullscreen: exclusive, can change resolution"

I'm new to YAML, so I may not have the structure quite right yet (e.g., when to use "-").

[wesp5]

Isn't the core team working on replacing all numbered strings with named strings for the next release?

[Geep]

Hmm, if so, I didn't see the memo... wouldn't be the first time. 

Going to all named strings is a whole lot of work, but would greatly aid DR and string maintenance. And if already completed would simplify and improve further translations

[stgatilov]

@Geep, doing massive restructure on old state is not a good idea.
Since you are marked as Contributor, don't you have direct SVN access to the assets repo?
If no, we should simply give you the access.

I can't say I see anything still used in your list to be deleted.
Usually the best way to check is to search over guis/ + scripts/ + maybe C++ code.
Feel free to delete unused and reorder the lines in the strings file.

I don't really like the yaml idea, because you are basically trying to extend the already overly massive strings framework.
Now we have tons of old Perl code that nobody wants to mess with, and you'll add some more code which nobody will want to mess with a few years later? And what will yaml give you, really?

If you reorder the lines in the existing format, you will get the grouping you want. The current order is messy simply because everyone is lazy to search for similar / related strings every time they want to add a new menu settings. And I'm afraid things will stay this way. Even if you make perfect order, it will slowly rot over time.

Finally, here are some of my thoughts regarding strings framework:

1. I would be happy to make all.lang the only file with strings, encoded in UTF-8. The engine should read it directly and use per-language table to convert permitted UTF-8 characters into its weird 8-bit encoding which is uses now (without generic UTF-8 decoding). This way we can finally get rid of the old Perl stuff.
2. The strings file should support full-text replacements in addition to placeholder replacements we have now. That would allow translators to translate a mission without forcing the author to turn it into a total mess which is harder to work with. The full-text replacements should be used for missions, placeholders should be used for core.

The p.1 idea will require changes in the engine, but I think they are quite lightweight. Reading UTF8->8-bit character table per language, and convert UTF-8 into native format when strings are loaded. What scares me more is trying to understand how it works now and constructing these new tables.

[Geep]

@stgatilov, I was NOT proposing renaming all the numbered strings to alphanumeric (even though that would likely slow down the ordering rot you mentioned.) I was just responding to @wesp5 hearing that someone was already planning on doing so, which surprised me.

Yes, I do have access to the SVN repo, thanks. I just try to avoid it as much as possible. For the current work, I'm using 2.14 as the stable base line (for .cpp code, all.lang, *.gui*, etc.) , because things are complicated enough as is.

The yaml is just an organizing tool that "might" be helpful for my AI translation efforts, because it can formally encode relationships between strings. Not sure about usefulness yet. I'm not proposing that the TDM engine should read it.

I could reorganize at least parts of all.lang similarly, if everyone's OK with moving further away from alphanumeric ordering of #str_ids.

The two ideas for improving strings I'm also on-board with. Particularly for #1, which I've also floated as a trial balloon in the past. BTW, for that, my gen_lang_plus has related C++ code that might be of some interest (albeit doesn't use the idTech4 class structure and is Windows-based).

For #2, I can see some disambiguation problems, but nothing insurmountable. Given that DR will never properly support #str_ids, seems reasonable.

What time frame are you thinking about for these?

 

[Geep]

Speaking of gen_lang_plus, I just recycled some its code to make a visualization tool "pivot_all_lang" for me to see all the translations of a given string together. Basically, it pivots data like...

[English]
    "#str_02001"    "Warning"
    "#str_02000"    "Error"    // comment with english
    "#str_02002"    "Connection Error."
[French]
    "#str_02001"    "Warning French"
    "#str_02000"    "Error French"    // comment with french
    "#str_02002"    "Connection Error French."

into this form...

    "#str_02000_english"    "Error"    // comment with english
    "#str_02000_french"    "Error French"    // comment with french

    "#str_02001_english"    "Warning"
    "#str_02001_french"    "Warning French"

    "#str_02002_english"    "Connection Error."
    "#str_02002_french"    "Connection Error French."

I've fed this with all.lang of 2.14, and here is the resulting "all_lang_pivot.txt" (about 800 KB). There are some added flags, which you can read about in the file preamble. Eventually I'll post the code + wiki description. For now, the output file:

https://drive.google.com/file/d/1BDz-DWe62NTGMA0dxdngnepagi3dwTNQ/view?usp=sharing

[datiswous]

33 minutes ago, Geep said:

Given that DR will never properly support #str_ids

Why is that?

[Geep]

42 minutes ago, datiswous said:

Why is that?

From what I could glean from old forum posts (mostly before my time), strong dislike of the #str methodolgy.

[stgatilov]

2 hours ago, Geep said:

@stgatilov, I was NOT proposing renaming all the numbered strings to alphanumeric (even though that would likely slow down the ordering rot you mentioned.) I was just responding to @wesp5 hearing that someone was already planning on doing so, which surprised me.

Right now we just make new placeholders string-based, but I guess nobody wants to do massive renaming of existing ones.

Quote

Yes, I do have access to the SVN repo, thanks. I just try to avoid it as much as possible. For the current work, I'm using 2.14 as the stable base line (for .cpp code, all.lang, *.gui*, etc.) , because things are complicated enough as is.

The problem is that if you massively reorder the strings and someone adds new strings in trunk, then those new strings will probably we dropped. Basically, any massive change either on trunk or on your side makes it unmergeable in the future.

Quote

What time frame are you thinking about for these?

I guess I'd better start with UTF-8 part. I really want to make the basic operation Perl-free.
As far as I understand, there is no need to change format: all.lang should be perfectly consumable as it is.
I think 2 weekends should be enough, but the next one I'll be on vacation... so should be ready in September, I suppose?...

As for the full-text replacements... I think it will be clear after the first part is done.
It will require some kind of format extension, so it is hard to predict.

1 hour ago, Geep said:

Speaking of gen_lang_plus, I just recycled some its code to make a visualization tool "pivot_all_lang" for me to see all the translations of a given string together. Basically, it pivots data like...

Mmmm... maybe we can reorder all.lang without breaking anything:

[English] "#str_02001"    "Warning"
[French] "#str_02001"    "Warning French"

[English] "#str_02000"    "Error"    // comment with english
[French] "#str_02000"    "Error French"    // comment with french

[English] "#str_02002"    "Connection Error."
[French] "#str_02002"    "Connection Error French."

Depending on how the parser was implemented originally, TDM might eat it fine, and you'll have this grouping.
Isn't it better for everyone, no?

[datiswous]

5 hours ago, Geep said:

From what I could glean from old forum posts (mostly before my time), strong dislike of the #str methodolgy.

Well DR is currently being worked on by a new guy.

[Geep]

15 hours ago, stgatilov said:

The problem is that if you massively reorder the strings and someone adds new strings in trunk, then those new strings will probably we dropped.

Towards the end of whatever I do with all.lang for 2.15, it would of course be necessary to check if any new strings have been added by others, and incorporate them. I would then make a bugtracker request for someone (who likes working with revision control systems) to assist the effort by adding the finalized file to the RCS.

[Geep]

16 hours ago, stgatilov said:

Depending on how the parser was implemented originally, TDM might eat it fine, and you'll have this grouping.
Isn't it better for everyone, no?

I don't know whether or not an average human translator for a particular language would prefer this pivoted file as the working file. Maybe breaking it up into less-sprawing subset files, e.g. french.lang as a UTF-8 file with JUST the pivoted english and french strings, no other languages. (and with all comments, not just the trailing-line ones that all_lang_pivot.txt retains).

 

[Geep]

12 hours ago, datiswous said:

Well DR is currently being worked on by a new guy.

Always good to get fresh blood. 

[Geep]

16 hours ago, stgatilov said:

so should be ready in September, I suppose?...

Hmm, sounds optimistic. The UTF8 to Latin-x conversions may take some time. My code uses some Windows-specific techniques, which probably need something different for your purposes.

[Geep]

I'm going back to work on my OP item 2b (new strings translated by AI).

I hope the pivoted file can help build a targeted glossary of a few select TDM terms in multiple languages.  I could then include that glossary in a multi-language AI translation prompt for the new strings.

After 2b is done, I'll probably postpone OP item 3, to consider reordering parts of all.lang (similar to yaml). Do the [English] part by hand, and ask AI to reorder the rest?

[Geep]

Looking at the hard-coded strings for which I offered new #str_id's, I decided to generate language strings for
just this item first:

 // In mainmenu_settings_controls.gui; appears as Settings/Controls/General/row 2
    "#str_menu_toggle_creep"    "Toggle Creep"    // Controls/General
//    "Disabled;Enabled"

The setting means (briefly) "change the Creep key from hold-to-use mode to press-to-toggle mode". Creep itself means move slowly (and steathly).

A challenge is that there are already translations for the "Creep" keybind setting, which it would be good to be consistent with to the extent possible:

Glossary for "#str_02422" "Creep" [English]:
    "Schleichen" [German]
    "Avancer lentement" [French]
    "Skradanie" [Polish]
    "Arrampicati" [Italian]
    "Sigiloso" [Spanish]
    "Mover-se sorrateiramente" [Portuguese]
    "Красться" [Russian]
    "Plížení" [Czech]
    "Araszolás" [Hungarian]
    "Plížiť sa" [Slovak]
    "Kryb" [Danish]
    "Furisarea" [Romanian]
    "Sürünme" [Turkish]
    "Sigilós" [Catalan]

For the "Toggle..." aspect, there are similar reference lists, e.g. "Toggle Crouch" vs "Crouch".

A constraint is that "Toggle Creep" is a Setting prompt, so the space for any translation is delimited.
I estimated that generally for Setting prompts, an appropriate string length upper limit would be
22 characters (including space & punctuation).

I prompted ChatGPT with the foregoing information (and general context about this being for a PC game platform). Several iterations were involved. Because of the character limit, some other synonyms were used in some cases, e.g.: "switch" or "alternate" for "toggle"; "move furtively" for "move slowly". To have consistency, the "Creep" string itself would need to change for several languages. Overall results:

For "Toggle Creep"...
German:      Schleichen umschalten
French:      Basculer marche lente
Polish:      Przełącz skradanie
Italian:     Alterna furtività
Spanish:     Alternar sigiloso
Portuguese:  Alternar furtividade
Russian:     Переключить 'Красться'    (here, single quotes replace the angle quotes that are not in TDM's char set.)
Czech:       Přepnout plížení
Hungarian:   Araszolás ki/be
Slovak:      Prepínanie plíženia
Danish:      Kryb til/fra
Romanian:    Comută Furisarea
Turkish:     Sürünmeyi aç/kapat
Catalan:     Alternar Sigilós
Dutch:    Sluipen schakelen
Swedish:    Växla smygning

Translation adds for "#str_02422"    "Creep"
Swedish:    Smyga
Dutch:    Sluipen

Translation changes for "#str_02422"    "Creep" (to better match Toggle Creep, and so allow latter to be 22 chars max)
French:    Avancer lentement -> Marche lente
Italian*:    Arrampicati -> Furtività
Portuguese:    Mover-se sorrateiramente -> Furtividade

* The word Arrampicati means "climb!", and seems wrong. Maybe this was supposed to be applied to the "Mantle" keybind setting

So, this is a slow process so far.

[Geep]

As suspected, Mantle and Creep were both translated in Italian as Arrampicati. Fixed. Foregoing all added to my all.lang WIP.

[stgatilov]

I got stuck on these two:

	"#str_02480"	"Sloven®ina"		// Slovak (® in ISO-8859-1 is č in our font)
	"#str_02481"	"Sloven¨®ina"		// Slovenian (southern slovenia) (¨ => š)

I believe Slovenia and Slovakia are two different countries, so the comments here are confusing.

I see that #str_02481 is not used anywhere.
In the code #str_2480 is used, and all the variable/filenames and comments call it "slovak".

The specific trouble for me is that this special doublequote symbol (code 168) is mapped to ? (question mark) in French, and the normal ? is also mapped to ?. So I get as situation that two UTF-8 characters are both mapped to question mark in French. I think I can just delete #str_2481.

But I'm now worried about general confusion.
Should we rename "slovak" to "slovenian" in the code and filenames?
I don't see any "slovak.lang" files in existing missions, so it should not break anything.

UPDATE: Here is the full list of UTF-8 characters which are mapped to the same character:

Lang "hungarian" key "#str_01050": char "63" obtained from both "{'õ'}" and "û"
Lang "hungarian" key "#str_02010": char "63" obtained from both "{'û', 'õ'}" and "?"
Lang "hungarian" key "#str_02462": char "63" obtained from both "{'?', 'û', 'õ'}" and "ñ"
Lang "hungarian" key "#str_02464": char "63" obtained from both "{'?', 'û', 'ñ', 'õ'}" and "ê"
Lang "hungarian" key "#str_02467": char "63" obtained from both "{'?', 'ê', 'ñ', 'û', 'õ'}" and "¬"
Lang "hungarian" key "#str_02469": char "63" obtained from both "{'?', 'ê', 'ñ', 'û', 'õ', '¬'}" and "à"
Lang "hungarian" key "#str_02480": char "63" obtained from both "{'à', '?', 'ê', 'ñ', 'û', 'õ', '¬'}" and "®"
Lang "hungarian" key "#str_08326": char "63" obtained from both "{'à', '?', '®', 'ê', 'ñ', 'û', 'õ', '¬'}" and "Õ"
Lang "slovak" key "#str_02462": char "63" obtained from both "{'?'}" and "ñ"
Lang "slovak" key "#str_02464": char "63" obtained from both "{'?', 'ñ'}" and "ê"
Lang "slovak" key "#str_02467": char "63" obtained from both "{'?', 'ñ', 'ê'}" and "¬"
Lang "slovak" key "#str_02469": char "63" obtained from both "{'?', 'ñ', '¬', 'ê'}" and "à"
Lang "slovak" key "#str_02480": char "63" obtained from both "{'à', '?', 'ñ', 'ê', '¬'}" and "®"
Lang "swedish" key "#str_02467": char "63" obtained from both "{'˝'}" and "Č"
Lang "swedish" key "#str_02480": char "63" obtained from both "{'˝', 'Č'}" and "č"

So I guess the script just replaces chars that it does not knowing specific language with question mark. Maybe simply allow it?

I think the general practice of trying to display the language names in its native language regardless of which language is active is bad, especially given that engine is not capable of displaying all the characters with each language. We should localize language names, i.e. display Russian as "русский" only when Russian is active, but "Russian" when English is active, "Rússo" in Portuguese, "Ruski" in Serbian, etc.

The user has to know a bit of English anyway in order to navigate through settings, and I believe everyone capable of downloading TDM already knows the name of his language in English.

[Geep]

15 hours ago, stgatilov said:

So I guess the script just replaces chars that it does not knowing specific language with question mark. Maybe simply allow it?

Yes. In my code (gen_lang_plus/UTF8_8859_Convert.cpp) I have screening routines that convert ALL possible UTF8 codes that are not in the target Latin-x to "?". This is defensive programming in case the translator is less aware of restrictions.

Let me look into that Slovak/Slovenia a bit.

[Geep]

ChatGPT confirms a sample of the strings in [Slovak] are indeed what's call "Slovak" in English. And further, in the Slovak language, “Slovak” (the language) is "Slovenčina". So the current main menu presentation is correct & consistent.

I agree that this can be commented out:

"#str_02481"	"Sloven¨®ina"		// Slovenian (southern slovenia) (¨ => š)

[Geep]

16 hours ago, stgatilov said:

I think the general practice of trying to display the language names in its native language regardless of which language is active is bad, especially given that engine is not capable of displaying all the characters with each language. We should localize language names, i.e. display Russian as "русский" only when Russian is active, but "Russian" when English is active, "Rússo" in Portuguese, "Ruski" in Serbian, etc.

The user has to know a bit of English anyway in order to navigate through settings, and I believe everyone capable of downloading TDM already knows the name of his language in English.

I certainly recognize the general technical drawbacks (see my "MultiLanguage Display" wiki article), but disagree that offering the user his own language name is a bad idea. Given that we currently have the language page working good enough, I don't see the need for change. (And I dream of a distant time when both TDM and the DAT font format can handle UTF8 natively within.)

Certainly navigation through settings in other languages can be improved. I hope to make some headway on that.

[stgatilov]

10 minutes ago, Geep said:

I certainly recognize the general technical drawbacks (see my "MultiLanguage Display" wiki article), but disagree that offering the user his own language name is a bad idea. Given that we currently have the language page working good enough, I don't see the need for change. (And I dream of a distant time when both TDM and the DAT font format can handle UTF8 natively within.)

I don't think it is okay for randomly force non-native letters onto languages.
For instance, there are 8 non-ASCII letters in the English set now.
Isn't it that every language has its own font texture?

{
    char 0xC2 0xAC : 0xAC             // ¬ -> ¬
    char 0xC2 0xAE : 0xAE             // ® -> ®
    char 0xC3 0xA0 : 0xE0             // à -> à
    char 0xC3 0xA2 : 0xE2             // â -> â
    char 0xC3 0xA7 : 0xE7             // ç -> ç
    char 0xC3 0xAA : 0xEA             // ê -> ê
    char 0xC3 0xB1 : 0xF1             // ñ -> ñ
    char 0xC3 0xBC : 0xFC             // ü -> ü
}

[stgatilov]

Anyway, the more important question is: do we want to change the format of UTF-8 strings file?

I mean the all.lang file can be renamed into all.utf8lang and consumed directly by the engine.
However, its format might seem a little bit... hard to extend.
Every line is just assumed to have two strings with fixed meaning.

Perhaps we want to add a keyword at the beginning of every line?
Something like this:

{
    language "english"
    map_key "#str_02050" "Byte"
    language "russian"
    map_key "#str_02050" "Байт"
    map_text "Unable to save" "Не удалось сохранить"
}

Here "map_key" is the old thing, "language" is just the more parser-friendly version of [Language], and "map_text" is potentially the command for writing full-text replacements.

[Geep]

    "#str_02050" "Байт"
    map_text "Unable to save" "Не удалось сохранить"