Western Language Support in 2024?

[Geep]

In considering my possible upcoming TDM projects, such as upgrading some fonts to Latin-1 or "TDM Latin", I reviewed the current status of support for European Latin-based languages.

Basically, for FMs, the translation system has fallen into disrepair and disuse (or perhaps kicked into the grave). Neither "converted" (i.e., #str using) FMs nor Language Packs are now being maintained and distributed by thedarkmod.com.

As to other sites, for Cyrillic/Russian, DarkFate provides such resources. Are there any such sites in native Western European tongues, say, German, French, Italian, that likewise offer translated TDM FMs?

So, turning back to thedarkmod.com, questions -
- is Western Language support (outside the main menu system) officially dead?
- if so, maybe it's time to stop pretending otherwise.
- if not, should the converted #str system be revived, with better infrastructure?
- or an alternative translation system be devised... if so, what?

I'm just trying to identify work worth doing. (I started to draft a doc with a litany of problems with Western language support, and possible sub-projects to address them, but TL;DR. And really, not pertinent if support is considered dropped.)

Thanks for your thoughts.

[datiswous]

20 hours ago, Geep said:

Neither "converted" (i.e., #str using) FMs nor Language Packs are now being maintained and distributed by thedarkmod.com.

I don't understand this. When I download a mission, I can still download and update the translation files for it using the build-in mission downloader.

Also, the language packs are downloadable for the missions that have them.

Nobody is making these language packs though. Not sure why. I heard the translation conversion script is hard to use. I only know 2 languages and there's no need for language packs in one of those (Dutch).

Edited April 21 by datiswous

[Frost_Salamander]

23 minutes ago, datiswous said:

Nobody is making these language packs though. Not sure why. I heard the translation conversion script is hard to use. I only know 2 languages and there's no need for language packs in one of those (Dutch).

I wanted to internationalise The Hare in the Snare before I released it but I couldn't get the scripts to work and had some other questions as well.  I posted about it Newbie Dark Radiant Questions and nobody replied, so I gave up.  It being my first FM I didn't feel confident digging into it and just wanted to release the FM. I just got the impression that it wasn't a big deal and nobody cared about it.

From memory, the script worked once but I needed to re-run it and just couldn't get it to work.  It's also Perl which nobody really uses anymore.  I think I recommended to a 'I want to help' person to convert it to Python, but that didn't go anywhere.  I don't think you need the script though if you internationalise from the very beginning (i.e. create your own dictionaries).

At the end of the day it's extra work for the mapper and if it adds too much overhead (i.e. is broken or is a PITA) then nobody is going to use it.

https://wiki.thedarkmod.com/index.php?title=Internationalization

 

[nbohr1more]

If all written strings were kept in XD files, the process would be easier.

We shouldn't need map edits for this.

The current system makes mission updates a real problem to the point that only long dormant missions are viable for translation.

[Geep]

6 hours ago, datiswous said:

I don't understand this. When I download a mission, I can still download and update the translation files for it using the build-in mission downloader.

Also, the language packs are downloadable for the missions that have them.

Then your experience is different than mine. I just recently downloaded a very old mission, The Outpost, that is listed as having German, French, Italian translations, to test out language capabilities. It was not converted (no "#str" found in .map file). Nor was there a language pack included (at least not from thedarkmod downloader).

[datiswous]

Well if you go to the mission downloader again, it will show a new update for the mission, which is the translation pack.

 

If I select Germain language and activate The Outpost, TDM will not start anymore (2.13 dev)

[Geep]

6 hours ago, datiswous said:

Nobody is making these language packs though. Not sure why. I heard the translation conversion script is hard to use.

That is one reason.

6 hours ago, Frost_Salamander said:

the script worked once but I needed to re-run it and just couldn't get it to work.  It's also Perl which nobody really uses anymore. 

That is 2 more reasons. You'd like a script that, if you had to run it again, would "do the right thing". Unfortunately, that right thing is very hard to program, and needs IMHO to be both bidirectional and with a better method of string version control, to support both the FM author's updates and potentially multiple translators.

4 hours ago, nbohr1more said:

The current system makes mission updates a real problem to the point that only long dormant missions are viable for translation.

Yes, another reason. Currently, it is my understanding that updating an FM (from the non-converted copy) and running the conversion script again causes mis-alignment of newly-generated #str values and previous .lang #str values.

Another important cause of "nobody is making these language packs" is that Dark Radiant at best tolerates converted FMs. It offers no special translation support, as expressed in this code comment: "...we don't have any support for parsing the mod-specific translation data...." [from DR's DifficultySettingsManager.cpp].

6 hours ago, Frost_Salamander said:

At the end of the day it's extra work for the mapper and if it adds too much overhead (i.e. is broken or is a PITA) then nobody is going to use it.

That's where we are now. So officially give up on FM Western translations? Or improve the #str system to make it work for everyone? Or invent a new system?

4 hours ago, nbohr1more said:

If all written strings were kept in XD files, the process would be easier.

We shouldn't need map edits for this.

A new system. What would that look like to the FM author? To a non-author translator

[OrbWeaver]

This is how i18n typically works in code:

1. Developers write the strings in English (or their native language), but mark all the strings with a function/macro which identifies them for translation. In C++ this might be _("blah") or tr("blah") — something which is short and easy to write.
2. A tool (which may be integrated into the build system), extracts all the strings marked for translation into a big list of translatable strings. This list is then provided to the translators, who do not need to be developers or compile the code themselves. They just create a translation for each listed string and send back a file in the appropriate format (which may or may not be created with the help of translation tools, perhaps with a GUI).
3. At runtime, the code looks up each translatable string, finds the corresponding translated string in the chosen language, and shows the translated version.

At no point do developers (who in this case would be mission authors) have to mess around with manually choosing string IDs. All they do is use the appropriate function/macro/syntax to mark particular strings as translatable. String IDs may be used internally but are completely invisible to developers.

I suggest that any system that involves instructions like "search the list of known strings for a similar string" or "manually choose a string ID between 20000 and 89999 and then write it as #str_23456" are over-complicated, un-ergonomic and doomed to be largely ignored by mappers.

[datiswous]

On 4/21/2024 at 8:30 PM, nbohr1more said:

If all written strings were kept in XD files, the process would be easier.

This might be frowned onto by tdm mappers/veterans, but I find it strange that it's considered ok to use text inside entities, which can then be found inside map files. That in general seems badly organised.

Like for example with text decals: https://wiki.thedarkmod.com/index.php?title=Text_Decals_for_Signs_etc.#The_Decal

Personally I think xdata could/should be used for all instances of text (except maybe darkmod.txt and subtitles), including readme files and GUI files.

Or at least some kind of external text file. Could be something else, but the xdata format seems good and supported.

So the sign text prefab could have a default xdata_contents spawnarg instead of gui_parm1 .

Edit:

Objective description text is also inside an entity inside the map file, but it could just as well be a reference to an xdata file.

I posted about this before, here:

 

Edited April 23 by datiswous

[nbohr1more]

4 hours ago, OrbWeaver said:

This is how i18n typically works in code:

1. Developers write the strings in English (or their native language), but mark all the strings with a function/macro which identifies them for translation. In C++ this might be _("blah") or tr("blah") — something which is short and easy to write.
2. A tool (which may be integrated into the build system), extracts all the strings marked for translation into a big list of translatable strings. This list is then provided to the translators, who do not need to be developers or compile the code themselves. They just create a translation for each listed string and send back a file in the appropriate format (which may or may not be created with the help of translation tools, perhaps with a GUI).
3. At runtime, the code looks up each translatable string, finds the corresponding translated string in the chosen language, and shows the translated version.

At no point do developers (who in this case would be mission authors) have to mess around with manually choosing string IDs. All they do is use the appropriate function/macro/syntax to mark particular strings as translatable. String IDs may be used internally but are completely invisible to developers.

I suggest that any system that involves instructions like "search the list of known strings for a similar string" or "manually choose a string ID between 20000 and 89999 and then write it as #str_23456" are over-complicated, un-ergonomic and doomed to be largely ignored by mappers.

No disagreement on this but one clarification:

You can use characters instead of numbers, eg. #str_mycrazyspeech . This was discovered around 2.10 but is not widely known.

[nbohr1more]

On 4/21/2024 at 6:43 PM, datiswous said:

Well if you go to the mission downloader again, it will show a new update for the mission, which is the translation pack.

 

If I select Germain language and activate The Outpost, TDM will not start anymore (2.13 dev)

The translation pack has the german string in the base pack rather than in the fm subfolder. I will fix it in the mission database.

Edit: Fixed in the mission db

Edit 2: Nope. Not exactly fixed.

It seems that lang files in the mission string folder need to be "complete" because they fully override the core strings.

If I am correct, this was broken in 2.11 when we permitted in-file overrides of core files in missions.

Edit 3:

Still broken in 2.10, rolling back to 2.07

Edit 4:

Still broken in 2.07. Something has gotta be wrong with the translation specific to this mission.

Edit 5:

The core mission XD files don't use the strings so nothing happens if the lang files are in the strings/fm subfolder. Probably means that the translation packs "never worked" for many missions unless impacted players sought out special editions of the missions on Tels' server. What a mess.

Fuzzy recollection time:

I think Tels was trying to push the team to mass convert all missions to move XD data into strings/fm/english.lang but nobody wanted to broach it and even mission authors weren't happy about this way of handling things.

If the translation pack takes precedence, the best way forward is to include the converted XD file into it.

Testing...

Edit 6:

Couldn't get the XD update to work, so I decided to checkout Darkfate's version. It works flawlessly.

I copied their pack into the standard translation pack and the added string files for the other languages worked as well. Darkfate's packs include map files so I'll need to study whether we can avoid that. Otherwise we are basically doubling our mission db or "damaging" our hosted versions to make them translatable. Since this mission is so small and probably will never be edited again we can probably use darkfate's version as-is.

[OrbWeaver]

9 hours ago, nbohr1more said:

I think Tels was trying to push the team to mass convert all missions to move XD data into strings/fm/english.lang but nobody wanted to broach it and even mission authors weren't happy about this way of handling things.

That is my recollection too.

The i18n system was basically Tels' personal pet project (hence the Perl script which is unmaintained because nobody in the world except Tels and Larry Wall have any interest in writing code in Perl). Because of the various implementation problems and general user-unfriendliness, Greebo didn't approve of merging it into the main mod, so it became a sort of optional extra that individual mappers could use by accessing various resources on Tels' personal server.

[datiswous]

I tried the German translations for fm The Thiefs Den. Doesn't work. When I include the russion translation pack from Darkfate, the original German translations work as well. The translation package from Darkfate adds a lot more files, including the xd and map files.

[nbohr1more]

The translation pack for The Outpost is now essentially the same as the one at Darkfate.org. I'll probably do this for most of the old dormant missions.

[datiswous]

I was looking into this a bit more and conclude that the translation system in itself is actually an universal external data system. So instead of xdata, you could use a #str_someword and then reference to that in the /strings/fm/english.lang file.

Well I guess it's probably not really news for some of us.

 

Edit: This actually works fine:

File english.lang for Thief's Den

	"#str_Nobody crosses me! Must get back Frothley's scepter Creep stole off me."	"Nobody crosses me! Must get back Frothley's scepter Creep stole off me."

Then in the objective editor you add #str_ before every objective text. This way it's easier and you can still see which is which.

Edited April 25 by datiswous

[datiswous]

Correctly me if wrong.

Apperantly xdata formatting doesn't work well with the lang references. So this:

maps/man2/mission_briefing
{
 "num_pages": "1"
 
 "page1_body" :
 
   "#str_b1"
   ""
   "#str_b2"
   ""
   "#str_b3"
}

Doesn't work.

You can only do this:

maps/man2/mission_briefing
{
 "num_pages": "1"
 
 "page1_body" : "#str_b1"
}

And then in the lang file you have to add the whole text of page 1 and get newlines via \n etc.

So writing a (text only) briefing  or readable and after that translating, is probably not something you want to do. I didn't actually test this with readable's, but it seems to me it would have the same limitation.

Edited April 25 by datiswous

[Geep]

I mentioned this elsewhere, but just a historical note: the existing i18n.pl conversion script expects only a numeric value after #str_ in its pattern matching regex (and possibly method of hashing).

I do think @datiswous's idea to have key/value pairs with values like:

"#str_Nobody crosses me! Must get back Frothley's scepter Creep stole off me."

is a good way forward. Maybe with this version, DR could actually choose over time to provide some .lang support. And probably the engine would have to create hash tables to avoid slow string matching with these long non-numeric strings.

(Oh, one other thing, since we're talking about The Outpost. When I played it earlier the month under 2.12, my screen would periodically go black for a second or two, every minute or two. I wonder if anyone else sees this, or just my odd Intel graphics chip?)

 

[Geep]

One of the problems with the old #str_number system, that would not be automatically solved in the new #str_phrase system, is lack of versioning/history. For example, suppose in the FM I provide a new string:

#str_Mother!

which (by magic TBD, ideally in DR) generates this placeholder in all the .lang files:

"#str_Mother!" "Mother!"

The translator in the fr.lang file later does this:

"#str_Mother!" "Mere!"

Still later, the FM author revises the string in DR:

"str_Mother!!!"

In a naive implementation, this breaks the link to the existing translation(s), which becomes orphan in all the .langs, and creates a new entry:

"#str_Mother!!!" "Mother!!!"

OK, how could we do better?

Case 1 (as above): the FM author knows the change is trivial, and so (at revision time in DR) might ask the translations to be retained but marked for review. So maybe the fr.lang files gets:

// WAS UNTIL 2025-01-01: "#str_Mother!" "Mere!"

"#str_Mother!!!" "Mere!" // Needs minor review

with removal of the orphans (after they become comments and moved to before their replacements)

Case 2, where the revision is not trivial: the FM author (at revision time in DR) might ask the translations to be replaced by english but marked for review, e.g.:

#str_Mother of Pearl!

causes the fr.lang files to have (with orphan removal as case 1):

// WAS UNTIL 2025-01-01: "#str_Mother!" "Mere!"

"#str_Mother of Pearl!" "Mother of Pearl!" // NEEDS REDO

Then the translator could eventually fix it:

// WAS UNTIL 2025-01-01: "#str_Mother!" "Mere!"

"#str_Mother of Pearl!" "Nacre!" // Done 2025-03-14 by Henri

 

[datiswous]

Maybe have a lang file check script with suggested fixes. Or even a console command for checking translation file compatibility.

[Geep]

I think you need to do things at the exact moment in DR that the user changes a field, so you can capture both the old value and the new value. Would be way better if DR just handled everything natively, but could be done through a hook to an external script too, I suppose. Or some complicated logging.

[Geep]

22 hours ago, datiswous said:

xdata formatting doesn't work well with the lang references.

I agree, that is an issue. It would be better if the system handling briefings/readables could be revised as you indicated, to handle individual sentences as #str_, rather than whole pages.

Baring that, having a key field like "#str_This is a whole page\nfull of text that goes on and on and on [...] until done" would appear as a very long single line. That is nasty to look at in the Readables Editor, and even worse in the .langs file, where the too-long text would appear twice on a line (once with #str_ prefix of English version, another in translation).

So for those, it would be better in the short term to stick to symbolic keys, e.g., existing #str_12345 or revised form #str_myfm_book_of_spells_page_1

I might add, in the longer term, enhancing the Readable Editor to use the .lang files would be an enormous improvement for FM authors and a significant accomplishment. A fair amount of work though, but probably doable in increments over several releases.

Edited April 26 by Geep

[nbohr1more]

The project to round up viable orphaned translations is mostly complete now.

https://bugs.thedarkmod.com/view.php?id=4726

Since Darkfate has their own translations it took awhile to merge the older translations with the darkfate formatting since they generated new string numbers from scratch.

A few missions are so radically changed that the old translations are practically worthless now ( and some Darkfate translations are also incompatible with the latest mission versions ) but most of the work was salvageable.

I added a "Translation Pack" column to the Missions wiki:

https://wiki.thedarkmod.com/index.php?title=Fan_Missions_for_The_Dark_Mod

all entries marked with "Yes" are working translations that use Darkfate's translation design ( includes xdata, GUI and Map files if the author didn't perform localization changes in their own mission pack ).

Sadly, these changes do not show in the mission downloader list. You will need to delete mission_l10n.pk4 files from the darkmod/fms/mission_name folders to acquire the new translation packs that "actually work".

Not all mission packs contain all languages. Mostly you can count on Russian being in there, but a good percentage have German and Italian strings too. French, Polish, and a few other languages are represented but not very broadly. Hopefully now that the packs work other contributors will add more language strings.

 

Mission administration details:

Going forward, it would be courteous if the mission admins ( like myself ) would diff the map files when new mission updates are supplied and replace all the objectives, inv_name, pop-up messages, and shouldered names in a copy of the new map file with the previous string codes then copy that file to the map folder in the translation pack folder.

Ideally, authors will be encouraged to run their missions through the converter so only new strings need be added to the packs but not everyone likes that workflow and how cumbersome it makes future edits to the mission.

[nbohr1more]

The map files in NHAT v4 were too large to diff in Notepad++ or Meld ( etc) so I decided to test out the old I18N.pl script and it still works fine for me.

Caveats?

It doesn't understand campaign packs so you have to split the maps and xdata and create a fake darkmod.txt without the "mission 1 , mission 2, etc"  part. Then you have to renumber the "#str_xxxx" entries in the other maps so they don't collide. Luckily find and replace can easily fix that.

A lot of work to get this working with the latest NHAT version, phew!

[datiswous]

Do we keep using the current system, or are there plans for a new way for example using an universal translation format?

The two biggest issues with the current system I find is that currently editing larger text is not great and the original map file (or more) has to be modified, which means if the mapper doesn't want to bother, it's not going to happen, because external modification of map files is a no-go I think.

[MirceaKitsune]

Not sure if related but: My dream with creating a multi city hub FM was to allow the player to travel to different countries by ship. Part of the plan was having a few readables written in relevant languages, granted I can use Google Translate to obtain a few sentences in an Asian or Arabic or other language. Problem is that last I checked, readables only allowed english characters, everything else will appear as a rectangle: Most fonts don't even support symbols like French or Romanian characters (eg: ăâîțș). Too much work would likely be needed to patch the existing fonts, but maybe we can find and include a font set designed explicitly for those languages that special characters in readables can fall back to?