Proposed I18N Improvements for TDM 2.15

[Geep]

23 hours ago, stgatilov said:

I'm pretty sure I can write a Python script for that relatively quickly.

I'm got this. Easy to adapt from my C++ pivot_all_lang program. Basics are mostly working for .pot file gen, but needs a bit more work on header & comments. Then expand to do gen .po files.

I've downloaded the poedit editor. Just kicking the tires on the free version so far. There's some paid versions that tie in with AI translations, and that may be worth it for me (at least for a year: $40 for Pro+). I've also downloaded a github project with a set of sample/text  .po (and .pot) files for > 200 languages & dialects. Very helpful.

Gettext supports languages with more than 2 singular/plural forms in the .po file. It fires when given a numeric value, e.g., in a "%d" string format, and applying a categorization do that number based on a c-style rule description. Evidently some of our languages are best translated with that feature. With 3 forms: Spanish, French, Italian, Polish, Portuguese, Romanian, Russian. This is not high priority, but presumably our "Translate()" function could do likewise.

[Geep]

Hmm, it now appears that Format B would prevent me from using built-in AI translation with gettext-compliant desktop editors like poedit. Consider 2 more variants that make use of the msgctxt ("context") field:

Format C (A variant of Format A that can use AI directly)

#. Used after applying the holy water. Appears also in inventory:
msgctxt "#str_08610"
msgid "Holy Water Arrow"
msgstr "" // Will be filled in to make language-specific .po

Format D (A variant of Format B that still can not use AI directly, but could be manually script-swapped to Format C, more readily than B).

#. Used after applying the holy water. Appears also in inventory:
msgctxt "Holy Water Arrow"
msgid "#str_08610"
msgstr "" // Will be filled in to make language-specific .po

Swapping manually would be a pain, so I'm leaning towards C. Enterprise-level translation systems (cloud or desktop) have that kind of swapping (and other good stuff) built-in, but probably too pricey for us.

[Geep]

@stgatilov -

I might add, gettext-compliance requires that the msgstr field in the .pot file is always left empty. So editors will barf if you try to stick the English there.

[Geep]

OK, I'm generating .pot in Format C, and it's poedit-friendly.

One of the remaining issues is that all.lang has useful heading comments, e.g., indicating a category, that apply to a group of subsequent #str. But in the generated .pot file, they are currently associated as comment(s) attached to the first #str in a group, but visually buried.

I think I'll change all.lang to have additional "meta" strings, with ids of form:

#str_i18n_group_...
#str_i18n_note_...

For example, consider the all.lang category and note:

// STANDARD INVENTORY CATEGORY NAMES
// These are often checked for in scripts.

These might become:
 

"#str_i18n_group_std_inven_cat" "STANDARD INVENTORY CATEGORY NAMES"
"#str_i18n_note_std_inven_cat" "These are often checked for in scripts."

Or alternatively, if the second line is less important to see at a glance in the poedit UI, it could be left as a comment but moved within all.lang:
 

// These are often checked for in scripts.
"#str_i18n_group_std_inven_cat" "STANDARD INVENTORY CATEGORY NAMES"

Then in poedit when the #str item is selected, the comment would be seen in the sidebar (otherwise visually buried).

[Geep]

Or maybe I'll just keep it simple, as #str_i18n_, e.g.

"#str_i18n_std_inven_cat1" "STANDARD INVENTORY CATEGORY NAMES"
"#str_i18n_std_inven_cat2" "These are often checked for in scripts."

[Geep]

FYI. I'm generating english.po files now, but getting some spurious "duplicate entry" errors in poedit. Still diagnosing, but possibly related to line-length issues. 

[Geep]

Yeah, had to implement line-length chunking to make it gettext compatible. Headaches due to Windows vs Linux line endings too. Mostly fixed now.

[stgatilov]

On 9/22/2026 at 7:39 PM, Geep said:

OK, I'm generating .pot in Format C, and it's poedit-friendly.

This is still logically incorrect.
The string we want to replace should be in msgid, and it is #str_NNN.
Otherwise the placeholder and full-text replacements will have to work differently. Like "oh, does msgctxt magically look like a placeholder? if yes, then swap it with msgid".

Format D looks better in this sense.

On 9/22/2026 at 7:39 PM, Geep said:

One of the remaining issues is that all.lang has useful heading comments, e.g., indicating a category, that apply to a group of subsequent #str. But in the generated .pot file, they are currently associated as comment(s) attached to the first #str in a group, but visually buried.

You should retain them as comments.
Filling the file with valid strings which must be functionally ignored is not a good idea, it will bite us back for sure.

[Geep]

8 hours ago, stgatilov said:

This is still logically incorrect.

I don't disagree, but there are important practical reasons why what I propose (Format C) is better, which I gather (i.e., AI sez) is often the po/pot solution used by translation projects with placeholder IDs.

Here's what Format C looks like in poedit. This began with my freshly-generated .pot from revised all.lang[English]. That was then opened in poedit, and a "French" translation (fr.po) started (with empty translations; eventually I'll prepopulate *.po with existing translations from all.lang).

Observe in the main panel, on the "Source text -- English" left side, each line shows the #str_id (in blue, from msgctxt) and then the English text (from msgid). Furthermore, there is a translation suggestion into French on the right. Such suggestions would not be provided if the English text was not in msgid.  

[stgatilov]

I wonder where poedit shows:

1. msgctxt
2. comments preceding the entry

If it does not display them, then maybe it's time to search for another editor.
Although poedit does look like a good option to me in general...

Well, some AI (Gemini?) claims the following:

Quote

Poedit: The most popular cross-platform desktop editor for PO files. It clearly displays comments, source code references, and context strings in a dedicated sidebar panel next to the translation fields.

[Geep]

1 hour ago, stgatilov said:

I wonder where poedit shows:

1. msgctxt
2. comments preceding the entry

As I mentioned, msgctxt is shown in blue (here with our #str_id) followed by "|" and msgid.

The item I selected in my screenshot didn't have preceding comments. Here's an example that does:

As you can see, this not so helpful when there's a bunch of preceding comments that - like here - are really group comments. For those, I proposed #str_i18n..., which would be helpfully visible as headings/notes in the main list. I agree that hack is not ideal.

How about we invent a different indicator, like #note_...  (otherwise same syntax treatment as #str_..., but ignorable by engine)?

[stgatilov]

Quote

How about we invent a different indicator, like #note_...  (otherwise same syntax treatment as #str_..., but ignorable by engine)?

I suppose it would be #comment_...

Okay, I suppose I can swap msgctxt if it looks like #str_xyz during load.

[Geep]

4 hours ago, stgatilov said:

I suppose it would be #comment_...

It occurred to me that (unlike msgid) msgctxt doesn't have to be unique (I think). In that case, we could use a tag without a unique suffix. For instance, #comment or #note. Or even better, just "-" or "". The "" choice could be implemented by not including msgctxt at all in the block.

In all.lang, we'd have to have some markup to differentiate comments into (at least) two classes:

• associated with particular #str_id
• or not; group, heading, or more general info.

so that my program can do the proper conversion. I'm thinking something like "//-" or "///" for the second case, with just "// " for the first case.

[stgatilov]

35 minutes ago, Geep said:

In all.lang, we'd have to have some markup to differentiate comments into (at least) two classes:

• associated with particular #str_id
• or not; group, heading, or more general info.

so that my program can do the proper conversion. I'm thinking something like "//-" or "///" for the second case, with just "// " for the first case.

Didn't you want to make section comments using fake #comment entries?

You can of course make all kinds of conventions for comment formats, but I am skeptical they will be strictly followed after by other people...

[Geep]

8 minutes ago, stgatilov said:

Didn't you want to make section comments using fake #comment entries?

Hmmm, just thinking to use empty or missing msgctxt to achieve same goal. I'll experiment a bit.

At moment, seeing if I can just keep long lines (instead of chucking), and use settings in poedit to accept them. I've added a command-line option switch for this behavior.

[Geep]

Yes, long-lines works OK with poedit (when wrapping turned off in Settings/Advanced).

Thinking about the group comments further, it's probably not a good idea to use empty msgctxt as a flag for this, because it may conflate with future FM-specific po usage.

Instead, what I've come up with using "//|" as a group comment flag in all.lang, which my program will then parse as putting "//" into pot's msgctxt and the rest of comment into msgid. Because poedit uses "|" to visually separate its display of msgctxt and msgid, this has the effect of making the all.lang entry and poedit entry visually similar. And translators familiar with C-language will see "//" as a comment.

For example (with just a few lines in all.lang edited for group comments):

BTW, I'm thinking to move away from using the "====" to indicate main headers, and just using prefixes like "1)", "1.1)", "1.1.1)".

 

[Geep]

I've been experimenting with different markups (and corresponding conversion code), trying to balance readability in all.lang and in poedit. I have one now that seems good enough, so I'm applying it throughout all.lang[English]. Details later.

[stgatilov]

On 10/2/2026 at 9:00 PM, Geep said:

I've been experimenting with different markups (and corresponding conversion code), trying to balance readability in all.lang and in poedit. I have one now that seems good enough, so I'm applying it throughout all.lang[English]. Details later.

If you have already converted to gettext format, can you post some example of PO file so that I could implement support in the engine?

[Geep]

2 hours ago, stgatilov said:

If you have already converted to gettext format, can you post some example of PO file so that I could implement support in the engine?

I'm finishing up a round of all[English].lang markup, and should have that, and the generated .pot and english.po files, for you later this week.These will be drafts, but in pretty good shape.

Other generated language .po files will be after that. These will follow the .pot template, but add translations from all.lang. For that, more coding will be needed, but shouldn't be too bad, given my recycled infrastructure.