Subtitles - Possibilities Beyond 2.11

[Geep]

Also, I was interested in getting metadata about the TDM fonts. Some forum posts suggested you just rename a font's .dat file to .ttf, and then font tools (like those provided by FreeType) could read it. But all I'm seeing is an unrecognized file format when a read is attempted. ???

[datiswous]

12 hours ago, stgatilov said:

Existing messages would fit the new limits (they would be a bit larger), hopefully without exceptions.

Do you mean more text space per message block or larger size of text? If there's more useble text per block available, that's nice I guess as long as you don't have to use it, but can use it.

[madtaffer]

Be able to override fonts of readables, books in the main menu or darkmod.cfg.

[Geep]

On 4/5/2023 at 10:37 AM, madtaffer said:

Be able to override fonts of readables, books in the main menu or darkmod.cfg.

Books are out of scope for this thread, which is just concerned with subtitles. As an aside, because of the way book text is laid out (separate independent overlays for "title" and "body" text), changing the font from what the FM author provided would tend to make a mess.

[Geep]

@stgatilov, I started to apply the inline "-x" option for extra time to a vocal set, but then realized that I needed to know better how it worked, in order to choose appropriate values.

So some questions:

1) Does the inline -x work...
   a) the same for "story" and "speech" verbosities
   b) differently

If (b), the next questions will need to be answered twice. I'm going to state these in mathematical terms, summarizing the behavior of the algorithm and reflecting the order in which values are combined. All variables are in seconds.

c = length of a given sound clip
x = inline -x value for extra time. Should be positive. Zero if not given.
d = calculated length of subtitle duration
L = global giving a lower bound on subtitle duration. Default is 1.0
X = global giving an extension to subtitle duration. Default is 0.2
U = global giving an upper bound on X. Default is 5.0

2) When there is no value of x, which equation describes the algorithm's behavior?

   a) d = max(c,L) + min(X,U)
   b) d = max(c+min(X,U),L)

3) When there is a valid value of x, the equation becomes...
   a.1) d = max(c,L) + min(X,U) + x
   a.2) d = max(c+x,L) + min(X,U)
   a.3) d = max(c,L) + min(X+x,U)
   b.1) d = max(c+min(X,U),L) + x
   b.2) d = max(c+x+min(X,U),L)
   b.3) d = max(c+min(X+x,U),L)

Thanks for help with this.

[Geep]

Actually, there are additional possibilities where X and x are not additive, but alternatives:

a.4) d = max(c,L) + min(X,x,U)

a.5) d = max(c,L) + min(max(X,x),U)

a.6) d = max(c,L) + min(x,U) // iff x > 0. In that case, x overrides X

b.4) d = max(c + min(X,x,U), L)

b.5) d = max(c + min(max(X,x),U), L)

b.6) d = max(c + min(x,U), L) // iff x > 0. In that case, x overrides X

[stgatilov]

The extend parameter for inline subtitles is not implemented yet.

Verbosity does not matter.

Inline subtitles will be displayed for min(max(c + x, 1), c + 5).
Here x = 0.2 by default, or whatever value is set in the extend parameter.
Of course all these constants are controlled by cvars, I used their default values for simplicity.

[stgatilov]

I still don't like the idea of -x parameter.

That would make subtitle decl files linebreak-dependent, i.e. its meaning would change from adding or removing a linebreak. Can you recall any other decl file type that behaves like this?...

UPDATE: Although it we forbid commands starting with minus, it can be parsed unambiguously regardless of linebreaks. It's not that -x style is widespread in decl files, but perhaps we can live with it...

[Geep]

5 hours ago, stgatilov said:

Inline subtitles will be displayed for min(max(c + x, 1), c + 5).
Here x = 0.2 by default, or whatever value is set in the extend parameter.

OK, I would express this as:

d = min(max(c + X, L), c + U) if x not defined
d = min(max(c + x, L), c + U) if x defined (where x > 0)

In this formulation, if the user chooses to set the maximum extension U to zero, then the values of x, X (default 0.2), and L (default 1.0) have no effect.

5 hours ago, stgatilov said:

The extend parameter for inline subtitles is not implemented yet.

Soon I hope. I can't finalize any more vocal sets barks until then.

5 hours ago, stgatilov said:

I still don't like the idea of -x parameter.

That would make subtitle decl files linebreak-dependent,

I'm not clear how parsing is dependent on line breaks here. I mean, this would be OK:
 

inline "say1.ogg" "Hello, world"
  -x 0.3
inline "say2.ogg" "Goodbye"

That said, I'm not wedded to "-x", but I would very much like to have the extension value either part or adjacent to the inline statement it modifies. Ideas:

inline "say1.ogg" "Hello, world" "" // last parameter is no longer optional, but if "", then treated as if not given

or

inline "say1.ogg" "Hello, world"
inline-extend 0.3 // or maybe inline-extend "say1.ogg" 0.3

 

[stgatilov]

52 minutes ago, Geep said:

In this formulation, if the user chooses to set the maximum extension U to zero, then the values of x, X (default 0.2), and L (default 1.0) have no effect.

The subtitle cvars are not for user configuration, they are internal.
They would change only if if TDM developer decides that... if player changes that, it would be his problem.
Just like very many cvars which are totally not for the user to change.

Quote

Soon I hope. I can't finalize any more vocal sets barks until then.

I just committed it in svn rev 10358, so expect it to arrive... I guess I'll publish dev build this weekend.

Quote

I'm not clear how parsing is dependent on line breaks here.

I decided that options always start with minus, while commands always start with letter.
That's how parsing can work in well-defined way, regardless of linebreaks.

The final syntax is:

• -durationExtend 0.5
• -dx 0.4

Also, for now it only work on inline comments (warning and subtitle removal when applied to SRT).

[Geep]

45 minutes ago, stgatilov said:

The subtitle cvars are not for user configuration, they are internal.

Good to know!

46 minutes ago, stgatilov said:

I just committed it in svn rev 10358, so expect it to arrive... I guess I'll publish dev build this weekend.

47 minutes ago, stgatilov said:

Also, for now it only work on inline comments (warning and subtitle removal when applied to SRT).

Since SRT now has it's own way of extending (using end time of last phrase), there is probably not a need for -dx with srt.

Hmm, I wonder what the calculation of d is for SRT. Probably could use this variable:

e = end time of last phrase within srt

Then maybe redefine variable x as:

x = e - c

In this case, x could be negative or positive. Then tie it in to the 3 internal globals. Maybe differs from inline.

[stgatilov]

6 minutes ago, Geep said:

Hmm, I wonder what the calculation of d is for SRT.

The duration for SRT is min(LastMessageEndTime, SoundDuration + tdm_subtitles_durationExtensionLimit)

[Geep]

Now that I think about it, "duration" needs to be expressed in terms of each srt phrase. So the equation you gave is good for a single-phrase srt that starts at time 0. More generally:

start = phrase start time (within clip)

stop = phrase stop time

diff = stop - start

In most cases:

d = diff

For the final phrase, there is this constraint:

stop = min(LastMessageEndTime, SoundDuration + tdm_subtitles_durationExtensionLimit)

But are these globals also applicable to SRT?

• the 1.0 second lower bound (I was calling L)
• the 0.2 extension (I was calling X)

BTW, how are overlapping time ranges within an SRT handled?

 

[stgatilov]

14 hours ago, Geep said:

Now that I think about it, "duration" needs to be expressed in terms of each srt phrase. So the equation you gave is good for a single-phrase srt that starts at time 0

I don't understand what you are talking about.
From the point of engine, there is one long sound, and many messages in SRT subtitle.
The engine cannot distinguish between messages inside a single sound file.

SRT file fully controls how long its subtitles are displayed for.
The tdm_subtitles_durationExtensionLimit is merely a sanity limit, to make sure subtitles don't go wild and behave independently.

Quote

But are these globals also applicable to SRT?

• the 1.0 second lower bound (I was calling L)
• the 0.2 extension (I was calling X)

No.

Quote

BTW, how are overlapping time ranges within an SRT handled?

I suppose all overlapping messages are displayed at once.
Duration is computed from the maximum endtime of all messages.

[Geep]

6 hours ago, stgatilov said:

don't understand what you are talking about.

I was just trying to distinguish between individual message (aka phrase) appearance time on-screen, and overall srt timeline maximum. You're thinking "duration" should to refer to only the latter, but I was momentarily using it to refer to the former.

6 hours ago, stgatilov said:

Duration is computed from the maximum endtime of all messages.

Probably it would be better with srt not to refer to "Duration" by itself, but instead:

• SoundDuration
• MessageDuration
• MaxMessageEndtime

"Duration" by itself is misleading, because it seems to imply something about how long messages appear on screen; that's not the case because the first-starting temporal message in an srt doesn't have to start at time zero. (And the summation of individual MessageDurations is even less informative, since there can be time gaps between messages.)

6 hours ago, stgatilov said:

I suppose all overlapping messages are displayed at once.

That's one way of reporting what for TDM is an srt error.

OK, I understand what the expected code behavior is now. Thanks.

[datiswous]

Would it be possible to use the subtitle system (or something similar) to display a translation for text you frob, and active gui pages (for example a page in a book)? (I guess this might give problems for pages with a lot of text)

This way you can keep the game in it's original look, while still giving a translation.

Edited June 28, 2023 by datiswous

[Geep]

@datiswous, I suspect this is possible if rather tedious for signs or 1-page broadsheets. You could arrange that the frob calls a script, and the script causes the player/narrator to say a custom .ogg file. That file would be just silence, of the same length as the desired subtitle duration for the translated text (or at least within 5 seconds of it, then using the duration extension methods). Alternatively, the narrator could actually "read aloud" the text.

The complexity of the .gui for books is rather daunting, but maybe you could tie into page-turning events.

I don't think you'd want to recommend this approach for general book translation throughout an FM.  If for no other reason, making the .ogg files would be a pain. Also, it doesn't seem to me to extend to more than one subtitle language.

I could imagine that, with engine changes, books could be made into first-class speakers (i.e., as if AIs), so you could optionally hear as well as read a book, and in any event see the subtitle in any language of the player's choice for which a translation is available.

Edited June 28, 2023 by Geep

[datiswous]

Maybe you can autogenerate text to speech and based on that display the subtitles. 

[Geep]

If this was a high-budget game, I could imagine someone repeatedly training up a text-to-speech LLM on an existing TDM AI voice. Not sure that's practical for no-budget TDM.

[Geep]

To the "Subtitles" wiki page, I just added a new section, mostly drawn from this forum -

"displaying_text_with_extended_duration_(new_for_2.12)"

[Geep]

@stgatilov, would it be correct to say that the latest public 2.12dev contains the original 4:3 fonts, and does not distribute the narrowed 16:9 fonts (e.g., for Carleton and possibly Stone)?

Also, I'd greatly appreciate it if you could expose a numeric variable whose value is set to indicate whether the current subtitle is of type "verbosity story", "verbosity speech", or "verbosity effect", that can be read (but not set) in tdm_subtitles_common.gui

[stgatilov]

17 hours ago, Geep said:

@stgatilov, would it be correct to say that the latest public 2.12dev contains the original 4:3 fonts, and does not distribute the narrowed 16:9 fonts (e.g., for Carleton and possibly Stone)?

It is available in dev builds as engine feature (6283), but it is not used in any GUI scripts.

I was hoping that someone would volunteer to propose, tweak, and test the change of font size and rectangle of subtitles. Perhaps I should just do rough approximation myself and commit changes

I feel that feature of showing location is pretty desolated too.
Indeed, I implemented it and it works, but it fells horrible. People are asking what the hell this moving white bars are, which totally confirms the idea that current design does not work.

Quote

Also, I'd greatly appreciate it if you could expose a numeric variable whose value is set to indicate whether the current subtitle is of type "verbosity story", "verbosity speech", or "verbosity effect", that can be read (but not set) in tdm_subtitles_common.gui

Yes, this is indeed possible, not really hard to do in C++.
To you want to color the text?

[Geep]

Quote

It is available in dev builds as engine feature (6283), but it is not used in any GUI scripts.

@stgatilov, I earlier read your 6283 description about 'So e.g. we can support fonts like "fonts/carleton@aspect=16:9" ', but I thought that meant there was a pre-created font variant with filename "carelton@aspect=16:9" to be distributed. But maybe "@aspect=16:9" is a gui keyword to trigger on-the-fly compression? [Duh, I guess I could just try that directly, but not at moment.]

Quote

I feel that feature of showing location is pretty desolated too.
Indeed, I implemented it and it works, but it fells horrible. People are asking what the hell this moving white bars are, which totally confirms the idea that current design does not work.

Yeah, I agree it's more confusing than helpful, and probably should be dropped. I'm sorry I suggested it (at least the idea of indicating location by a perimeter light; maybe not indicating volume too). I don't have a better idea at the moment.

Quote

Yes, this is indeed possible, not really hard to do in C++.
To you want to color the text?

It would be interesting to play around with visually distinguishing "story" from "speech" subtitles. Lots of ways to do that, but having the enumeration available is the prerequisite.

[Geep]

@stgatilov, both the compressed Carleton and Stone fonts look better than the original uncompressed IMHO. We could give the user a choice of either, while keeping the gui scale factor fixed at 0.25 (e.g., 12 pt). If Stone's artifacting you mentioned was not too problematic.

I did a statistical analysis of those two fonts, and Stone strings seemed to be slightly less wide than Carleton by about 7% (however, Stone 24pt requires scaling down to get to 12pt, and the analysis may not be treating the resulting "half-pixel" widths right.)

But if true, background field widths based on Carleton should still be excellent for Stone.

Another topic. I'd like to play around a bit with the 5 sound-source-location variables you were using to position the white bars. In file tdm_subtitles_message.gui, these were of form:

"gui::subtitle0_oriBackLeft"   // where 0 if subtitle was shown in vertical "slot 0"; could be 1 or 2

and name suffixes were _oriFront, _oriFrontLeft, _oriFrontRight, _oriBackLeft, and _oriBackRight

Could you briefly explain what values these could take on, how they would work or differ, how often they get updated? Thanks

Edited September 3, 2023 by Geep
typo: *uncompressed*

[Geep]

FYI. Just redid the stat analysis, assuming that the engine, after dividing Stone 24pt font width by 2 to get 12pt, then rounds up. In that case, the Stone string is still less wide than Carleton, but now only 4-5% less, instead of 7-8%.