English Subtitles for AI Barks

[Geep]

A project is underway, led by Geep, to eventually provide English language subtitles for all the non-story phrases (e.g., barks) of the stock AI characters. The non-story phrases are identified by the "verbosity speech" subtitle tag. The hope is that these subtitles would be distributed as part of the future TDM 2.12 core and so available to all FMs, both old and new. Since there are 5-10K such phrases in total, this project will take a while. Let me know if you'd like to help.

This forum thread can be used for project coordination, and to provide info about -

• What subtitle sets (e.g., for particular AI) are underway and completed
• Emerging tools or methods for bulk testing
• Style guidance for TDM subtitle authors. To support this project, a style guide will be developed as experience accumulates. While motivated by providing some consistency to English speech-tagged subtitles, it may be of some interest to FMers creating story-tagged subtitles, and those subtitling in other TDM-supported languages.

Potential improvements to TDM's subtitling system - particularly those that would involve engine/GUI changes - can be discussed in this companion thread: Subtitles - Possibilities Beyond 2.11

Current Status - Subtitles for Barks

The Thug - NEW UPDATE May 6, 2023 released as testSubtitlesThug2 FM. This takes advantage of new 2.12dev features and changes to style guidelines.

The Lord - NEW UPDATE May 4, 2023 released as testSubtitlesLord2 FM

The Wench - Soon! To be released as testSubtitlesWench FM; pending resolution of rename of a few sound files (Bugtracker 6284).

The Young Builder (Builder 4) - NEW UPDATE! May 30, 2023 released (again) as testSubtitlesYoungBuilder FM. (Corrects error in briefing syntax of original May 17 release.)

Average Jack - coming soon

TDM vocal sets are listed here.

Current Status - Utilities

buildSubtitleShader.exe - Latest release of April 10, 2023. This program fabricates a TDM sound shader file specifically for the testSubtitles... series of FMs used here. It does so by wrapping each sound file name in a directory into a sound shader with uniform incremental naming. While limited to a single directory, the latest release simplifies merging runs from multiple directories. @datiswous reports that this Windows console program also works under Linux/wine, where it is used with similar "testSubtitle..."-derived FMs for verbosity "story" subtitling.

checkDurationsInSRT - NEWLY UPDATED (BUG FIX)! May 6, 2023 Release
This Win/console program scans a directory for .srt files, examines the subtitles, and warns about those phrases/messages that are potentially too short or too long in time, or that seem to require too high a reading rate, expressed in characters per second. It also looks for within-file subtitle messages that overlap in time.

soundDurationsCSV.exe - March 7, 2023 Release
This Windows console program scans a directory of sound files, reporting their names and (using pre-installed ffprobe) durations with millisecond resolution. The resulting .csv file can then be imported into a custom Excel spreadsheet for subtitle editing. For more, see also Feb 20th comments .

findTooLongSubtitles.exe - March 21, 2023 Release
This Win/console program scans a directory for .subs and .srt files, checks the length in characters of each subtitle line, and reports those that exceed a specified maximum.

analyzeFieldWidth of April 27 (release not scheduled)

Used to determine, for 12pt Carleton font, how many characters might reasonably fit within a given gui field width.

Current Status - Spreadsheets

Work by Geep on each vocal set starts with instantiation of a pre-configured Excel spreadsheet. This is used for subtitle editing and to do automatic calculations to help manage constraints and decision making. The columns and calculations are still evolving, as are related templates, how-to documents, and workflow documents. It is hoped, when evolution solidifies, to release a stable version of this material. Interim spreadsheets are being released as snapshots with corresponding subtitle releases.

Current Status - Style Guide for Barks

Work in progress, as major Word doc. Expect summer release.

Current Status - Third-Party Tools

For editing of "story" SRT files, datsiwous found "kdenlive" particularly helpful [to do: link to how-to below]. Geep has been working with the simpler, audio-only "Cadet" for barks.

Edited May 30 by Geep
Update of YoungBuilder FM

[Geep]

I'm starting with The Thug's barks (appx 400), for which maybe half of the needed subtitles were available on the Thug Vocals wiki page. (These pages are no longer maintained and so mostly obsolete.) I've got a test map suitable for testing a single subtitle, but it doesn't scale for bulk testing. Also looking at existing caption/subtitle style guides.

[thebigh]

Is there an easy way to handle situations where multiple guards are barking at once?

[datiswous]

The subtitle system afaik has one graphical look for all subs, so it's hard to see the difference except if you specify it.

Quote

Guard1: If there's trouble, I'm ready

Guard2: Me too!

It would be nice if you could set colors for different persons.

Edited January 27 by datiswous

[Geep]

Multiple guards (up to 3) should have their speech appear in separate fields.

Within the limitations of 2.10/2.11, my draft guidance for generic barks ("verbosity speech") is never to include AI name info. However, for an FM-specific phrase ("verbosity story") just said by one character, you could include the name or other ID. (I'm still researching if there is a consensus on best-practice format to indicate the name).

Beyond 2.11, color is one of many way to go. Reminds me, I was going to set up another thread for that... done. See link to Subtitles - Beyond 2.11 in OP above.

Edited January 27 by Geep
Link to Beyond 2.11

[datiswous]

Apparently srt has some basic formatting options, but they're not used by tdm (tested), which is a shame. Especially font color and line position would be useful I think.

[stgatilov]

I thought about naming sounds by who said them but did not got any particular scheme.
One problem here is that subtitles are supported inside the sound engine, where sounds no longer have any source: you just have a sound and probably location, but not something like who said it.

I made sure that a subtitle for a sound does not move between slots.
Other than that, there are 3 slots for simultaneous subtitles, and I guess some prioritization. But if there are many guards talking at once, it would be chaos.

As for colors: it should be possible to generate color automatically but it won't help anything.
It should be possible to specify colors inside subtitle files, similar to the way verbosity is now specified.

To be honest, I'm not sure it's a good idea to give names, since player usually does not know who speaks.
But that's of course mapper's preference: sometimes player surely knows who is talking in a cutscene, so why not.

[Geep]

@datiswous, I imagine the TDM fonts don't support bold, italics, underline, etc. Maybe just different sizes.

I don't think color by itself would be helpful for speaker identification, unless there was a color halo or name tag above each vocalizing AI. Might be useful for word emphasis, though.

@stgatilov, if there is location data available to the subtitle code, that could be used in various ways, using a different GUI:

1) Let the 3 slots be either left justified, centered, or right justified, depending on relative speaker location (including off screen). This could be implemented by 9 actual windowDefs (all of same size; 3 each for each current slot, overlaid, with each of the 3 justifications.) Alternatively, with a 3x3 grid, all windowDefs the same size, but 1/3 the current width.)

2) Or instead, at the top edge of each slot, show a short horizontal bar, whose left/right position is moved to be under the relevant AI. If off screen, add an arrow head (<-- or -->).

Naming is harder, if the subtitle code can't get at that info. Though at the point the sound engine is passed the sound to render, presumably it knows the speaker, and could independently and in parallel visualize the name information (particularly if the sound engine passed back which GUI slot it was going to use for subtitle.)

It is true that associating a name (say, using a small tab-like field) with a slot is less useful when the player doesn't know the AI names (i.e., with no floating names above the characters).

If the technical issues of naming could be overcome, then there's still the question of when you'd want to give a name (Rupert) versus type (thug #3) versus generic (speaker #2). Also have to handle special cases of (narrator) and (player).

Dreaming... Instead of text names, more fun would be to show a thumbnail face of each AI next to their slot. With a question-mark face when they are off-screen?

Edited January 29 by Geep

[datiswous]

Do you have the text for all the speech? For example do you know if it's horse piss or whore's piss ? 

[Geep]

The wiki has AI vocal scripts for most but not all characters. I'm guessing the wiki info is derived from early-draft vocal scripts supplied to the voice artists. Looking at the Thug's script, I see "I'd drink horse piss [etc]"

There are occasional differences between an .ogg file and it's draft text phrase. Some of that is just the voice artist riffing. Other cases seem to be a more intentional change of plan. For instance, with respect to swearing, "shit" is generally suppressed... "horse shit" became "horse dirt" or "horse filth". "Bloody" became more widely used, probably to make the English more British, less American.

I'm about 2/3rds through the Thug translation. Going slow because I'm also working up related tools and style guides. And I don't plan to upload subtitle sets for any AI until after 2.11 is out, presumably Real Soon Now.

[Geep]

Here's a screen shot of an FM I've been building to help test/review subtitles for an AI, in reproducible order with 100% coverage. It uses the TDM-distributed .ogg files, but has custom soundshaders. Each such soundshader wraps exactly one .ogg file, and has a uniform naming that includes an index number. The collection of soundshaders is housed in a single file. The hope is that that (prior to embedment in the FM) this file can be easily generated from a directory listing of the .ogg files for a particular AI, and subsequent manipulation.

There are 3 buttons to step through the file. After each button press, you hear the vocal and see the subtitle. Also, briefly the index number within the list appears (a floating "7" near the statue's shoulder in this screenshot.) You can also - using a custom CVar in the console - jump to a particular index number.

Unsurprisingly, you can't edit the subtitle within the FM. Just note what needs changing and do it with a text editor. Also, this FM is not intended to evaluate TDM's stock soundshaders, nor AI lip syncing (so just a statue speaking here.)

If you hit the buttons fast enough, multiple sound files play at once, and multiple subtitle fields appear (up to the max of 3 the .gui offers)

[datiswous]

1 hour ago, Geep said:

Unsurprisingly, you can't edit the subtitle within the FM. Just note what needs changing and do it with a text editor.

Ehhh, cool idea, but why not do everything with a video editor and then you can immediately edit the subtitle in the editor?

[Geep]

2 hours ago, datiswous said:

Ehhh, cool idea, but why not do everything with a video editor and then you can immediately edit the subtitle in the editor?

For the Thug, all the subtitles I've done so far have been rather short and so have lent themselves to the simple "inline" approach. I agree that a different character that has more long monologues, and so needs "srt", would benefit from a video editor (even if there's no real video, just audio) as you described earlier. I may need to call on your srt expertise later.

I've been creating the subtitles with 3 windows open -

1- to select and play each .ogg with a minimal Windows player.

2- A view of the AI vocal script (e.g., source view of the Thug wiki entry), to use as a copy/paste starting point.

3 - The .subs file being built in a text editor.

So, the FM I described is not used in the first place to create subtitles, but rather to do a second-pass review (e.g. by me) or third-pass review (as part of quality control during incorporation into TDM).

 

[datiswous]

Ah I forgot they're inline off course. I've been thinking of puting all the ogg files in one line, generating the subs and then converting them to inline, but I didn't know about the ai vocal scripts on the wiki.

[Geep]

I have a question about the current TDM implementation of "srt"...

For an audio clip using srt with multiple subtitle phrases, the start and end timestamps for each phrases are relative to the clip start. For implementation reasons, it would not be reasonable to have negative time. But it could be both reasonable and useful to have a subtitle that continues on-screen a bit longer than the audio clip. Is that currently possible?

[datiswous]

You mean only the last phrase? So for example you have an srt subtitle file with 20 phrases and the last phrase will be shown a couple of seconds longer than the audio file plays?

Or do you mean for every audio phrase in an audio file? You can do whatever you want. You can set the time for every subtitle phrase how long it will display during playback.

Edited February 6 by datiswous

[stgatilov]

2 hours ago, Geep said:

For an audio clip using srt with multiple subtitle phrases, the start and end timestamps for each phrases are relative to the clip start. For implementation reasons, it would not be reasonable to have negative time. But it could be both reasonable and useful to have a subtitle that continues on-screen a bit longer than the audio clip. Is that currently possible?

As far as I remember, it is not possible.
If sound ends, it is no longer in the sound system, so it does not generate subtitle anymore.

On 2/3/2023 at 10:45 PM, Geep said:

Unsurprisingly, you can't edit the subtitle within the FM. Just note what needs changing and do it with a text editor.

I think some command can pull the subtitle changes from disk.
It's either reloadDecls or reloadSounds.

[Geep]

@datiswous, just the last phrase.

@stgatilov, confirmed that subtitle display stops when sound ends.

If I edit contents of an .srt file, neither reloadDecls nor reloadSounds sees that change.

Edited February 7 by Geep

[stgatilov]

1 hour ago, Geep said:

If I edit contents of an .srt file, neither reloadDecls nor reloadSounds sees that change.

Perhaps you can add a space to .subs file and do reloadDecls?

[Geep]

1 hour ago, stgatilov said:

Perhaps you can add a space to .subs file and do reloadDecls?

Maybe. I'll keep it in mind to try, if I get a srt workflow going at some point.

[Geep]

The first-draft of the thug subtitles is done, and also successfully loaded into my test jig with custom soundshaders for the second-draft review/revise. That review will be paused while i work through some issues about best subtitling practices to target. For example, word per minute reading constraints and their impacts.

[Geep]

@stgatilov, in the Subtitles_decls section of the Subtitles wiki page, you show example code, that includes an srt reference.

Wouldn't it be better to guide people to place the corresponding .srt files within the "subtitles" tree (where their .subs live), rather than the "sound" tree?

Also, should FM authors be encouraged (if not required) to prefix .srt files with "fm_" ?  Like "fm_sound8_long.srt" ?

Finally, in the "displayed text" section, under the srt command, there's a desperate need to see example .srt content, in this case, something made up for sound8_long.srt (or fm_sound8_long.srt) like:

1
00:00:06,612 --> 00:00:10,376
Something's wrong with this crystal ball.

2
00:00:15,482 --> 00:00:20,609
Bugger me!
It's not showing the right dream.

3
00:00:25,336 --> 00:00:28,167
Ah! Here we go.

--end

[datiswous]

1 hour ago, Geep said:

Wouldn't it be better to guide people to place the corresponding .srt files within the "subtitles" tree (where their .subs live), rather than the "sound" tree?

Yeah, I thought the same and this is what I effectively do (putting srt files in folder subtitles).

[stgatilov]

22 hours ago, Geep said:

Wouldn't it be better to guide people to place the corresponding .srt files within the "subtitles" tree (where their .subs live), rather than the "sound" tree?

Technically, it does not matter.

It might be convenient to have .srt files at same location and with same name as the sound files that they correspond to.
I think that was the reason to suggest placing .srt into sounds.

Or maybe I did not think to much about it

Quote

Also, should FM authors be encouraged (if not required) to prefix .srt files with "fm_" ?  Like "fm_sound8_long.srt" ?

Are you encouraged or required to prefix your sound samples with "fm_"?

I know many mappers do use some kind of prefix, and that's good.
Others don't and that can cause accidental name collisions.
Nothing new here.

Quote

Finally, in the "displayed text" section, under the srt command, there's a desperate need to see example .srt content

There is a link to .srt description. Also, .srt is kinda standard format.

[datiswous]

Btw. there's some confusing text in that article:

Quote

FM-specific subtitles are always included from the decl named fm_root, which must be in the fm_root.subs file. For instance, the contents of the fm_test.subs file from the example above will have no effect until we add the file fm_root.subs with contents:

https://wiki.thedarkmod.com/index.php?title=Subtitles#include

There's no fm_test.subs in an example. I think it should be fm_conversations.subs . Not sure, but maybe you could check?

Edited February 9 by datiswous