English Subtitles for AI Barks

[Geep]

Introduction

A project is underway, led by Geep, to eventually provide English language subtitles (or if you prefer, "closed captions") for all the non-story phrases (e.g., barks) of the stock AI characters. The non-story phrases are identified by the "verbosity speech" subtitle tag. The hope is that these subtitles would be distributed as part of the future TDM 2.12 core and so available to all FMs, both old and new. Since there are 5-10K such phrases in total, this project will take a while. Let me know if you'd like to help.

This forum thread can be used for project coordination, and to provide info about -

• What subtitle sets (e.g., for particular AI) are underway and completed
• Emerging tools or methods for bulk testing
• Style guidance for TDM subtitle authors. To support this project, a style guide will be developed as experience accumulates. While motivated by providing some consistency to English speech-tagged subtitles, it may be of some interest to FMers creating story-tagged subtitles, and those subtitling in other TDM-supported languages.

Potential improvements to TDM's subtitling system - particularly those that would involve engine/GUI changes - can be discussed in this companion thread: Subtitles - Possibilities Beyond 2.11

Current Status - Subtitles for Barks

All TDM vocal sets planned for TDM 2.12 have been delivered.

Final Touches: Fixup Needed to Run These Under 2.12 Beta 3

The Thug - Update May 6, 2023 released as testSubtitlesThug2 FM. This takes advantage of new 2.12dev features and changes to style guidelines.

The Lord - Update May 4, 2023 released as testSubtitlesLord2 FM

The Wench - Two postponed April versions - Nov 25, 2023 released as testSubtitlesWenchOriginal and testSubtitlesWenchPatched FMs. Two versions because of Bugtracker 6284. IMPORTANT: See final version at end of this list, released Jan. 14, 2024.

The Young Builder (Builder 4) - Update May 30, 2023 released (again) as testSubtitlesYoungBuilder FM. (Corrects error in briefing syntax of original May 17 release.)

Average Jack - June 8, 2023 released as testSubtitlesJack FM.

The Pro - June 23, 2023 released as testSubtitlesPro FM.

The Maiden - July 8, 2023 released as testSubtitlesMaiden FM

The Grumbler - July 24, 2023 released as testSubtitlesGrumbler FM

The Mature Builder (Builder 3) - August 15, 2023, released as testSubtitlesMatureBuilder FM

The Lady (aka Noblewoman) - Sept 9, 2023, released as testSubtitlesLady FM

The Moor - Sept 14, 2023, released as testSubtitlesMoor FM

The Commander - Sept 30, 2023, released as testSubtitlesCommander FM

The Simpleton - Oct. 21, 2023, released as testSubtitlesSimpleton FM

Builder 1 & 2 (shared vocal set) - Nov. 8, 2023, released as testSubtitlesBuilder1and2 FM

The Critic - Nov. 25, 2023, released as testSubtitlesCritic_FM

Manbeast - Dec. 10, 2023, released as testSubtitlesManbeast FM

Final Touches: Lady02 - Dec 21, 2023, released as testSubtitlesLady02 FM. Assisted by datiswous.

Final Touches: The Drunk - Jan 6, 2024, released as testSubtitlesDrunk FM. Assisted by MirceaKitsune.

Final Touches: The Cynic - Jan. 13, 2024, released as testSubtitlesCynic FM. Incorporates early demo subtitles by Dragofer.

Final Touches: The Wench, Final Revision - Jan. 14, 2024, released as testSubtitlesWenchFinal FM . Assisted by Dragofer. See also original May version (released in November) in the list above.

All TDM vocal sets planned for TDM 2.12 have been delivered. Thanks, @nbohr1more, for integrating these into the release.

All AI vocal sets, including additional ones for future consideration - perhaps as "verbosity effects" - are listed here.

Current Status - Utilities

buildSubtitleShader.exe - Latest release of April 10, 2023. This program fabricates a TDM sound shader file specifically for the testSubtitles... series of FMs used here. It does so by wrapping each sound file name in a directory into a sound shader with uniform incremental naming. While limited to a single directory, the latest release simplifies merging runs from multiple directories. @datiswous reports that this Windows console program also works under Linux/wine, where it is used with similar "testSubtitle..."-derived FMs for verbosity "story" subtitling.

checkDurationsInSRT - May 6, 2023 Update (bug fix)
This Win/console program scans a directory for .srt files, examines the subtitles, and warns about those phrases/messages that are potentially too short or too long in time, or that seem to require too high a reading rate, expressed in characters per second. It also looks for within-file subtitle messages that overlap in time.

soundDurationsCSV.exe - March 7, 2023 Release
This Windows console program scans a directory of sound files, reporting their names and (using pre-installed ffprobe) durations with millisecond resolution. The resulting .csv file can then be imported into a custom Excel spreadsheet for subtitle editing. For more, see also Feb 20th comments .

findTooLongSubtitles.exe - Minor update, June 23, 2023 Release
This Win/console program scans a directory for .subs and .srt files, checks the length in characters of each subtitle line, and reports those that exceed a specified maximum.

analyzeFieldWidth of April 27, 2023 (release not scheduled; too idiosyncratic). Used to determine, for 12pt Carleton font, and a representative sampling of bark subtitles, how many characters might reasonably fit within a given gui field width. Later, parts of this code were used to build calcStringWidth.exe discussed next.

calcStringWidth.exe - Oct. 11, 2023 Release. Calculates the display width of an input string, for 4 candidate subtitle fonts Carleton and Stone, uncompressed or 16:9 compressed.

statsForSubtitles - June 25, 2023 Release.  Starting with The Pro vocal set, this Windows console program automates the gathering and reporting of the basic statistics included with the announcement of every vocal set release in this forum thread.

Current Status - Spreadsheets and Related Explanatory Documents, including Workflow and Templates

Work by Geep on each vocal set starts with instantiation of a pre-configured Excel spreadsheet. This is used for subtitle editing and to do automatic calculations to help manage constraints and decision making.

The announcement of each AI subtitle release here includes a link to the corresponding spreadsheet. At first, the spreadsheet's columns, formats, calculations, and highlighting (and its distillation into template form) would change quite a bit between each AI. As of June, 2023 and AverageJack, it has largely stabilized, and is now documented:

June 10 Documents Explaining Workflow and Excel Spreadsheet/Template (for AverageJack)

The v5 template was used for vocal sets from June's Average Jack to November's The Critic. Further clarifications are now available:

Dec 7 Update of the "Explained" Doc for Spreadsheet Template v5

Subsequent vocal sets, through to 2024's The Cynic, use template v6:

Final Touches: Jan. 10, 2024 "Explained" Doc for Spreadsheet Template v6

Current Status - Style Guide for Barks

As of Sept 2, Subtitle Style Guide - Part 1 is now available.

Final Touches: As of Jan 16, 2024: Subtitle Style Guide - Part 2 with more discussion and references.

Current Status - Third-Party Tools

For editing of "story" SRT files, datsiwous found "kdenlive" particularly helpful [to do: link to how-to below]. Geep has been working with the simpler, audio-only "Cadet" for barks.

Edited June 22, 2024 by Geep
Replace word "NEW!" with "Final Touches:"

[Geep]

I'm starting with The Thug's barks (appx 400), for which maybe half of the needed subtitles were available on the Thug Vocals wiki page. (These pages are no longer maintained and so mostly obsolete.) I've got a test map suitable for testing a single subtitle, but it doesn't scale for bulk testing. Also looking at existing caption/subtitle style guides.

[thebigh]

Is there an easy way to handle situations where multiple guards are barking at once?

[datiswous]

The subtitle system afaik has one graphical look for all subs, so it's hard to see the difference except if you specify it.

Quote

Guard1: If there's trouble, I'm ready

Guard2: Me too!

It would be nice if you could set colors for different persons.

Edited January 27, 2023 by datiswous

[Geep]

Multiple guards (up to 3) should have their speech appear in separate fields.

Within the limitations of 2.10/2.11, my draft guidance for generic barks ("verbosity speech") is never to include AI name info. However, for an FM-specific phrase ("verbosity story") just said by one character, you could include the name or other ID. (I'm still researching if there is a consensus on best-practice format to indicate the name).

Beyond 2.11, color is one of many way to go. Reminds me, I was going to set up another thread for that... done. See link to Subtitles - Beyond 2.11 in OP above.

Edited January 27, 2023 by Geep
Link to Beyond 2.11

[datiswous]

Apparently srt has some basic formatting options, but they're not used by tdm (tested), which is a shame. Especially font color and line position would be useful I think.

[stgatilov]

I thought about naming sounds by who said them but did not got any particular scheme.
One problem here is that subtitles are supported inside the sound engine, where sounds no longer have any source: you just have a sound and probably location, but not something like who said it.

I made sure that a subtitle for a sound does not move between slots.
Other than that, there are 3 slots for simultaneous subtitles, and I guess some prioritization. But if there are many guards talking at once, it would be chaos.

As for colors: it should be possible to generate color automatically but it won't help anything.
It should be possible to specify colors inside subtitle files, similar to the way verbosity is now specified.

To be honest, I'm not sure it's a good idea to give names, since player usually does not know who speaks.
But that's of course mapper's preference: sometimes player surely knows who is talking in a cutscene, so why not.

[Geep]

@datiswous, I imagine the TDM fonts don't support bold, italics, underline, etc. Maybe just different sizes.

I don't think color by itself would be helpful for speaker identification, unless there was a color halo or name tag above each vocalizing AI. Might be useful for word emphasis, though.

@stgatilov, if there is location data available to the subtitle code, that could be used in various ways, using a different GUI:

1) Let the 3 slots be either left justified, centered, or right justified, depending on relative speaker location (including off screen). This could be implemented by 9 actual windowDefs (all of same size; 3 each for each current slot, overlaid, with each of the 3 justifications.) Alternatively, with a 3x3 grid, all windowDefs the same size, but 1/3 the current width.)

2) Or instead, at the top edge of each slot, show a short horizontal bar, whose left/right position is moved to be under the relevant AI. If off screen, add an arrow head (<-- or -->).

Naming is harder, if the subtitle code can't get at that info. Though at the point the sound engine is passed the sound to render, presumably it knows the speaker, and could independently and in parallel visualize the name information (particularly if the sound engine passed back which GUI slot it was going to use for subtitle.)

It is true that associating a name (say, using a small tab-like field) with a slot is less useful when the player doesn't know the AI names (i.e., with no floating names above the characters).

If the technical issues of naming could be overcome, then there's still the question of when you'd want to give a name (Rupert) versus type (thug #3) versus generic (speaker #2). Also have to handle special cases of (narrator) and (player).

Dreaming... Instead of text names, more fun would be to show a thumbnail face of each AI next to their slot. With a question-mark face when they are off-screen?

Edited January 29, 2023 by Geep

[datiswous]

Do you have the text for all the speech? For example do you know if it's horse piss or whore's piss ? 

[Geep]

The wiki has AI vocal scripts for most but not all characters. I'm guessing the wiki info is derived from early-draft vocal scripts supplied to the voice artists. Looking at the Thug's script, I see "I'd drink horse piss [etc]"

There are occasional differences between an .ogg file and it's draft text phrase. Some of that is just the voice artist riffing. Other cases seem to be a more intentional change of plan. For instance, with respect to swearing, "shit" is generally suppressed... "horse shit" became "horse dirt" or "horse filth". "Bloody" became more widely used, probably to make the English more British, less American.

I'm about 2/3rds through the Thug translation. Going slow because I'm also working up related tools and style guides. And I don't plan to upload subtitle sets for any AI until after 2.11 is out, presumably Real Soon Now.

[Geep]

Here's a screen shot of an FM I've been building to help test/review subtitles for an AI, in reproducible order with 100% coverage. It uses the TDM-distributed .ogg files, but has custom soundshaders. Each such soundshader wraps exactly one .ogg file, and has a uniform naming that includes an index number. The collection of soundshaders is housed in a single file. The hope is that that (prior to embedment in the FM) this file can be easily generated from a directory listing of the .ogg files for a particular AI, and subsequent manipulation.

There are 3 buttons to step through the file. After each button press, you hear the vocal and see the subtitle. Also, briefly the index number within the list appears (a floating "7" near the statue's shoulder in this screenshot.) You can also - using a custom CVar in the console - jump to a particular index number.

Unsurprisingly, you can't edit the subtitle within the FM. Just note what needs changing and do it with a text editor. Also, this FM is not intended to evaluate TDM's stock soundshaders, nor AI lip syncing (so just a statue speaking here.)

If you hit the buttons fast enough, multiple sound files play at once, and multiple subtitle fields appear (up to the max of 3 the .gui offers)

[datiswous]

1 hour ago, Geep said:

Unsurprisingly, you can't edit the subtitle within the FM. Just note what needs changing and do it with a text editor.

Ehhh, cool idea, but why not do everything with a video editor and then you can immediately edit the subtitle in the editor?

[Geep]

2 hours ago, datiswous said:

Ehhh, cool idea, but why not do everything with a video editor and then you can immediately edit the subtitle in the editor?

For the Thug, all the subtitles I've done so far have been rather short and so have lent themselves to the simple "inline" approach. I agree that a different character that has more long monologues, and so needs "srt", would benefit from a video editor (even if there's no real video, just audio) as you described earlier. I may need to call on your srt expertise later.

I've been creating the subtitles with 3 windows open -

1- to select and play each .ogg with a minimal Windows player.

2- A view of the AI vocal script (e.g., source view of the Thug wiki entry), to use as a copy/paste starting point.

3 - The .subs file being built in a text editor.

So, the FM I described is not used in the first place to create subtitles, but rather to do a second-pass review (e.g. by me) or third-pass review (as part of quality control during incorporation into TDM).

 

[datiswous]

Ah I forgot they're inline off course. I've been thinking of puting all the ogg files in one line, generating the subs and then converting them to inline, but I didn't know about the ai vocal scripts on the wiki.

[Geep]

I have a question about the current TDM implementation of "srt"...

For an audio clip using srt with multiple subtitle phrases, the start and end timestamps for each phrases are relative to the clip start. For implementation reasons, it would not be reasonable to have negative time. But it could be both reasonable and useful to have a subtitle that continues on-screen a bit longer than the audio clip. Is that currently possible?

[datiswous]

You mean only the last phrase? So for example you have an srt subtitle file with 20 phrases and the last phrase will be shown a couple of seconds longer than the audio file plays?

Or do you mean for every audio phrase in an audio file? You can do whatever you want. You can set the time for every subtitle phrase how long it will display during playback.

Edited February 6, 2023 by datiswous

[stgatilov]

2 hours ago, Geep said:

For an audio clip using srt with multiple subtitle phrases, the start and end timestamps for each phrases are relative to the clip start. For implementation reasons, it would not be reasonable to have negative time. But it could be both reasonable and useful to have a subtitle that continues on-screen a bit longer than the audio clip. Is that currently possible?

As far as I remember, it is not possible.
If sound ends, it is no longer in the sound system, so it does not generate subtitle anymore.

On 2/3/2023 at 10:45 PM, Geep said:

Unsurprisingly, you can't edit the subtitle within the FM. Just note what needs changing and do it with a text editor.

I think some command can pull the subtitle changes from disk.
It's either reloadDecls or reloadSounds.

[Geep]

@datiswous, just the last phrase.

@stgatilov, confirmed that subtitle display stops when sound ends.

If I edit contents of an .srt file, neither reloadDecls nor reloadSounds sees that change.

Edited February 7, 2023 by Geep

[stgatilov]

1 hour ago, Geep said:

If I edit contents of an .srt file, neither reloadDecls nor reloadSounds sees that change.

Perhaps you can add a space to .subs file and do reloadDecls?

[Geep]

1 hour ago, stgatilov said:

Perhaps you can add a space to .subs file and do reloadDecls?

Maybe. I'll keep it in mind to try, if I get a srt workflow going at some point.

[Geep]

The first-draft of the thug subtitles is done, and also successfully loaded into my test jig with custom soundshaders for the second-draft review/revise. That review will be paused while i work through some issues about best subtitling practices to target. For example, word per minute reading constraints and their impacts.

[Geep]

@stgatilov, in the Subtitles_decls section of the Subtitles wiki page, you show example code, that includes an srt reference.

Wouldn't it be better to guide people to place the corresponding .srt files within the "subtitles" tree (where their .subs live), rather than the "sound" tree?

Also, should FM authors be encouraged (if not required) to prefix .srt files with "fm_" ?  Like "fm_sound8_long.srt" ?

Finally, in the "displayed text" section, under the srt command, there's a desperate need to see example .srt content, in this case, something made up for sound8_long.srt (or fm_sound8_long.srt) like:

1
00:00:06,612 --> 00:00:10,376
Something's wrong with this crystal ball.

2
00:00:15,482 --> 00:00:20,609
Bugger me!
It's not showing the right dream.

3
00:00:25,336 --> 00:00:28,167
Ah! Here we go.

--end

[datiswous]

1 hour ago, Geep said:

Wouldn't it be better to guide people to place the corresponding .srt files within the "subtitles" tree (where their .subs live), rather than the "sound" tree?

Yeah, I thought the same and this is what I effectively do (putting srt files in folder subtitles).

[stgatilov]

22 hours ago, Geep said:

Wouldn't it be better to guide people to place the corresponding .srt files within the "subtitles" tree (where their .subs live), rather than the "sound" tree?

Technically, it does not matter.

It might be convenient to have .srt files at same location and with same name as the sound files that they correspond to.
I think that was the reason to suggest placing .srt into sounds.

Or maybe I did not think to much about it

Quote

Also, should FM authors be encouraged (if not required) to prefix .srt files with "fm_" ?  Like "fm_sound8_long.srt" ?

Are you encouraged or required to prefix your sound samples with "fm_"?

I know many mappers do use some kind of prefix, and that's good.
Others don't and that can cause accidental name collisions.
Nothing new here.

Quote

Finally, in the "displayed text" section, under the srt command, there's a desperate need to see example .srt content

There is a link to .srt description. Also, .srt is kinda standard format.

[datiswous]

Btw. there's some confusing text in that article:

Quote

FM-specific subtitles are always included from the decl named fm_root, which must be in the fm_root.subs file. For instance, the contents of the fm_test.subs file from the example above will have no effect until we add the file fm_root.subs with contents:

https://wiki.thedarkmod.com/index.php?title=Subtitles#include

There's no fm_test.subs in an example. I think it should be fm_conversations.subs . Not sure, but maybe you could check?

Edited February 9, 2023 by datiswous