Jump to content
The Dark Mod Forums

Automatic subtitles generation


datiswous

Recommended Posts

Edit: in post 5 I discovered Whisper which does this task MUCH better. So don't use vosk. Some of the info till post 5 is still relevant for subtitle editing in Kdenlive in general.

I previously posted about this in a status update. To make it a bit more in-view for the future I post the info also in this topic.

I recently figured out how to make subtitles work for missions following this wiki guide: https://wiki.thedarkmod.com/index.php?title=Subtitles

You can type in the subtitle-text manually either in the .subs or .srt files (in a text-editor) or use an video editor for that (recomended for .srt). What is also possible on some advanced editors including the free and open source multiplatform (Windows, Linux and Intel-Mac) Kdenlive editor is to auto generate the subtitle text for you from the audio or video file. You can then export to an .srt file that works directly in tdm. If you want to use the subs files for shorter sentences, you can just copy text from the .srt files.

In Kdenlive you can install speech to text libraries from VOSK. For this to work you have to download and install Python. Info how to do the process of installation and usage can be seen in the following video (6.5 minutes):

Spoiler

 

To sum it up:

Configure first time:

  1. Install Python. (on Windows) During setup, you have to select Advanced Options and there mark Add Python to environment variables (super important!).
  2. In Kdenlive go to menu settings, click on configure Kdenlive.
  3. In that configure window, click in the left menu on Speech to text.
  4. There you click on the link to download speech models.
  5. On the website ( https://alphacephei.com/vosk/models ) you can click on a model download link, but keep the click pressed and move your mouse with the link to the configure Kdenlive window. Kdenlive then asks to install the model from url. vosk-model-en-us-0.22-lgraph is probably decent for most use cases. but you can install and test them all.

To use it:

  1. First load an audio or video file into the view by dragging the file in one of the audio or video bars at the bottom (video: v1, v2 or audio: a1, a2).
  2. Click on menu Project > Subtitles > Edit Subtitle tool. You see an extra Subtitles bar on top.
  3. Now you select the audio or video file (it is sellected when it is outlined with an orange border) in the specific bar and then click on menu Project > Subtitles > Speech recognition.
  4. In the Speech recognition dialog, you select the correct language model and choose option Selected clip.
  5. After generation, you can preview the generated subtitles via the top right window. Make sure it is at starter position for playback. Using an audio file, you see a black background with the subtitles on top.
  6. Now you can tweek the position and edit the text directly in the Subtitles bar. This takes up the most time. Unfortunatelly the generation is not flawless, so you have to correct some words.

Tweeking the subtitles for Requiem took me hours, becouse I wanted them to line up differently. Usually the subtitles are not generated as full senteces. This looks sloppy. If you want to add subtitles quickly without spending much time on it, it can be done this way. If you want to do it right, it still takes a lot of time in my experience.

To export to .srt is shown in the following video:

Spoiler

 

Although actually it's just one step:

  1. Click on menu Project > Subtitles > Export subtitle file.

Alternativelly you can just save the kdenlive project and then the srt is exported as well. Every save will update the srt file.

 

I might create a wiki article about it later.

Kdenlive edit window:

image.thumb.png.06d604033e7905085202fdd69641659e.png

 

Edited by datiswous
  • Like 2
Link to comment
Share on other sites

The subtitle generation doesn't take different voices into account. So often different voices end up in the same subtitle line (so you have to devide them by editting). Can for example be seen in fm Hair in the Snare where the mission intro has 2 voices.

Edited by datiswous
  • Like 2
Link to comment
Share on other sites

  • 1 month later...

I just found that if you save in kdenlive as a kdenlive project, the subtitle srt track is automatically exported as well and every time you save in kdenlive again the srt file is updated. So manual exporting isn't actually needed. The srt is saved in format filename.kdenlive.srt .

  • Like 2
Link to comment
Share on other sites

Automatic subtitles generation with Whisper

I found a far better alternative to auto generate almost perfect srt files: Whisper.

https://openai.com/research/whisper

https://github.com/openai/whisper

For example I did a test with file Simeon3.ogg, a 44 seconds voice file from fm A house of locked secrets. By using command in terminal:

whisper Simeon3.ogg --model small.en

After a very short time (could be due to it using nvidia cuda, not sure) it creates a bunch of export files, including an srt file with contents: (be warned that you will read the contents of an audio file of a mission, potentially spoiling something)

Spoiler
1
00:00:00,000 --> 00:00:07,000
When our time in this world is at an end, our body returns to the earth and our soul goes to the builder's side.

2
00:00:07,000 --> 00:00:15,000
Sometimes the body refuses to embrace the earth, these become the undead, to be purified with the hammer and holy water.

3
00:00:15,000 --> 00:00:21,000
However, sometimes it is not the body, but the spirit that stays in this world after death.

4
00:00:21,000 --> 00:00:26,000
These spirits are known by many names, ghosts, shades, phantoms.

5
00:00:27,000 --> 00:00:34,000
Some are benign and some are harmless, but others can wreak terrible havoc on the world of the living.

6
00:00:34,000 --> 00:00:40,000
As it is our duty to repel the undead, so it is also our duty to repel these spirits.

7
00:00:40,000 --> 00:00:44,000
The holy symbol you have picked up shall enable you to do so.

This is almost exactly how it is supposed to be. Not only does it pick up the language all perfect, it also creates full sentences with punctuation. This is far better than the VOSK method in Kdenlive, which I had to edit afterwards.

After that I load it in Kdenlive together with the sound file and make a couple of easy corrections to the flow. Basically you have to make sure that in the gaps in the audio file the subtitle sentences also stop. See example below:

Spoiler

image.thumb.png.7a02e8c32ec2dbaec2bea725fdb00588.png

This is probably a 10x faster workflow.

 

Edit: You can also list all the files in the command. For example (from dir with voice files):

whisper Carlotta1.ogg Carlotta2.ogg Carlotta3.ogg -o ./../../subtitles --model medium.en -f srt

This command generates the subtitles from these 3 files and saves them in the subtitles folder in (only) srt format.

If you want inline, you copy them over from the srt files. Instead you can just use srt for all voice files.

image.png

 

Edit 2

Currently I use this workflow:

  1. In the folder with voice audio files I make a folder subtitles.
  2. Then I open a terminal window in the voice folder.
  3. Then I do the following command in the terminal:
    whisper Carlotta1.ogg Carlotta2.ogg Carlotta3.ogg -o ./subtitles --model medium.en -f srt
  4. Afterwards I move the subtitle folder to another folder with voice files and repeat steps 2 and 3 or if I'm done I move the folder to the root folder of the mission.
Edited by datiswous
Link to comment
Share on other sites

Btw Whisper can apparently also do direct text translation, which could be useful later if TDM will support multi-language subtitles.

Edit: This might not work for En -> Other language.

Edit2: Aparently Whisper can directly transcribe for a language, so if you would make speech files in a different language, it would generate to that language subtitle. Maybe I could test it with translated missions, if they exist.

Edited by datiswous
Link to comment
Share on other sites

  • datiswous changed the title to Automatic subtitles generation

Edit: See this post for better solutions for Windows.

I tried to install Whisper on a Win10 pc, but couldn't get it installed. I found this Whisper gui-software which works just as well:

https://github.com/chidiwilliams/buzz

After the subtitles are created, you have to double-click on each item and export to srt.

Spoiler

image.png.70e806ec811c5ed01ff9666908efd502.png

Edit: Hmm, I seem to get different results.. Oh well.

Edited by datiswous
Link to comment
Share on other sites

  • 4 weeks later...
On 3/10/2023 at 2:36 AM, datiswous said:

Edit: You can also list all the files in the command. For example (from dir with voice files):

whisper Carlotta1.ogg Carlotta2.ogg Carlotta3.ogg -o ./../../subtitles --model medium.en -f srt

This command generates the subtitles from these 3 files and saves them in the subtitles folder in (only) srt format.

To get a horizontal list of files in Linux, navigate to the folder with soundfiles in a maximized terminal (cmd) window and type:

dir --format=horizontal

You see then all the sound files listed in a horizontal line. You can then type whisper and copy-paste that line of files after that. I usually first create the full command line temporarly in a text editor before posting it in a terminal window.

Edited by datiswous
Link to comment
Share on other sites

  • 9 months later...
Posted (edited)

Subtitle Edit ( https://www.nikse.dk/subtitleedit ) now has support for subtitle extraction via Whisper.

See https://www.nikse.dk/subtitleedit/help#audio_to_text . This works well on Windows. In the extraction window you can download all the needed extra dependencies the first time you use it. After the generation of the srt files, you can use the editor to tweak the files, or move to a seperate editor of your choice (including texteditors).

Aperantly it also works under Linux:

https://www.nikse.dk/subtitleedit/help#linux

If it doesn't, see info above to use the commandline in Linux.

 

Kdenlivehttps://kdenlive.org ) now also has Whisper subtitle extraction build in. This works well in Windows, but I couldn't get it working in Linux.

You have to go to Settings > Configure Kdenlive. Then go to section Speech to text. On top of the window you select option Whisper. Then you have to install some stuff by clicking on an install button (this doesn't work in Linux currently). The extraction via cpu is considered slow, but I thought it's not so bad using an 8th generation i3 processor during a test with a large speech file. You can afaik only do this one by one, so it's not as fast.

Edited by datiswous
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recent Status Updates

    • Ansome

      Finally got my PC back from the shop after my SSD got corrupted a week ago and damaged my motherboard. Scary stuff, but thank goodness it happened right after two months of FM development instead of wiping all my work before I could release it. New SSD, repaired Motherboard and BIOS, and we're ready to start working on my second FM with some added version control in the cloud just to be safe!
      · 0 replies
    • Petike the Taffer  »  DeTeEff

      I've updated the articles for your FMs and your author category at the wiki. Your newer nickname (DeTeEff) now comes first, and the one in parentheses is your older nickname (Fieldmedic). Just to avoid confusing people who played your FMs years ago and remember your older nickname. I've added a wiki article for your latest FM, Who Watches the Watcher?, as part of my current updating efforts. Unless I overlooked something, you have five different FMs so far.
      · 0 replies
    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
×
×
  • Create New...