Writing a multi-thredded wrapper around an audio encoder in BASH...

[lost_soul]

Operating under the principle that nothing ever goes fast enough and I have a hex-core CPU, I came up with the following.

 

Problem: The audio encoders (flac, oggenc, opusenc) all use a single core. This means if you want to encode a massive collection of music, it is going much slower than it can.

 

Proposed solution a: We create a database in /tmp called files.lst. Then we run a script that searches the current directory and ALL sub directories for any .wav or .flac files. On each file that is found, fork and do the following:

*check /tmp/files.lst for the name of the audio file and its current status. Status can be either %ip for in progress, %fn for finished, or (unlisted) for pending. This way we don't try to encode the same file twice. If the file status is %ip or %fn, we ignore it, otherwise, we spawn an oggenc process on the file to encode it and we add it to the database in /tmp/files.lst as %ip. When that oggenc is finished, we update the status to %fn.

*We also need a way to limit the number of forks running at a time, otherwise you will get 900 oggenc processes running in a directory because it had 900 wav files.

 

Proposed solution b: We split the input file into chunks based on how many CPU cores are detected (or specified). Then we pass the chunks of audio to separate instances of oggenc and encode them all at the same time. When that is finished, we take the multiple outputs from all the oggenc processes and mash them together into a single audio file representing the encoded version of the original wav file before it was split.

 

Which way is better, and is there already a script to do this sort of thing?

Edited September 23, 2013 by lost_soul

[AluminumHaste]

I always though the second solution was the way to properly implement multi threading. Programs like EZ Audio Converter let you set how many threads to use for encoding/transcoding audio, but what it does is assign each core a separate audio file. So you can do like 8 files at once.

But if you only have 4 files to transcode then having 4 extra cores will do nothing. So I like your way.

[Lux]

Couldn't you also just run a do;while loop in there that, say every 5sec queries processes, pipes that to a text file using grep so as to count the number of .ogg encoding threads running, and if there are <X amount it can add until it reaches max, at which point it just queries until spots open up? The X max amount could be set at the script initialization with a processor core count. (I'm a linux scripting noob) :\

 

Of course your solution is the more elegant, proper, solution.

Edited September 23, 2013 by Lux

[i30817]

I always dislike when people use terms i know inaccurately. Forking processes is not threading.

[Ladro]

=> http://code.google.com/p/flac2all/

 

This video exlain the results: