TDM file format support info

[datiswous]

Where can I find info on supported file formats?

It would be nice if there would be one place where I can find info about file formats for different uses. For example:

• models
• ragdols (?)
• animation
• video
• textures
• images inside guis (briefing, readables)
• sound files

I cannot find this info on the wiki, but maybe it's there, but difficult to find.

Edited December 2, 2024 by datiswous

[datiswous]

So I think for models there's ase, lwo and obj (?).

For sound there's ogg and wav.

Video there's mp4 and avi (ROQ) (?)

Textures tga, dds and maybe png and/or jpg (at least one of those 2)

[datiswous]

Nobody knows anything?

[nbohr1more]

We support the same formats as Doom 3 except we added ob,j md3, rgtc dds, ogg, png, and a few ffmpeg codecs.

https://modwiki.dhewm3.org/File_formats

[datiswous]

What is rgtc?

44 minutes ago, nbohr1more said:

ob,j md3, rgtc dds, ogg, png, and a few ffmpeg codecs.

Is tga, png, jpg and dds all also possible inside gui's?

Edited December 4, 2024 by datiswous

[nbohr1more]

Yes, but I think png and jpg require a material def that can then be referenced by the GUI keyword for images via the material path

[datiswous]

Is there a wiki page which lists this?

[stgatilov]

Here is what I know:

static models:

LWO:

• somewhat default
• small uncompressed size since it is binary
• loading code is horrible, can be slow and break your model (smoothing groups might be pain)

ASE:

• larger uncompressed size since it is text (compressed size is probably OK?)
• loading code is as awful as for LWO

OBJ:

• the only format added specifically for TDM, so not very widespread
• uncompressed size is large because it is text (compressed size is probably OK?)
• loading is very fast and does not "fix"/break your model, you'll get exactly the topology & smoothing you exported

ragdolls:

The only format is .af text format, which is handmade for Doom 3.
No changes here since Doom 3.

It is notoriously hard to create/edit.
The only tool is the Windows-only builtin command editAFs, and this is the only in-game editor which has no outside alternatives. We wanted to port it to either DarkRadiant or at least better GUI framework, but it did not happen.

animation:

MD5:
md5mesh --- Doom 3 custom text format which represents a mesh attached to skeleton (joints, weights, etc.)
md5anim --- custom text format that represents an animation of a skeleton.
I'm pretty sure Blender has some working exporter for these.

MD3:
This is Doom 3 custom text format which represents linear vertex morphing.
It is rarely used, but it is a perfect fit in case animation is clearly not "skeletal" (e.g. animated grass or water).
I suppose some converters can be found?... Ask @Arcturus, he used it.

Also there are some hardcoded pieces of code which transform a mesh into something else, that are enabled via "deform" keyword in material. The most known example is particles. All of this is very specific and uses custom formats.

There is also MA loader (Maya ascii format?), but it is not used an I have no idea how functional it is.
Also there is FLT model, but no idea what it is.

video:

The mainstream idea is to use (mp4 + h264 + aac) for FMV.
There is also legacy support for ROQ, but I guess you should not use it today.
I think all the other formats are disabled in FFmpeg build that we use.

There is no common understanding about in-game videos, I'm not even sure it is a useful case.
One specific case is animated textures (e.g. water), and they are usually implemented as a pack of copy/pasted if-s in material file, which point to different frame images. Something like "Motion JPEG/TGA/PNG/DDS" Maybe one day we will make this approach easier to use...

textures and images:

TGA:
The simplest uncompressed format, basically default in case compression is not used.

PNG:
Lossless compression, but pretty slow to load.
Should be OK for some GUI images, but probably not a good idea to e.g. write all normal maps as PNGs: I imagine loading times will increase noticeably if we do so

JPG:
Lossy format, but pretty small and rather fast to load.
I think it is usually used for screenshots and maybe GUI images, otherwise not popular.

DDS:
This is container which can in principle store both compressed and uncompressed data, but I think we use it only for precompressed textures. TDM supports DXT1, DXT3, DXT5, RGTC compressed formats (each of these formats has a bunch of pseudonymes).

If you use compressed DDS, then you can be sure it will be displayed as is, as no further processing it applied to it. Sometimes it is good, sometimes it is bad.
The engine has fast compressor/decompressed for all texture compression formats, so if you supply TGAs and they are compressed on load, you only lose in mission size.

Images inside guis (briefing, readables) use the same material loading system, so they should support all the same image formats.

sound files:

OGG:
Relatively widespread open format with lossy compression.
I guess it is used for everything by default, which seems OK to me.
OGG files are loaded on level start, but decompressed on the fly.

WAV:
Simple uncompressed sound file.
Obviously larger than OGG, not sure why it is used sometimes (maybe less compression artifacts?)
The only thing I know is that the "shakes" feature of materials specifically asks for WAV.

[datiswous]

Thanks. Is it ok to put this info on a wiki page? Maybe remove some negative wording (horrible). You can edit it later off course.

[nbohr1more]

Wiki sounds good

[stgatilov]

5 hours ago, datiswous said:

Thanks. Is it ok to put this info on a wiki page? Maybe remove some negative wording (horrible). You can edit it later off course.

Yeah, you can do with it whatever you want.

[HMart]

Just need to say this 

Quote

lwo ... break your model (smoothing groups might be pain)

This was not true at all for me, when I modeled for Doom 3 at lest, all my lwo models worked fine, including smoothing, thou I use Modo 601 that was the exact modeling tool Seneca a model artist from idSoftware used and I also used his custom Doom 3 exporter pluging for Modo that cleans up and prepares .lwo files to Doom 3 prior to exporting.

This is what I do in Modo 601 to export clean lwo files, perhaps helps those using Blender as well.

1 - put model origin at world 0,0 and move model geometry if necessary so its origin is at the place you want, like the model base. 

2 - (optional) clean geometry using some automatic system, if available, so your model doesn't export with bad data (like single vertices floating disconnected). 

3 - bake all model transforms, so rotation, translation and scale are all put at clean defaults (that will respect any prior scaling and rotation you may have done)

4- (in modo smoothing works this way)  go to the material section and if you want your model to be very smooth (like a pipe for example) set material smoothing to 180%, you can set smoothing per material. If you want more smoothing control, you can literally cut and paste the faces/quads and you will create smoothing "groups" or islands that way.

5 - use or select a single model layer and export that (the engine only cares for a single lwo layer) if you want a shadow mesh and collision mesh as well, put them all on the same layer as the main model.

6 - never forget to triangulate model before export (Seneca plugin did that for me automatically)

I did this and all my models exported fine and worked well ingame. Thou .lwo was invented by lightwave and Modo was developed by former lightwave engineers/programmers, so perhaps their lwo exporter is/was well optimized/done compared to other 3D tools. 

Edited January 4 by HMart

[stgatilov]

I remember one case where smoothing groups were a pain to transfer to TDM correctly:

And I looked into TDM code which preprocesses LWO models.
Of course it usually works fine, but it is not a reliable solution. If you encounter a model which is broken by this import code, you will have to workaround it on your side, either by tweaking the model, or by tweaking the exporter.
Also, I believe I saw a large LWO model (terrain) which took a lot of time to load because of this preprocessing.

OBJ format is very good because it supports multi-patch texture atlas and smooth groups topologically. There is zero preprocessing on TDM side: you'll get exactly what is written in the OBJ file. And loading large model is much faster despite OBJ being text format.

Anyway, if someone wants to maintain a wiki article about the formats, then I think it would be very helpful to list all the working export tools. Names, versions, instructions and forum threads, known caveats, etc.

[peter_spy]

If OBJ works correctly in TDM, it should be a recommended format, at least over ASE. Even though both are text-based, OBJ is much more lean, and it compresses faster and better. This is a high-poly cube (24k polygons) saved and compressed in both formats:

I haven't tested OBJ with 2.11 yet, but ASE has other downsides, like it has to be exported from a clean scene, and the scene material library can only have materials that are used for a particular model.

OBJ should just use what's been assigned to it, and it should use model pivot point instead of world/scene origin point (0,0,0). If that's true, then exporting single models should be much easier now. You won't need a separate clean scene do do your exports.

[datiswous]

On 1/2/2025 at 8:53 PM, stgatilov said:

ragdolls:

Are knocked out ai also ragdolls?

[stgatilov]

9 minutes ago, datiswous said:

Are knocked out ai also ragdolls?

Yes, humanoid body is the main ragdoll we have.
And other AI bodies as well. And maybe ropes are ragdolls too.

[datiswous]

On 1/2/2025 at 8:53 PM, stgatilov said:

OGG files are loaded on level start, but decompressed on the fly.

Would that mean using WAV might give you better performance?

Edit: Just wondering. It's probably not a good enough reason to use it.

Edited January 6 by datiswous

[datiswous]

I made a draft from the info from @stgatilov:

https://wiki.thedarkmod.com/index.php?title=File_formats

I will edit it further later. Feel free to add info etc.

[HMart]

On 1/2/2025 at 7:53 PM, stgatilov said:

Also there is FLT model, but no idea what it is.

afaik .flt was a old OpenFlight model format that Jon Carmack was using to test MegaTextured terrains using idTech4 as a test bed, I don't think anyone ever used it, besides him. Never tested it but is probably broken and doesn't work, like MD3 also didn't worked initially.

[OrbWeaver]

If we're going for completeness, it might be worth mentioning:

• TGA doesn't have to be uncompressed, it can use RLE compression (which can give better compression ratio than DXT for very smooth or low-detail images with large solid areas).
• PNG has variable compression, but is always lossless.
• Ogg Vorbis also has variable compression, but is lossy, with lower bitrates giving worse quality sound. Should not be used for repeated editing due to generation loss.

[peter_spy]

Btw. IMO it would be useful to use .dds images in regular myFM/textures folder structure, so you don't have to replicate that in dedicated dds folder. TDM doesn't use texture quality settings anyway, it's a deprecated Doom3 thing.

[datiswous]

On 1/2/2025 at 8:53 PM, stgatilov said:

TDM supports DXT1, DXT3, DXT5, RGTC compressed formats (each of these formats has a bunch of pseudonymes).

On the wiki page you added:

Quote

DXT3 supports transparency

But on this wiki page: https://wiki.thedarkmod.com/index.php?title=DDS_creation#DDS_Creation,_In_General

It states:

Quote

DXT5 for textures with alpha channel transparency

So which one is correct?

 

From info here it seems they both support transparency:

http://wiki.polycount.com/wiki/DXT

 

Edit: Maybe you mean BC3 instead? Or maybe you mean DXT3 and up ?

Edited January 9 by datiswous

[HMart]

Afaik both are correct, I was taught to use dxt5 for normal maps 

[datiswous]

I see there's also support for BC5 (for normalmaps, since 2.10), but not BC7 (probably because it requieres a DX11 graphic card).

Edited 21 hours ago by datiswous

[datiswous]

On 1/2/2025 at 8:53 PM, stgatilov said:

There is no common understanding about in-game videos, I'm not even sure it is a useful case.

I don't know what you mean with this. There are whole wikipages about this and it's implemented in multiple missions.

You can also display video's as textures directly.

 

On 1/2/2025 at 8:53 PM, stgatilov said:

One specific case is animated textures (e.g. water), and they are usually implemented as a pack of copy/pasted if-s in material file, which point to different frame images. Something like "Motion JPEG/TGA/PNG/DDS"

Do you have an example of this in a mission?