[Resolved in TDM 2.10] Doom 3 and 3Dc (ATI1/BC4 & ATI2/BC5)

[MoroseTroll]

Well, I've made some hacks on Doom3.exe in order to "convince" it to support 3Dc format (ATI1/BC4 and ATI2/BC5). Of course, all other texture formats still should work as well. I must admit that I haven't try these hacked Doom 3 executables with 3Dc normals, but I've ran both Doom 3 and TDM with these hacked executables, and everything seems was ok. Does anyone want to try?

If yes, please notice that my archive consists of two executables: Doom3_LATC.exe and Doom3_RGTC.exe. Jean Michiel Paul van Waveren, id Software, and Ignacio Castaño, nVidia, think that LATC is more convenient than RGTC in some cases, so I provide both versions: LATC and RGTC.

I must warn you: there is no warranty that my executables will work as it should, so use it on your own risk.

[Serpentine]

LACT is ideal because of the implicit fallback for unsupported hardware/driver(now that nvidia finally supports 3dc in its shitty workaround way, there shouldn't be anything that doesn't), however that'd still require testing as doom3's dxt implementation is not very robust at all.

 

While this work is very interesting because compressed normals would be a _huge_ benefit to TDM, we cant really distribute modified executables. since converting to dds isnt too hard I'd think that testing the differences would be something quite interesting to do!

[New Horizon]

we cant really distribute modified executables. since converting to dds isnt too hard I'd think that testing the differences would be something quite interesting to do!

We can't distribute the executable, but can we patch it? That's quite common in the Thief world.

[nbohr1more]

If you could combine this with the 4GB patcher that LEGION provided (and helped to create St Alban's) that would be a nifty tool. Would incorporating an executable patcher in the mod be problematic legally? What other support might be patched-in this way ... hardware instancing ?

 

This would be the ultimate prototyping method for your GPL wishes...

 

[MoroseTroll]

Guys, I've changed just 364 bytes in Doom3.exe (you can use "fc /b Doom3.exe Doom3_LATC.exe" to find the difference), so there is absolutely no difficulties for me to create some sort of tiny patcher. But before this, I must be assured that I've made the right thing and that 3Dc really works in Doom 3. If I've made some shit in my hack, then TDM Team can ask id Software to release an unofficial Doom 3 patch (with 3Dc support) just for the Doom 3 mod community or even especially for TDM. But this is just a some kind of little dream. All I want to know is whether my hack does work or does not, and I beg you please, anybody who is interested in this matter, to test it.

[7upMan]

Please excuse the noobish question, but what exactly is the benefit of compressed normales, other that they probably require less video RAM and thus free some resources?

[nbohr1more]

1) 3Dc decompression is supported in hardware so there is no performance hit for using it.

 

2) 3dc compresses much more data than DDS so the amount of video ram (and file size) are significantly reduced

 

3) That extra RAM can be used for more Geometry data (bigger missions) or Post Processing buffer space...

[7upMan]

4) ... or models with more polygons? That would be great, because right now some models seem to have hardly more polys than the original Thief models.

[nbohr1more]

It still won't help with the CPU dependent shadows but... yes, polygonal detail could be increased a little... Or Normal Map detail could be upped tremendously (pick your poison)...

[Tels]

Well, I've made some hacking on Doom3.exe in order to "convince" it to support 3Dc format (ATI1/BC4 and ATI2/BC5). Of course, all other formats are still should work as well. I must admit that I haven't try these hacked Doom 3 executables with 3Dc normals, but I've ran both Doom 3 and TDM with these hacked executables, and everything seems was ok. Does anyone want to try?

If yes, please notice that my archive consists of two executables: Doom3_LATC.exe and Doom3_RGTC.exe. Jean Michiel Paul van Waveren, id Software, and Ignacio Castaño, nVidia, think that LATC is more convenient than RGTC in some cases, so I provide both versions: LATC and RGTC.

I must warn you: there is no warranty that my executables will work as it should, so use it on your own risk.

 

This is very interesting. What exactly did you need to change in D3 to get this to work, or in other words, maybe it can be done without patching the EXE file?

 

As for patching it on-the-fly, at least in linux this is trivial, you could probably even do it in memory after load. Much better would be of course to copy the file, patch it, then run the patched version. That might at the same time also trick any ATI "optimizations" from catalyst not to fire (as they usually look for the executable name), which would be a nice benefit.

 

Edit: The read at http://developer.nvidia.com/object/real-time-normal-map-dxt-compression.html is alone worth it Thank you, that is very useful and interesting!

[Bikerdude]

this sounds awesome guys..

[AluminumHaste]

1) 3Dc decompression is supported in hardware so there is no performance hit for using it.

 

2) 3dc compresses much more data than DDS so the amount of video ram (and file size) are significantly reduced

 

3) That extra RAM can be used for more Geometry data (bigger missions) or Post Processing buffer space...

 

You forgot to mention that 3Dc has way less compression artifacts than standard compression.

[MoroseTroll]

7upMan: AFAIK, 3Dc "ATI2" format has the same memory footprint as DXT5, 128 bits (16 bytes) for 4x4 block of texels. The only benefit of ATI2 comparing with DXT5_nm is much higher quality compression. ATI2 contains two components: X, Y.

3Dc+ "ATI1" format has the same memory footprint as DXT1, 64 bits (8 bytes) for 4x4 texels. This format is perfect for shadowmaps, HDR maps and any other one-component textures.

 

nbohr1more: Yes, 3Dc is supported by hardware but only by DX10-compatible one and higher, i.e. pre-DX10 graphics cards don't support it, at least in OpenGL . So, in order to use 3Dc, the user must have a DX10- or DX11-compatible hardware. Otherwise, normals should be converted in DXT5_nm (ideal case) or even stored uncompressed (worst case).

 

Tels: Like I said, I've changed just 364 bytes in Doom3.exe. What exactly have I done? I've patched every routine I can find which operates DXT1/DXT3/DXT5/RxGB textures in order to introduce ATI1&ATI2 with them. Of course, this work has been done in assembler, so if you want to know what exactly I have changed, just view the patched fragments in some disassembler or debugger. Can this patch be performed without modifying the exe-file, i.e. on the fly, by modifying exe's image in RAM? I don't know, because I haven't done something like this before. But is it a real problem? I think that the end user would agree to patch the exe-file in order to run TDM if this patch would give him/her a better game experience by faster loading, quicker and smoother fps.

[7upMan]

Well, I expect a MoroseTroll's member status to change soon from "Member" to "Graphics Guru". Seriously, you work in Assembler? AFAIK only Demo programmers do that. 

[MoroseTroll]

Well, I'm working in Assembler, but I'm definitely not a graphics guru. Really! The changes I've made in Doom3.exe are very simple - in fact, they are way too simpler than WideScreenMod for Dark Engine or DDFix.

So, does some want to experiment with my hacked Doom3 executables? Or should I visit Doom3World.org?

Edited December 8, 2010 by MoroseTroll

[STiFU]

MoroseTroll, currently all our normalmaps are stored as uncompressed TGA because the DXT compression results in nasty artefacts. We could run a few tests with 3dc, but considering 3dc is only supported by modern cards, which have enough VRAM and GPU power anyway, there is really no benefit in reducing the normalmap quality from uncompressed to 3dc. The people who would profit from compressed normals are those with inferior hardware and they can only run crappy DXT, you understand? Moreover, the update size would vastly increase, because we'd had to ship both DXT and 3dc compressed normalmaps. Ok, added up, they will probably still end up half the size of a regular TGA despite the zip (pk4) compression, but still there are other complications that arise with this, like how do we tell Doom 3 which compressed files to use (currently there is only a toggle switch that tells D3 whether or not to use precompressed textures)? Please note though, that my opinion is by all means not representative for all the team. I am still kind of curious whether this actually works, so I might give it a shot this evening.

 

4) ... or models with more polygons? That would be great, because right now some models seem to have hardly more polys than the original Thief models.

If you think someone will rebuild the models to get more details out of them: Not gonna happen! Also, I really think they are of sufficient quality. Especially Nosslak's newest additions... You're just in a different relation to things if you see them grow up!!

 

Also the GPU still has to handle the more complex geometry, which will result in fps reduction.

[7upMan]

I get what you mean. As long as there are folks out there who actually play TDM on a Geforce 4 MX, more polies are out of question.

[MoroseTroll]

STiFU: Of course, I know about the artefacts of DXT-compressed normals, but please look at this phrase from the article I've mentioned above:

For most tangent-space normal maps better quality compression can be achieved by only storing the X and Y components in the DXT5 format and deriving the Z. This is also known as DXT5nm compression, and is most popular in today's computer games.

Does it mean that DXT5nm format is more-less suitable for many today's games? I think, yes. So why not try to use it in TDM? Yes, I know that DXT5-compressed normals are not accepted by the engine's "addnormals" instruction, but how much of these instructions are in use? I've counted two, maybe three dozens of them in TDM, so I think there won't be so big problem to replace all the normals used in "addnormals" instructions to their final results using Blender or whatever. BTW, do you know that another Doom 3 mod, named Ruiner, is doing that way (I mean it doesn't use "addnormals" at all)?

Moreover, Ignacio Castaño, nVidia, states that he have drastically improved a quality of his offline DXT-compressor, including DXT5nm. So I hope that modern DXT5nm-compression is little better than you say - just look the charts with PSNR.

 

7upMan: Why are you thinking that somebody here, at TDM Forum, or at DarkFate forum, is using GeForce 4MX? That's ridiculous! Most of the DarkFate members I know are using DX10-hardware and even newer. Well, somebody is using DX9-hardware - Radeon 9800 & X1950, but even these GPUs are way too faster than GeForce 4MX. Yes, someone is using notebooks and even netbooks with shitty Intel integrated video, but, AFAIK, as a second or even third PC. Playing Doom 3 on such a crap is a big problem, not alone TDM.

Edited December 8, 2010 by MoroseTroll

[STiFU]

You seem to be a lot more knowledgeable in this topic than I am. But I was just arguing against 3dc, because we need to maintain DX9-Hardware compatibility. If you say DXT5nm results in high quality normals and Dx9 compatibility is given, and if Doom 3 supports it, this should indeed be investigated a little closer. However, we are currently all heavily working on getting the next update out the door, so this will have to wait a little.

[Tels]

Tels: Like I said, I've changed just 364 bytes in Doom3.exe. What exactly have I done? I've patched every routine I can find which operates DXT1/DXT3/DXT5/RxGB textures in order to introduce ATI1&ATI2 with them. Of course, this work has been done in assembler, so if you want to know what exactly I have changed, just view the patched fragments in some disassembler or debugger. Can this patch be performed without modifying the exe-file, i.e. on the fly, by modifying exe's image in RAM? I don't know, because I haven't done something like this before. But is it a real problem? I think that the end user would agree to patch the exe-file in order to run TDM if this patch would give him/her a better game experience by faster loading, quicker and smoother fps.

 

Apart from the fact that you'd need to repeat the patch for Linux, having a non-patched exe just feels cleaner. Of course, it might simply not possible, so still this is great work.

 

Btw what if the user has a DX9 or older card?

 

@Stifu: Load times would still be less for users of newer cards (loading 250 vs 64 Mbyte of textures definitely makes a difference), also, I would't say that it won't help newer cards, less memory accesses for the same output sounds good to me.

 

@7upman: Oh, so I am a demo programmer?

[Tels]

STiFU: Of course, I know about the artefacts of DXT-compressed normals, but please look at this phrase from the article I've mentioned above:Does it mean that DXT5nm format is more-less suitable for many today's games? I think, yes. So why not try to use it in TDM? Yes, I know that DXT5-compressed normals are not accepted by the engine's "addnormals" instruction, but how much of these instructions are in use? I've counted two, maybe three dozens of them in TDM, so I think there won't be so big problem to replace all the normals used in "addnormals" instructions to their final results using Blender or whatever. BTW, do you know that another Doom 3 mod, named Ruiner, is doing that way (I mean it doesn't use "addnormals" at all)?

Moreover, Ignacio Castaño, nVidia, states that he have drastically improved a quality of his offline DXT-compressor, including DXT5nm. So I hope that modern DXT5nm-compression is little better than you say - just look the charts with PSNR.

 

I think the problem with this is that we haven't compressed the normal maps, because there *could* be someone using them in addnormals. It is not like TDM is locked, every mapper can create material definitions nilly willy and if these suddenly don't work anymore, they cry

 

However, I am with you, compressing the normals would far outweight the benefits of being able to use them in addnormals, plus many of them (as you said) aren't even used or useful for addnormals.

 

7upMan: Why are you thinking that somebody here, at TDM Forum, or at DarkFate forum, is using GeForce 4MX? That's ridiculous! Most of the DarkFate members I know are using DX10-hardware and even newer. Well, somebody is using DX9-hardware - Radeon 9800 & X1950, but even these GPUs are way too faster than GeForce 4MX. Yes, someone is using notebooks and even netbooks with shitty Intel integrated video, but, AFAIK, as a second or even third PC. Playing Doom 3 on such a crap is a big problem, not alone TDM.

 

There are regulary people with old hardware who post here, and also some teammembers express concern for this. I am more of the "well, duh, invest 50$ for a decent graphic card and throw your old heater away" crowd, and I think 7upman was using sarcasm to say the same - the entire "we need less polygons etc" discussion seems sometimes driven by the argument that hardware (which is way outdated) should still be able to run TDM. Instead we should use the gained performance to improve the visuals (but this is my opinion, not official team etc yadda yadda).

[7upMan]

 

7upMan: Why are you thinking that somebody here, at TDM Forum, or at DarkFate forum, is using GeForce 4MX?

 

Well,

 

 

* Slowly Engine

May disable most visual options. So, i read, that someone running DooM III on GeForce 2 MX440. And other says "Still game requests too much for medium quality graphics".

But maybe there was just a misunderstanding?

 

@7upman: Oh, so I am a demo programmer? 

 

 

Omigod. *falls to his knees*  Ich bin unwürdig!!!   

[STiFU]

Btw what if the user has a DX9 or older card?

If you have an older card you can't play Doom 3 at all.

 

@Stifu: Load times would still be less for users of newer cards (loading 250 vs 64 Mbyte of textures definitely makes a difference), also, I would't say that it won't help newer cards, less memory accesses for the same output sounds good to me.

Possibly. This is really something we could investigate a little closer in the future, as I said above. But still, the people who would profit the most are those with crappy hardware...

[Tels]

If you have an older card you can't play Doom 3 at all.

 

You only answered one half: And what about an DX9 card?

 

Possibly. This is really something we could investigate a little closer in the future, as I said above. But still, the people who would profit the most are those with crappy hardware...

 

Yeah, but that doesn't mean that the people with not-so-crappy hardware wouldn't/couldn't benefit, either. But now I remember why we don't compress the normals (or get anything done these days), because there is always someone who comes and declares it "useless, not worth it, only worth for persons X" etc, and then the people who do the actual work get annoyed, disgusted and go away. Leaving only the "discussion" and no result...

[STiFU]

You only answered one half: And what about an DX9 card?

It was merely a joke. As I stated above, breaking DX9-Hardware compatibility is a no-go, so 3dc is no option.