[Resolved in TDM 2.10] Doom 3 and 3Dc (ATI1/BC4 & ATI2/BC5)

[Serpentine]

You're getting confused here.

 

3dc on nvidia uses a work around using two DuDv maps, its done in the background and afaik works on 6xxx(2005, tho support was added via drivers much later on afaik) and newer, which should be fine.

on ATi's side, 3dc works on 9xxx(the 2003-4 cards) and newer, which is more than enough.

Intel supports it in most new chips too(windows drivers only, afaik), and we don't even really support them... so moot there.

 

The question is more of 'does it even work'.

 

Compression of normals does not just benefit low end hardware, atm we cant use high res normalmaps at all due to the memory footprint being 16mb for 2048x2048, so releasing them is next to impossible. Auto mipmap generation is also being phased out in OpenGL-land afaik, something which currently is done at load time, slowing the process significantly. The loss in visual quality is negligible for most surfaces, we don't use a lot of extremely low frequency normals and at the end of it all... tga is not being removed as an option.

 

Until we know if these changes :

1. Do what they say on the can
   
2. Work on expected hardware
   
3. Is there any horrible/unexpected knock on?
   

 

It's a bit useless to debate it.

[Tels]

The first poster has gone all the way to actually build support for it (in windows), so you *can* test them. (I can't as I am on linux). That is much further than we ever came (we always stop at the "pro and con discussion", but never attempt normal compression anyway).

 

So, people, try it, and please post your test results.

[7upMan]

How will this work, if it works at all? Is there a software that I fire up and that will convert or compress all my uncompressed normal maps into compressed ones? I'm a bit lost here, so I'd definitely appreciate some advice.

[AluminumHaste]

How will this work, if it works at all? Is there a software that I fire up and that will convert or compress all my uncompressed normal maps into compressed ones? I'm a bit lost here, so I'd definitely appreciate some advice.

 

You need ATi Compressonator tool (Link) afaik. Example

[Tels]

You need ATi Compressonator tool (Link) afaik. Example

 

Could somebody please compress a few normalmaps as an example for us? Running that program on Linux can be tricky...

[MoroseTroll]

You seem to be a lot more knowledgeable in this topic than I am. But I was just arguing against 3dc, because we need to maintain DX9-Hardware compatibility. If you say DXT5nm results in high quality normals and Dx9 compatibility is given, and if Doom 3 supports it, this should indeed be investigated a little closer. However, we are currently all heavily working on getting the next update out the door, so this will have to wait a little.

I wonder how many DX9-cards are really in use by TDM-fans... It seems to me that almost everyone of them has at least DX10-cards, and many - even DX11-cards. I must agree with you that the maintaining DX9-compatibility of TDM is still important, but I hope after one year it won't. I wish I would see some kind of an automatic hardware survey like the Steam's one...

As for the next update of TDM: of course, there is no doubt that you, guys, have a lots of work. Maybe in the next one?

Edited December 9, 2010 by MoroseTroll

[MoroseTroll]

Apart from the fact that you'd need to repeat the patch for Linux, having a non-patched exe just feels cleaner. Of course, it might simply not possible, so still this is great work.

 

Btw what if the user has a DX9 or older card?

Firstly I'd like to be assured that my 3Dc-hack works on the Windows-executable of Doom 3. If it so, I think I can hack the Linux-executable too.

DX8-cards I know are: nVidia GeForce 3 / 4 Ti (GeForce 4MX is a DX7-card), ATI Radeon 8500-9250. All these cards are drastically slow to run Doom 3 even in 800*600 with medium quality, and I'm quite sure that nobody from TDM fans is using them.

DX9-cards (nVidia GeForce FX / 6 / 7, ATI Radeon 9500..9800 / X300..X850 / X1300..X1950) are still on track, but I'm sure that after one year they will be upgraded to much more powerful DX11-ones.

So yes, 2011 will be a DX9-year for TDM, but I'd like propose you, guys, that you'd suggest all TDM fans to upgrade their DX9-cards to DX10- or DX11-ones.

[Aprilsister]

Not gonna happen, dude.

[ungoliant]

I would love absolutely nothing more than to upgrade my radeon x1950xtx.... but i kinda got a situation right now. i'm not even sure if i'm going to have internet much longer, much less buy a f'ing graphics card

[MoroseTroll]

Serpentine: AFAIK, 3Dc support has been integrated into ATI Radeon X800 (and all cards later) in 2004, not earlier, with GL_ATI_texture_compression_3Dc. Alas, this OpenGL extension was never supported by nVidia (according to Realtech-VR OpenGL Extensions Viewer), so the only way to use 3Dc on non-ATI/AMD cards in Doom 3 is to use GL_EXT_texture_compression_latc/rgtc, which is possible only on DX10-compatible hardware.

 

Aprilsister, ungoliant: Ok, ok, now I see that my proposal about upgrading DX9-cards (and using 3Dc) was, say, too early .

Well, let's think about DXT5nm instead of 3Dc (but I'll post the same topic on Doom3World.org ). DXT5nm is supported on every DX9-card (and even on DX8-ones), so I hope it would be nice to try use it instead of uncompressed TGAs. If I'm not mistaken, all I need to use DXT5nm-compressed normals is to move them into "DDS" folder, right? So, I'll try to do it this weekend.

[STiFU]

Yeah, that's correct. Just make sure the subtree of folders stays the same, e.g. textures\darkmod\stone\flat\bla.tga becomes dds\textures\darkmod\stone\flat\bla.dds. Also image_usePrecompressedTextures has to be enabled, but this is on by default in TDM.

 

What I am particularly interested in would be to compare performance with and without normalcompression through our benchmark demo. Loadtime and average fps should be compared. It is important that the files are in both cases either outside or inside pk4. If one compares the results of uncompressed normals inside pk4 with compressed normals outside pk4, the measurements for loadtime will be of. Of course, we would also have to do a quality comparison.

[7upMan]

What tool would be used to create DXT5nm normals? The Compressonator doesn't seem to use it.

[New Horizon]

Nvidia texture tools are supposed to be able to do it.

 

Not sure if D3 will be able to use them, but it's worth a shot. Perhaps it would require hacking the executable as well?

http://developer.nvidia.com/object/texture_tools.html

[MoroseTroll]

I've learned from my analysis of Doom3.exe that it treats DXT5 and RxGB as identical texture formats. So, if I'm correct, I hope there will be no difficulties to feed some DXT5nm-compressed normals (with nVidia Texture Tools, for example) to Doom 3.

[Serpentine]

I've learned from my analysis of Doom3.exe that it treats DXT5 and RxGB as identical texture formats. So, if I'm correct, I hope there will be no difficulties to feed some DXT5nm-compressed normals (with nVidia Texture Tools, for example) to Doom 3.

 

It does in some regards, however during my tests I found that the implementation does not allow anything other than RxGB to be used for normal maps, if the file is a DXT. You can however use them as a diffuse or any other stage. I couldn't find the logic for this however and didn't dig too far down. I'm pretty sure that you could mangle the files to work by claiming the pixelformat is RxGB, however we're not in the business of breaking standards

 

If you're interested in reversing parts of doom3's executable I'd suggest getting hold of the latest MacOS binaries as they have not been stripped (at all, from what I've seen), at least to help understanding logic and such. If you cant find a nice copy I'm sure I can upload mine.

[STiFU]

Maybe JC could help us out here. I know he talked a lot about storing alternative data sets in the alpha channel of normalmaps, so maybe he could do something about supporting dxt5nm. I just did a quick test with this myself, out of curiosity and Doom failed to load the texture.

[MoroseTroll]

It's me. Again .

 

First. You, guys, have made a great job with the 1.03 version. The game really looks better.

 

Second. I have performed TGA->RxGB transformation for the 1022 normals (1.3 GB in total) and have tested the time of a mission loading (RxGB vs. TGA):

NHAT1/3 - 62 vs. 105 sec., speed boost = 1.69 times.

NHAT2/3 - 89 vs. 136 sec., speed boost = 1.53 times.

NHAT3/3 - 83 vs. 132 sec., speed boost = 1.59 times.

My rig's specification is: Phenom X3 8450 / 2.1 GHz, 4 GB DDR2, Radeon HD 4770 / 512 MB with Catalyst 10.12 (AI is off), Windows XP SP2.

As you can see, the compressed (RxGB) normals are significally improving the time of a mission loading, especially on a relatively weak CPUs like mine.

Moreover, I've found about 400 TGA files that are not normals. Why are they still not compressed into DDS? Some of them are height maps (*_h.tga) and alpha-channels - these ones could be converted into one-channel grayscale DDS, which could reduce their size by three times. All others could be converted directly into DXT1/DXT3/DXT5. This could save the time of mission loading even more.

 

Third. I've tried to implement in TDM more advanced normal map compression, DXT5nm, instead of RxGB, but I've failed. Can your shader gurus tell me, what did I wrong? I've modified three vfp-files, as suggested there, in order to replace RxGB to DXT5nm, but with no luck .

Edited January 20, 2011 by MoroseTroll

[STiFU]

Serpentine is still working on a script to sort all the texture issues, including wrong dataformats, wrong sizes, alpha channel where no alpha is necessary et cetera. It is very complex.

 

But this is some interesting stuff. I am always in favor of speeding up loadtimes. I am gonna tell JC Denton about this thread. Maybe he has some more in-depth information about implementing DXT5nm into D3.

 

You were talking about one-channel grayscale DDS. I am interested in that, because they could be used to store an AO-pass for a model that shares generic textures for example.

[MoroseTroll]

Well, I can provide my scripts, if you, he or anybody else is interested to take a look at them - here. Make sure that you've installed 7-Zip (in order to repack the PK4 archives) and ATI Compressonator 1.21 (in order to convert the normals into RxGB). Extract the content of this archive into "darkmod" folder (it'd be wise to make a backup of it) and run the "RunMe.bat" file. Be warned that my scripts are using a substituted Z: drive, so if you are already using some device mounted to this letter, please modify the "RunMe.bat" file and change the line "set %vd%=z" with another (not Z) letter.

In my case it took me about two hours to perform the first phase. The second phase is... a hand work . That's right - ATI Compressonator 1.21, apparently, has some bugs, which prevent it to convert some TGA in the batch mode. So, if you're brave enough, all the time of this conversion you'd click several dozen times on Windows error message caused by the faulty ATI Compressonator's Boxfilter.dll. As such, several dozens of normals won't be converted in RxGB form, so you'd made this by yourself, in the interactive mode of the ATI Compressonator.

Edited January 19, 2011 by MoroseTroll

[STiFU]

But what's that grayscale dds format you were talking about? I never found anything like that and it would be useful in quite a few cases. For example, many specularmaps only use grayscale anyway, so we could detect those and convert them to grayscale dxt.

[MoroseTroll]

As far as I can remember, I've learned from my analysis of Doom3.exe that it can use so named GL_ALPHA8_EXT (0x803C) and GL_INDEX8_EXT (80E5) formats. I suppose, in DirectX, these formats should be named as Alpha 8 / Grayscale 8 / Index 8 or even A8/G8/I8. I'm sure that the ATI Compressonator 1.21 does support A8/G8/I8 in the "Alpha compression" / "Grayscale compression" / "Palette compression" options.

BTW, A8/G8/I8 formats are not DXT-based, instead - they are index-based. But anyway, I hope they will be useful in order to reduce the size of grayscale/alpha images and thus - the time of mission loading as well.

Edited January 19, 2011 by MoroseTroll

[Serpentine]

Nah, using pure alpha/luma is a bad idea, it's something thats continually neglected by drivers (usable directly in shaders for the most part) and id made a point of having them disabled by default (there's a cvar), indexed palletes are a pain in the ass to work with and will result in headaches for portability and perhaps comparability since they are not required to be supported by drivers and hardly any (de)compression libs actually support them strictly. There's a lot of stuff you can do with DDS files, but few of them are really viable in the long run.

 

As for converting to DDS, my script is working quite well at detecting common mistakes and making the relevant choices. I however cant trust that everything is perfect and will need a good bit of time for testing. I'll move some stuff over in 1.04, and hopefully the remaining (and vast bulk) in 1.05. Also included in that is removing and remapping of duplicates and culling of extra channels from normalmaps and such.

 

It can also detect normalmaps quite well(and incredibly slowly ), so in the future it might be quite easy to convert everything to a sane format, without hoping that formats/names are correct... but until then I really don't think its worth doing.

[Tels]

As for converting to DDS, my script is working quite well at detecting common mistakes and making the relevant choices. I however cant trust that everything is perfect and will need a good bit of time for testing. I'll move some stuff over in 1.04, and hopefully the remaining (and vast bulk) in 1.05. Also included in that is removing and remapping of duplicates and culling of extra channels from normalmaps and such.

 

This isn't meant personal:

 

The past has shown us that the "perfect" is the enemy of "good", e.g. you spent ages perfecting your work, and in the meantime, the "good" result of f.i. converting 80% never happens - because the perfect state is never reached.

 

So, since the "we convert and fix them eventually" has now been moved to "next months" a couple of times, why not start with what you have? Converting/fixed the files it already can would already be an improvement. We can deal with whatever it can't do right now later, or?

[New Horizon]

This isn't meant personal:

 

The past has shown us that the "perfect" is the enemy of "good", e.g. you spent ages perfecting your work, and in the meantime, the "good" result of f.i. converting 80% never happens - because the perfect state is never reached.

 

So, since the "we convert and fix them eventually" has now been moved to "next months" a couple of times, why not start with what you have? Converting/fixed the files it already can would already be an improvement. We can deal with whatever it can't do right now later, or?

 

I don't think Serpentine is referring to achieving absolute perfection with the textures, it's more about the script not doing everything 'perfectly', which is a bit different. I think we can trust Serp's judgement.

[Tels]

I don't think Serpentine is referring to achieving absolute perfection with the textures, it's more about the script not doing everything 'perfectly', which is a bit different. I think we can trust Serp's judgement.

 

My worry is just Serp will run out of time or motivation (or even die) before the script is done (and so far I don't think anyone else has even seen the script). I won't complain no matter what the result is, but it would be a shame if we'd never get any result at all.