DDS Texture discussion

[OrbWeaver]

DDS for normalmaps doesn't work for everybody though, that is the problem. We could only use this if we were willing to drop support for all pre-DX9 cards.

 

They also tend to look ghastly -- just examine some of the floor textures in Deadly Shadows at an acute lighting angle.

[SneaksieDave]

DDS for normalmaps doesn't work for everybody though, that is the problem. We could only use this if we were willing to drop support for all pre-DX9 cards.

Confirmed, even with my new card - no DDS normalmaps.

[New Horizon]

Confirmed, even with my new card - no DDS normalmaps.

 

Hmmm, this might the way around that problem.

 

image_useNormalCompression "2" "Default Setting"

 

2 = use rxgb compression for normal maps,

1 = use 256 color compression for normal maps if available

 

I'm guessing that using 1, would allow the use of standard normal maps, instead of the doom 3 specific rxgb normal maps. Not sure though, but I'll run some tests.

[OrbWeaver]

I'm guessing that using 1, would allow the use of standard normal maps, instead of the doom 3 specific rxgb normal maps. Not sure though, but I'll run some tests.

 

AFAIK option 1 ONLY works for pre-DX9 cards, and option 2 ONLY works for DX9 and above -- you cannot choose one or the other, you have to choose the right one for your card.

 

Also, even if that option was available, throwing away the RxGB compression reduces the quality down to TDS-equivalent (I don't know how much of a quality improvement RxGB actually supplies, but one thing's for sure: TDS normal maps looked like crap).

[Gildoran]

AFAIK option 1 ONLY works for pre-DX9 cards, and option 2 ONLY works for DX9 and above -- you cannot choose one or the other, you have to choose the right one for your card.

I second this.

 

Also, even if that option was available, throwing away the RxGB compression reduces the quality down to TDS-equivalent (I don't know how much of a quality improvement RxGB actually supplies, but one thing's for sure: TDS normal maps looked like crap).

I disagree with this statement. Without a doubt TDS's normalmaps were made ugly by compression, but that isn't the case in D3. Although DXT1 is known to create artifacts in normalmaps, DXT5 usually looks better by compressing the images less. All D3 normalmap DDS files are stored as RXGB DXT5. Though there may be some cases where DXT5 normalmaps aren't preferable, I typically don't notice the difference in quality between the DXT5 normalmaps and the original uncompressed versions. (the one exception I've seen is the window testmap I've made) Usually a DXT5 DDS looks better than a TGA of the same file size. And in the few cases where it doesn't, there are per-stage material keywords to avoid compression.

 

By far, currently the biggest cause of ugly TDM textures isn't DDS files, it's people not comprehending how to properly use normalmaps. Many people bake lighting into the diffuesmap and many use bad judgment about the intensity of the grain versus the overall shape of the normalmap.

[New Horizon]

(I don't know how much of a quality improvement RxGB actually supplies, but one thing's for sure: TDS normal maps looked like crap).

 

True, they did look like crap. From my discussions with Sledge though, they looked that way because they were actually compressed far beyond the norm to aid in performance. The resolution of a majority of the textures were also cut down from the originals. John P's replacement textures look great in comparison.

 

If option 1 were to work on dx9 cards as well as older cards, I don't think it would be a bad idea to use them. We could then get away from needing to use the compressonator, and folks would be able to use the tools of their choice.

 

The fixation on better and better compression can sometimes get in the way of practicality. Compared to the old games, and TDS...TDM is going to look fantastic, even if we were able to use purely DDS, and I really wish we could...because this TGA nonsense is such a waste of space and resources. It's just so illogical to be loading up 3 or 4 meg textures, when a 1 meg or less .dds of the same resolution will look just as nice.

 

I'll do some searching around and see what I can find out. Perhaps we will be able to find a way around these limitations.

[New Horizon]

For one thing, DXT1 is known to create artifacts in normalmaps, so DXT5 was created with normalmaps in mind, and looks much better.

 

True, and I think TDS often used only dxt3 for the normal maps. I could be wrong though.

[Gildoran]

After writing that about DXT5 being designed with normalmaps in mind, I realized I had misread wikipedia - it was talking about 3Dc (a variant of DXT5) being designed with normalmaps in mind. However, I stand by my claim that RXGB DXT5 normalmaps look just as good as uncompressed images in the vast majority of situations; with the textures I've made, the compression usually isn't noticable, and is worth it given the higher resolutions that can be used.

[OrbWeaver]

Although DXT1 is known to create artifacts in normalmaps, DXT5 usually looks better by compressing the images less.

 

That's what I don't understand. According to the Wikipedia article ( http://en.wikipedia.org/wiki/Texture_compression ), DXT5 encodes colour information the same way as DXT3, which in turn encodes colour information the same way as DXT1. The only difference is the way alpha information is handled, which is irrelevant in the case of normal maps.

 

Of course, it may be that the Wikipedia article is missing some important detail, but if it is correct I can't see how DXT5-compressed normal maps would look any different from those using DXT1.

 

Has anyone investigate the possibility of using JPG? I have a vague notion that Doom 3 will load JPEG images, and a high-quality JPEG can look as good as uncompressed at a fraction of the size.

[Gildoran]

As for the wikipedia article, see my post above.

 

My understanding is that the problem with JPGs is that D3 has to decompress them. They don't use much disk space, but it takes D3 more time to load them (because of decompression), and they take up the same amount of space in memory as a TGA, so they don't give the benefit of being able to use more/higher-res textures.

 

JPGs might be fine for cases where DDS compression looks bad (or to replace the TGA normalmaps that we currently have in case the graphics card can't read RXGB DXT5), but I think we should opt to use DDS by default. (however, I have more recently been using JPGs for editor images, since DDS files can't be used for them and at least a JPG will save some disk space unlike a TGA)

[New Horizon]

@orb

 

We tested jpg long ago, it was a disaster....there were tons of visible artifacts.

 

The best solution would be DDS, the big if is whether or not there is a way to get them working on a wider range of systems.

 

As for DXT1 vs. DXT 5, I'm guessing DXT5 just compresses the image less than a DXT1 or 3. DXT5's are larger from my experience with them.

[Gildoran]

To illustrate why I think we should go with DDS rather than TGAs, and to try to dispell the notion that DDS compresses things badly, I'd like to present the following example of TGAs/DDSs with equivalent file sizes:

 

For this screenshot, I used TGAs (with no RLE compression) as the source images, turned off all compression and restarted D3.

The diffusemap is 769K (512x512) and the normalmap is 1537K (512x1024):

 

For this screenshot, I used DDSs as the source images, turned on all compression and restarted D3.

The diffusemap is 683K (1024x1024) and the normalmap is 1366K (1024x1024):

 

Despite the DDS files having slightly smaller file-sizes, they're clearly higher quality and the artifacts are not significantly noticable. (in fact, does anybody here even notice them?) IMHO, for a given file-size, you get more bang for your buck using DDS.

 

Note: Though the JPGs have different filesizes, I saved them both with exactly the same quality settings, and the difference in visual quality was also shown by the original TGA screenshots.

[OrbWeaver]

As for DXT1 vs. DXT 5, I'm guessing DXT5 just compresses the image less than a DXT1 or 3. DXT5's are larger from my experience with them.

 

That's what I originally thought, but all the evidence I have seen points to DXT1 and DXT5 compressing the colour data in exactly the same way by the exact same amount, it is just the alpha information which is stored differently. I would expect DXT5 to be larger because of the additional alpha information, which is not used for normal maps (we would be better off using DXT1 in fact).

 

Despite the DDS files having slightly smaller file-sizes, they're clearly higher quality and the artifacts are not significantly noticable. (in fact, does anybody here even notice them?) IMHO, for a given file-size, you get more bang for your buck using DDS.

 

That seems reasonable -- DDS at a higher resolution is better than TGA at a lower resolution.

[Gildoran]

That's what I originally thought, but all the evidence I have seen points to DXT1 and DXT5 compressing the colour data in exactly the same way by the exact same amount, it is just the alpha information which is stored differently. I would expect DXT5 to be larger because of the additional alpha information, which is not used for normal maps (we would be better off using DXT1 in fact).

I was under the impression that the reason D3 went with RXGB, was to somehow store things in the alpha channel to improve the quality of the normalmap... Unfortunately, I don't know what RXGB means (I assume it's some sort of twizzling), so I don't know how true that is.

 

In any case, I don't think D3 is capable of using DXT1 for normalmaps. (though admittedly I haven't tried)

 

That seems reasonable -- DDS at a higher resolution is better than TGA at a lower resolution.

That's exactly my point - with DDS files, we can go with higher resolutions, which usually outweighs the problem of artifacts.

[SneaksieDave]

How do the pre-DX9 cards fit into this? I still haven't seen them work on such (or have I in Doom3?). Is mode 1 somehow shitty, or just a different path to the same result for older cards?

 

Any normalmaps I tried compressing with DXT1 failed, although that may have been my old card, and not Doom3, so I'm not sure either.

[OrbWeaver]

I was under the impression that the reason D3 went with RXGB, was to somehow store things in the alpha channel to improve the quality of the normalmap... Unfortunately, I don't know what RXGB means (I assume it's some sort of twizzling), so I don't know how true that is.

 

Yes, I was referring to standard DXT5 not RxGB. My understanding of the latter is that it swaps the red and alpha components, so that red gets the full 64 bits of alpha data rather than squeezing into the other 64 bits which it has to share with blue and green. I don't know what sort of quality improvement this gives; I guess it improves the blockiness in at least one dimension.

 

How do the pre-DX9 cards fit into this? I still haven't seen them work on such (or have I in Doom3?). Is mode 1 somehow shitty, or just a different path to the same result for older cards?

 

I am unsure of exactly what the "256-colour compression" means, whether it refers to standard DDS or something else (perhaps it means 8-bit palettized or something). Nevertheless, I don't think we have a solution to the "one type of card or the other" problem, which makes everything else moot.

[New Horizon]

We really need to make a serious decision about this soon. As Bittoman said to me on doom3world....

"And one last thing, you can't expect everything to work on DX8 cards when the game's requirement is DX9 minimum"

 

- edit:

 

The above was not entirely correct...doom 3 requires a dx8 card to run...dx9 only does the heat haze stuff, and likely some other fluff.

 

By the time we release, it really shouldn't be that big of an issue...and if it is, a low end card to run doom 3 shouldn't cost that much. Considering I paid $140 for a 6800..a more powerful card will be available for a similar price by then.

 

It would just be nice to push forward with this and get our texture repository scaled down to something reasonable. Maintaining duplicate texture repositories in tga and dds is silly.

 

Happy Friar posted some screens that were done on a dx8.1 card, all with dds normals working correctly. I don't know if I'm following the issue fully...but on doom3world, someone suggested that this could simply be a problem with the compression of the file itself, and that anytime they encountered this issue...compressing the file again usually fixed the issue.

 

I have no idea. It would just be nice to simplify things.

[Domarius]

With that firey discussion in the off-topic forums about hanging onto the T2 engine, I'm surprised there's a debate about this

 

I went out and bought a cheap DX9 compatible card just to play T3, since it flatly refused to run on anything less.

 

The release date of this game is only going to become later and later, not earlier, so our chances of people not caring that it won't work with DX8 is going to only become less as time goes on.

[sparhawk]

Maintaining duplicate texture repositories in tga and dds is silly.

 

Why? I thought the purpose was that we have a high-res repository, that people can use if they want to, and we release with the DDS scaled down textures? Also, wouldn't texture artists create a TGA version anyway? Or do they create DDS directly? I thought that DDS can be created only with a converter.

[OrbWeaver]

Yeah, surely we're not talking about ditching the TGA repository completely?

 

I thought the intention was to use DDS for releasing the mod to keep the download size down. If we ditch the TGA "master copies" of the textures, then it is impossible to edit existing textures without recompressing them, which is a very bad idea.

[Gildoran]

I second what Sparhawk and Orbweaver have just said. I'm very much against deleting the original TGAs, even if I'm not necessarily for keeping them in the main TDM repository.

[New Horizon]

What? I'm not talking about the hi-res repository, I'm talking about our mod repository and storing TGA and DDS versions of every single texture within the mod. I guess I should have phrased it more clearly. I meant it's silly to be storing both TGA and DDS in the same mod repository. The hi-res repository is a different matter completely and will remain. Counting the hi-res repository, we're actually keeping 3 versions of the textures. Hi-res, a medium set of TGA's and DDS. It's a bit much. Given what we've seen with the quality we would be able to keep with hi-res DDS files, it's really quite redundant to bo be keeping middle of the road TGA files and the hi-res. The middle road should go. DDS for the main release and TGA for the ultra hi-res upgrade at a later date.

[SneaksieDave]

Where are we with getting DDS working for all types of images on DX8 cards, any findings yet?

[New Horizon]

Where are we with getting DDS working for all types of images on DX8 cards, any findings yet?

 

Haven't really gotten anywhere new, other than the guys over at d3world telling me that they should already work. We're stuck with the proprietary format iD used for dds normal maps. The doom 3 world guys posted some screens on a DX8 card with the normals working. Their hunch is that it's an issue with some setting when our normal maps are being compressed. Another guy says that doom 3 requires dx9 minimum and someone else said dx8...my box clearly says dx 9.0b...so...I don't know. Basically, the only things affected by a card being dx8 should be some heat haze effects, as the dds normals should work normally, since they're dealt with by the game engine I guess.

 

Some things might work on a dx8 card, but if the engine requires a dx9 card to work fully...I feel we should really be focusing on that and not trying to support cards that will likely be fairly rare by the time we release. It shouldn't be hard to find a cheap card to run d3 when we release the mod.

 

I'm just not sure if tearing our hair out is going to be worth it in the end. I would love to see this work on older cards, but is it worth it?

[sparhawk]

What? I'm not talking about the hi-res repository, I'm talking about our mod repository and storing TGA and DDS versions of every single texture within the mod.

 

OK. That was not so clear to me. I agree that it doesn't make sense to have the textures duplicated in DDS and TGA format.

 

Counting the hi-res repository, we're actually keeping 3 versions of the textures. Hi-res, a medium set of TGA's and DDS. It's a bit much.

 

Can the medium res be derived from the hires? In that case it would be OK to scrap it. Otherwise we should probably put the TGAs also in a seperate repository.