DDS Texture discussion

[OrbWeaver]

I just found out that the compressonator texture I tested with today (linked above) doesn't give me normalmaps in DarkRadiant.

 

Almost certainly. I would be astonished if DarkRadiant supported RxGB normal maps when almost nothing else does.

 

Say the word if we need a defect entry. I'm getting so confused with everything this way and that that I'm wanting to just go to bed.

 

I'm kind of in too minds about this -- on the one hand, if we are using all-DDS (including RxGB) we would need support for it, on the other hand, all-DDS is really a hack and there was never any realistic expectation that the RxGB DDS files would need to be read by anything other than Doom 3.

 

Does the library do operating system calls on it's own? Like file IO or such? If not, then it shouldn't be a problem to run it on Linux as well with a wrapper.

 

I haven't tested it. I wouldn't want to go down that road though, it's too much of a kludge and doesn't solve the licensing issues either.

[SneaksieDave]

I'm kind of in too minds about this -- on the one hand, if we are using all-DDS (including RxGB) we would need support for it, on the other hand, all-DDS is really a hack and there was never any realistic expectation that the RxGB DDS files would need to be read by anything other than Doom 3.

Okay, I guess we've got that on the table now too, then. Hm.

 

Case not close.

[OrbWeaver]

I have been able to determine that RXGB is identified as such in the DDS file, which is why other applications cannot open them.

 

$ ddsinfo.py cobble_004.dds
DDS file, 512x512
	131072 bytes
	10 mipmaps
	DXT1

$ ddsinfo.py cobble_004_local.dds
DDS file, 512x512
	262144 bytes
	10 mipmaps
	RXGB

 

It may therefore be possible to add support for them at some stage, provided that the encoding can be worked out (if it hasn't already). It would also be theoretically possible to add support for the format to the GIMP DDS plugin, since it is open source.

[New Horizon]

Okay, I guess we've got that on the table now too, then. Hm.

 

Case not close.

 

If we're going to be offering Dark Radiant as a reliable alternative to DoomED, we should try to read those suckers. I hope it's possible...otherwise we're stuck with yet another damned issue against DDS. I'm not sure who to curse..iD or ATI for this non-standard DDS crap.

 

Is all of the code for what type of DDS files are used, in the doom 3 engine, hidden? Perhaps there is some aspect of that in the SDK that could be switched to allow the use of standard compression DDS files? The doom 3 engine can read standard DDS, we would just need to be able to tell it to use regular, instead of unleaded.

[SneaksieDave]

I'm not sure who to curse..iD or ATI for this non-standard DDS crap.

Yeah, I really cannot understand id's decision here. If they're going to go proprietary (which I can't believe they did in the first place), why go with the smaller name? If they hadn't included working versions of the textures with Doom3, they'd have excluded a huge portion of the audience - all those with <GF5 generation cards - from even seeing the game correctly. Makes no sense.

[OrbWeaver]

http://www.techspot.com/tweaks/doom3/doom-7.shtml

 

Robert Duffy provides some more details regarding normal map compression;

 

One thing of note on the normal map compression is that generally speaking if you DXT a normal map you get really crappy results. NVIDIA hardware supports palletized compression which yields good compression and normal maps retain hard and round edges really well. Unfortunately this compression does a poor job in other cases and you end up getting splotchy areas. ATI does not support the palletized compression so we needed a better solution. ATI had done some research on various methods of normal map compression and we ended swapping the red and alpha (which is zero in the case of a normal map) channels. This effectively allows the compression to do a much better job and is just 1 extra instruction in the fragment program to move the alpha channel into the red channel. The bottom line on what happens on each card is as follows.

 

All modern NVIDIA and all ATI hardware use the compressed normal maps in Medium and Low qualities with the sizzled components. NV10/20 hardware (GF4MX and GF3) uses palletized normal maps in Medium and Low qualities.

 

So the bottom line is, RXGB is a quality improving technique that works on modern graphics cards, but the older Geforces only support P8-compressed normal maps (while modern cards don't).

[SneaksieDave]

Asked, and answered. Good thing they did research this in advance, or there would've been one monster sized patch shortly after release.

[New Horizon]

I really don't think the results of doing a standard DXT on a texture are that bad. John P's higher res textures in T3 look just fine. I just don't really buy into their need to do this.

[OrbWeaver]

It depends very much on the texture. Smooth normal maps at sharp lighting angles can look very poor with DXT compression (above is uncompressed, below is DXT5):

 

 

Obviously the motivation at ID was to maximise image quality, which was reasonable since the game was released on a DVD and not constrained by network bandwidth.

[SneaksieDave]

Hey! That bottom one reminds me of TDS.

[OrbWeaver]

Hey! That bottom one reminds me of TDS.

 

Exactly.

[New Horizon]

But then they just opened up a whole new can of worms. Oh well.

 

So what's going to happen here folks? I'm somewhat disillusioned by the complexity of all this. To be honest, I don't think we should be worried about supporting these older cards right now, we should just move ahead with using the d3 dds formats, for our own sanity and simplicity. The hi-res repository will be maintained, so our original high quality materials will still be available.

 

What we need to keep in mind also, is that when this sucker goes open source...steps can be made to allow it to run more easily on older setups...there will be better optimizations. If a vast majority of folks are still using cards that won't support this format, that issue can be seen to then. Trying to see to everyone immediately creates more complexity and just adds a hell of a lot more overhead. We just don't have access to the resources within the engine to do that right now. To be honest, I will be very surprised if people don't have a video card to run this by Christmas 2008...our tentative 1.0 release date.

[OrbWeaver]

So what's going to happen here folks? I'm somewhat disillusioned by the complexity of all this.

 

I don't think it's all that complex; there are really two options that I can see:

 

1. Go ahead with RxGB/DDS, and accept the increased difficulty of manipulating/processing/handling/viewing such files due to the non-standard format.

2. Stick with TGA, and accept the increased bandwidth and storage requirements of the uncompressed files.

 

I guess the characteristics of the new Subversion repository will have a bearing on this decision, as and when it gets set up, since if there is more bandwidth available the size may not be such an issue.

[New Horizon]

I don't think it's all that complex; there are really two options that I can see:

 

1. Go ahead with RxGB/DDS, and accept the increased difficulty of manipulating/processing/handling/viewing such files due to the non-standard format.

2. Stick with TGA, and accept the increased bandwidth and storage requirements of the uncompressed files.

 

I guess the characteristics of the new Subversion repository will have a bearing on this decision, as and when it gets set up, since if there is more bandwidth available the size may not be such an issue.

 

I would be inclined to go with number 1. It's not just the storage requirements, it's also the performance as well. TGA's are huge dogs, and jpeg isn't an option...since it's far worse than dds. Then if we try to minimize the impact of TGA's, we have to reduce the resolution to half in order to have some reasonable file sizes. As far as creating the rxgb dds files, I am pretty sure the doom engine can do that for us...I just haven't been able to figure out exactly how it works yet.

[OrbWeaver]

Well it should only affect the load performance, not the rendering performance, since once the texture is sent the graphics card it doesn't matter how it was stored on disk. However I tend to agree that using RxGB would be the better option for now, since everybody (I think) has a DX9-compatible card and the hi-res repositor will still be available for those who want it.

 

Also, for final distribution it would be trivial to create two archives -- one with the TGAs and one without, so people could choose according to their bandwidth needs and desired quality.

[New Horizon]

Also, for final distribution it would be trivial to create two archives -- one with the TGAs and one without, so people could choose according to their bandwidth needs and desired quality.

 

Word

[SneaksieDave]

So let's summarize, to make sure all points are considered at the same time.

 

DDS

Pro:

-one format

-less space

-high res

-can still have separate repository for TGA

-Mip Maps (depending on how far away you are from the texture, it scales to smaller resolutions, based on distance)

-unlike RLE compressed TGA's, DDS files do not expand at load time. A 1.3 meg DDS remains 1.3 megs.

 

Con:

-lack of DR support (currently)

-lack of support on older cards

-some difficulty handling the files (e.g., can't simply open in photoshop)

 

TGA

Pro:

-ease of use

-already supported in DR

-works on all video cards

 

Con:

-messy repository of patchwork files (sizes, some DDR, some TGA, etc)

-more space

-lower res (or even more space)

-RLE compressed TGA's are not supported natively by video cards, must be expanded at load time.

 

Anything else to consider?

 

-----

My biggest gripe with DDS right now is the need for this damn compressonator thing, yet another program I don't want on my system, and the difficulty in use, as compared to simply opening a TGA in PS6. DarkRadiant support is also vital. The space savings would be more than welcome though. Textures (geometry and characters) are currently more than a gig of space, which is obscene, considering how little we actually have.

 

-EDITED

 

New Horizon added a few exta Pros for DDS...Mip Maps and 'non-expansion at load time'

[New Horizon]

-----

My biggest gripe with DDS right now is the need for this damn compressonator thing, yet another program I don't want on my system, and the difficulty in use, as compared to simply opening a TGA in PS6. DarkRadiant support is also vital. The space savings would be more than welcome though. Textures (geometry and characters) are currently more than a gig of space, which is obscene, considering how little we actually have.

 

Looking into a command for doom 3 called image_writeprecompressedtextures. Not sure how the heck it works really, I guess you just set it to 1 and open a map textured completely with TGA's in a setting that only uses DDS, low for example, and I think the game will write them up automatically and store them for later use. I think.

 

Compressonator isn't so bad really. Just do all your TGA work in photoshop, and then when you're finished working on a texture set..do a batch process on your normals.

[greebo]

New Horizon, is the Wiki article concerning the Texture Guidelines still accurate or does it needs updating (regarding the revised Texture Guidelines sticky thread)?

[New Horizon]

New Horizon, is the Wiki article concerning the Texture Guidelines still accurate or does it needs updating (regarding the revised Texture Guidelines sticky thread)?

 

I'm going to hold off on the wiki update, just until we've settled on what we're doing 100%.

[Domarius]

Well it should only affect the load performance, not the rendering performance, since once the texture is sent the graphics card it doesn't matter how it was stored on disk.

That sending process is responsible for sudden jerks in the frame rate as you walk down a new area that has textures you haven't seen very recently. If that problem can be reduced by an amount that is proportionate to the difference in disk space usage, then the performance increase should be quite noticible.

[SneaksieDave]

A lot of performance assumptions are being bandied about, but I'd venture the guess that loading 4Mb is probably around the same speed (if not faster) than loading 1.3Mb followed by decompressing it. As for game-time, it's probably the same, assuming the game fully prepares the textures - whatever the format - in advance of needing them.

 

So we may see no game performance change at all, and I wouldn't be suprised if loading slowed down. Remember how much longer it took to load a TDS map with everything packed as opposed to after extracting everything?

[New Horizon]

That sending process is responsible for sudden jerks in the frame rate as you walk down a new area that has textures you haven't seen very recently. If that problem can be reduced by an amount that is proportionate to the difference in disk space usage, then the performance increase should be quite noticible.

 

It was extremely noticeable when I did my original tests back in the day. When I dropped the original TGA's from 1024 down to 512, the game went from choking to very smooth. David (sledge) advised me very early on that we should really try to avoid such bloated textures....lest we wished to see some jerky moments. When I decreased the player arm textures from 1024 to 512 locally, the big pause that plagued everyone went away on my system. Huge 4 meg textures are just ridiculous. lol

[SneaksieDave]

Are they not expanded into the same thing once in virtual memory, thus making realtime handling of them the same? Anyone know for sure?

[New Horizon]

A lot of performance assumptions are being bandied about, but I'd venture the guess that loading 4Mb is probably around the same speed (if not faster) than loading 1.3Mb followed by decompressing it. As for game-time, it's probably the same, assuming the game fully prepares the textures - whatever the format - in advance of needing them.

 

So we may see no game performance change at all, and I wouldn't be suprised if loading slowed down. Remember how much longer it took to load a TDS map with everything packed as opposed to after extracting everything?

 

DDS textures aren't decompressed though, they're supported by the video card and are read 'as is'. When I was doing my earlier research on RLE compressed TGA's, I missed a very important bit of information....RLE compressed TGA's are not supported by video cards, so they are expanded at load time.

 

TDS was just very poorly optimized, that's the main reason why it was so slow. They had to come up with the work around of packing things into the map to eliminate the load times, sledge told me it was because the engine had to search for the files...or something to that nature...packing them in the map reduced the searching. Doom 3 is far better optimized.