DDS Texture discussion

[greebo]

I noticed that you copied the tga's to the hires, and I'm not really happy at the moment. The repository is already pretty big and now we have duplicated the tga's into two repositories. I don't know if it would be possible to delete the history of the files while retaining the history itself. this would shrink the main mod repository quite a lot.

 

You're referring to the repository size on the server or the actual size of one checkout?

 

It's true that the hires repository is now 400 MB larger than before - I didn't know any other way to move them from the main into the hires repository.

[sparhawk]

The server size. For clients, the repository will be smaller then before.

[OrbWeaver]

The size issue is an unavoidable consequence of large binary files + version control.

[sparhawk]

Yeah, I know, but I didn't really think that such binary files will be copied over and over. I must look if I can add an additional harddisk, because I already ran out of discspace once. After all, I use it for other stuff as well, not only for the mod.

[Springheel]

Hmm, I just noticed that the "tiling_1d" and "nontiling" extensions were added *after* the "_local" and "_s" extensions in the filenames. A .dds normalmap now looks like "rough_dark_flagstone_local_tiling_1

d.dds". Is that going to cause any headaches?

[greebo]

It caused a tiny headache while I was selecting all the "_local.tga" for the batch compression, but it wasn't too bad after all. We can of course change it with another sweep of the script, but I feel my bones getting weak about the thought of this. If it's really necessary, we can of course do it.

[Springheel]

There shouldn't be too many of them, so let's just leave it.

 

BTW, I'm not entirely sure what changes were made to the folder structure...can you update that part of the wiki page?

[greebo]

Only the nontiling and tiling_1d sections were merged. I already updated the folder structure in the article (the stone/flat/smooth, door/wood/ornate and the skybox folders were too deep).

[greebo]

I just got the automated mail notification from Adobe that I can download the Photoshop SDK. So far, so good, I though, but in the E-Mail I got from the Product Manager eight hours later, I'm being told that the file format plugin code is not available in this package, so I have to request the Advanced SDK. I shall fill out that form he attached to the E-Mail, which appears to be an 11 page license agreement, and fax to some number (they don't even bother telling me the country this number is located in). According to the product manager, this process might take up to three weaks.

 

WTF? I mean, WTF? I'm not going to sell my ass by signing this agreement just to take a look at the damned SDK. It's not that it could hurt Adobe if I wrote another import plugin so that even more users can work with Photoshop. Seriously, how arrogant can you get?

[OrbWeaver]

Welcome to the mindset of large software companies. They seem to believe that being able to produce free add-ons for their commercial software is an enormous privilege that you should have to beg for (and possibly pay as well).

 

You'd probably be better off writing tga2dds/dds2tga conversion utilities, or perhaps improving the open-source GIMP DDS plugin to load RXGB.

[New Horizon]

I just got the automated mail notification from Adobe that I can download the Photoshop SDK. So far, so good, I though, but in the E-Mail I got from the Product Manager eight hours later, I'm being told that the file format plugin code is not available in this package, so I have to request the Advanced SDK. I shall fill out that form he attached to the E-Mail, which appears to be an 11 page license agreement, and fax to some number (they don't even bother telling me the country this number is located in). According to the product manager, this process might take up to three weaks.

 

WTF? I mean, WTF? I'm not going to sell my ass by signing this agreement just to take a look at the damned SDK. It's not that it could hurt Adobe if I wrote another import plugin so that even more users can work with Photoshop. Seriously, how arrogant can you get?

 

What info do they need? I'll gladly do the deed and fax whatever info. I would really like to see something put together for Adobe.

[greebo]

@New Horizon: I can forward you the PDF they sent, although if you faxed the info, this would be inconsistent with my E-Mail address, as I used my Austrian gmx spam account. I don't know if you want to go through the hassle, but if yes, I can tell you where to subscribe to get the "regular" SDK: http://www.adobe.com/devnet/photoshop/sdk/

 

You need to create an "Adobe ID" to get access to the regular SDK, once that is done, you can request the regular SDK here: http://partners.adobe.com/public/developer.../sdk_request.do

You need to specify what type of plugin you want to develop, so it's required that you mention a file import/export filter, otherwise they won't tell you about the Advanced SDK, I suppose.

 

You should get another E-Mail within a few days from "Brian" sending you the PDF needed to get access to the "Advanced" SDK. They basically want your name/company, and your address (marketing contact/engineering contact) and so forth. I haven't even read in detail through all the paragraphs, so I don't know if we're even allowed to redistribute the plug-in (but I suppose so, because that's what nVidia probably went through as well).

[New Horizon]

@New Horizon: I can forward you the PDF they sent, although if you faxed the info, this would be inconsistent with my E-Mail address, as I used my Austrian gmx spam account. I don't know if you want to go through the hassle, but if yes, I can tell you where to subscribe to get the "regular" SDK: http://www.adobe.com/devnet/photoshop/sdk/

 

You need to create an "Adobe ID" to get access to the regular SDK, once that is done, you can request the regular SDK here: http://partners.adobe.com/public/developer.../sdk_request.do

You need to specify what type of plugin you want to develop, so it's required that you mention a file import/export filter, otherwise they won't tell you about the Advanced SDK, I suppose.

 

You should get another E-Mail within a few days from "Brian" sending you the PDF needed to get access to the "Advanced" SDK. They basically want your name/company, and your address (marketing contact/engineering contact) and so forth. I haven't even read in detail through all the paragraphs, so I don't know if we're even allowed to redistribute the plug-in (but I suppose so, because that's what nVidia probably went through as well).

 

Yeah, I was looking around for some info and apparently this is the simplified version of what they used to do. I think they used to require folks to be paid members of some Adobe 'club'. Heh. Apparently a lot of the info is outdated, so the PDF is likely just a left over formality. Seems pretty lame. I've signed up for the regular SDK...I'll go about seeing if I can get the advanced one too.

[Nyarlathotep]

Personally, I'm with Orbweaver on this one. Fuck Adobe. Let's just create a plug-in for the GIMP, and maybe a stand-alone conversion utility.

[New Horizon]

Personally, I'm with Orbweaver on this one. Fuck Adobe. Let's just create a plug-in for the GIMP, and maybe a stand-alone conversion utility.

 

I personally don't like the gimp very much, but I would love to see a plugin for it too. Many folks will be using Adobe, so it doesn't hurt to look into it. I'll get my hands on the SDK, and we'll move forward from there.

[greebo]

I rather put a disclaimer here, that I can't guarantee that I will accomplish anything useful with the SDK. I will certainly have a look at it, but I can't make any promises, since the SDK could turn out to be utter crap or it could be terribly time-consuming, who knows. If it takes too much time, I rather spend it on something else (more important).

 

Anyway, your efforts are appreciated, New Horizon. Let me know when they "approve" you.

[Springheel]

For our sculpted textures it probably makes sense to leave them as TGA, as this takes less space than the various combinations of baked sculpted+roughness images. *sigh*

 

This problem is causing some more headaches. I was just thinking about how we should better organize our model textures by moving anything that is just straight 'material' (like generic metal, generic stone, etc) to the texture folders, and only keep the textures that are specifically designed for objects in the models/textures folder. Right now there are tiling generic wood, stone and metal textures inside the model/textures folder, when really they could be used on brushes and should be in the textures folder (and may in fact be there already, duplicated). We could then avoid duplicating textures by using the 'addnormals' function to keep the normalmap from the model itself (basically what you did for the gargoyle). That would cut down on a lot of texture files.

 

But if .dds textures can't be used that way, however, it means anything in the regular texture folder is unusable--all the natural rock, flat metal, boards and other wood textures that would be useful for models would have to either be converted *back* to .tga, or duplicated in the models/textures folder.

 

There's also the messy problem of not being able to tell which textures are .dds and which are .tga from the editor, so if a modeler is looking at textures in the editor to choose which one he wants, he might pick a .dds one by mistake.

 

Come to think of it, it's only the normalmaps that suffer from this problem. I'm starting to wonder if we shouldn't just leave all the normalmaps as .tgas.

[angua]

This problem is causing some more headaches. I was just thinking about how we should better organize our model textures by moving anything that is just straight 'material' (like generic metal, generic stone, etc) to the texture folders, and only keep the textures that are specifically designed for objects in the models/textures folder. Right now there are tiling generic wood, stone and metal textures inside the model/textures folder, when really they could be used on brushes and should be in the textures folder (and may in fact be there already, duplicated). We could then avoid duplicating textures by using the 'addnormals' function to keep the normalmap from the model itself (basically what you did for the gargoyle). That would cut down on a lot of texture files.

This is a great idea, I've seen quite a few textures in the models folder that would really be useful.

 

But if .dds textures can't be used that way, however, it means anything in the regular texture folder is unusable--all the natural rock, flat metal, boards and other wood textures that would be useful for models would have to either be converted *back* to .tga, or duplicated in the models/textures folder.

 

There's also the messy problem of not being able to tell which textures are .dds and which are .tga from the editor, so if a modeler is looking at textures in the editor to choose which one he wants, he might pick a .dds one by mistake.

 

Come to think of it, it's only the normalmaps that suffer from this problem. I'm starting to wonder if we shouldn't just leave all the normalmaps as .tgas.

We've already readded those as tgas that are used with addnormals at the moment. It might really be useful to keep the locals as tga (at least those which are likely to be used with addnormals, such as the generic stone or metal textures.)

[Springheel]

This is a great idea, I've seen quite a few textures in the models folder that would really be useful.

 

It should also keep the models/texture folder a little more managable.

 

We've already readded those as tgas that are used with addnormals at the moment. It might really be useful to keep the locals as tga (at least those which are likely to be used with addnormals

 

Yes, I think we may want to do that. I'm not sure we should try and distringuish between textures that are likely to be used with addnormals and those that aren't, however. It's hard to anticipate what a modeller might want to use. Making all the normalmaps .tgas again is probably the most user-friendly option (though a short-term pita).

[Springheel]

So no other input on this issue? Greebo, does it make sense to you?

[greebo]

As most of the addnormals() issues are resolved in the current state, I'd vote for not reverting the DDS normals back to TGA.

 

All of the TGA normalmaps are available in the hi_res repository, so they can rather easily be retrieved using the SVN Repository Browser.

 

Moving the re-usable model textures out of the props/textures folder is a very good idea.

[Springheel]

All of the TGA normalmaps are available in the hi_res repository, so they can rather easily be retrieved using the SVN Repository Browser.

 

Here are my concerns about that, though as a modeler, mapper and the main force behind setting up the .dds folders, you will probably have greater insight into this than I do, so just tell me if I'm wrong.

 

Issue 1. Right now there are only a few textures being used with addnormals (personally I didn't know about it until you told me, and I assume others didn't as well). I'm hoping to make much more use of this in the future, however, so let's say I'm making a new model and I want to add a stone block texture to it using addnormals. I look through the editor and find the block texture I want. I track down the material name and enter it into my material. But it doesn't work. Why? Because it's a .dds file, though I had no way of knowing that while scrolling through the editor or material files. If I wasn't aware of the .dds issue, this could cause a lot of headaches.

 

Issue 2. Since I DO know about the issue, I'd next have to retrieve the right .tga file from the hires repository and transfer it to the main texture folder. Now we have two copies of the same texture, one .tga and one .dds. That seems like an unnecessary waste. I could of course delete the .dds normalmap when I transfer the .tga version, but beyond the added potential for confusion, that leads to....

 

Issue 3. Compressing normalmaps has to be done with a special plugin that not every modeler or texturer is going to have. The plugin that might compress a diffusemap properly will not necessarily work on a normalmap (or so I understand). This could cause problems for people who aren't aware of this, making their normalmaps not work. I expect, to try and avoid this problem, some people will be uploading their textures as .tgas in the main texture folder (especially if some model textures are now going to go there) hoping other people can convert them later. So when we find .tga normalmaps in the main texture folders, how do we know whether they are there because 1. the person uploading them didn't make .dds files yet, or 2. they're being used in a shader with addnormals? Checking to see if there is a .dds version wouldn't help, as shown in scenario 2.

 

Anyway, it just seems to me that keeping some normals as .tga and some as .dds is only going to cause confusion and make the texture folders harder to maintain. My vote is to be consistent and keep all normalmaps as .tgas. It seems like the only reason NOT to do this is to save space, but if we start using addnormals regularly, we'll probably save just as much space by not duplicating normals as we would be keeping some of them .dds.

 

 

(Btw, do we know for sure that .dds files work with OTHER special keywords in shaders, like blends?)

[greebo]

Well, I personally don't mind looking up the texture, but I'm of course aware of the DDS/addnormals issues, which others may be not. This could be approached by documenting these texture issues, but it's admittedly unintuitive.

 

The Compressonator can be used to compress both diffuse and normalmaps (I created a tutorial to do this on the wiki recently), so this is not an issue.

 

The other shader keywords seem to work fine, it's primarily the MapExpressions, as far as I know.

 

To be honest, I don't want to argue or decide in any direction. If you want me to reupload all the TGAs - I will of course do that. The idea of converting the textures to DDS was not my idea originally, so I will not strongly defend it. I just executed what I thought was the team's decision. The strongest advantages of DDS files are their smaller size and the reportedly decreased map loading time, whereas TGAs are mostly easy to use.

[Komag]

How much smaller are they? How much difference is the whole mod right now with DDS versus TGAs?

[OrbWeaver]

It is possible that addnormals and other stage modifications deliberately do not use the DDS source image for quality reasons -- you would not want to load already-compressed images, modify and combine them together and then re-compress them to upload in DXT format to the graphics card.