Cleaning up textures

[Serpentine]

Whoa there Nelly, we're off course and there are loads of bad assumptions going on. Also I realized how few light textures we use and doubt it'd be beneficial. I also didnt act on this as I saw there'd be some 'interesting' posts. Well that and I have an exam in a few hours However for the sake of clarity:

 

TGA is not hardware accelerated, it is read, the pixel data is then moved to an OpenGL internal format which is then rendered. This is why DDS is fast, it does not need to be fully parsed and is able to just be accepted as an already good internal format and used directly for random reads etc of mipmaps (we can assume that as doom3 uses the correct extension ). I was never suggesting moving to RLE all the TGAs or anything crazy like that, just the light ones which I was looking at anyway.

 

When doing texture loads the archive is queried for the file which is then deflated/extracted to memory and the pixel data is read from it into memory, think bitmap. The size of the compressed version has little influence. The deflation from the zip is negligible for the most part as the file size of the RLE version is smaller once in memory and would make your point moot already, however it is the step in which the file is then read for the raw pixel data is where the encoding comes into play as it's cheaper to expand RLE than read through raw, it also has a lower memory footprint for the working copy if for some reason left in memory once it's already become an internal format.

 

In summary : RLE in 'good' cases loads faster and uses less memory when doing so without any visible difference in quality. In bad cases it's pretty much the same/ever so slightly faster. (For fun you can convert your local copy and see the massive 2-3 seconds you'll save! )

 

DDS on light textures would be a bad idea since DXT encoding would result in loads of artifacts and... well I don't think it would try mip levels for them anyway. I don't think it even supports the relevant raw formats that you'd want to use.

 

TDM uses standard PKZip style zips, none any of the fancy deflation methods that later winzip extended zips or 7z etc LZMA are used for TDM releases.

By default we also don't compress textures on load as that's a bucket of worms not worth playing with.

 

Also humpf svn -rename is retarded as hell ;/

Edited October 25, 2010 by Serpentine

[Tels]

@Serpentine: I was more talking about the TGA files that I added lately because I don't have a working DDS encoder anymore. And about heightmaps, and normalmaps etc.

 

If someone finds a way to compress normalmaps (the few cases where they are used in map-expressions excluded) that would save tons of loading time.

[greebo]

STiFU ran some RLE tests some time ago, and there was no negative impact, rather the opposite, even if the positive load time effect was small.

 

http://modetwo.net/d...post__p__209136

 

My conclusion is that RLE only impacts stone-age systems, if at all, but we don't have a side-by-side comparison with such a system which proves that.

[STiFU]

I am currently tackling issue #2439.

 

Here is a comment I added:

I've been looking for a reason to learn Python anyway, so I am looking into this right now.

 

In base Doom 3, Lighttextures are always TGA and I bet ID did that for a reason, so I'll just keep them as they are. You also missed some of the specular maps in that list, which are supposed to be converted to DXT1 too.

 

My script also found a couple of other folders with improper TGA files: "editor", "fog", "lode", "test_rain", "test_transparency", "test_volumetric", "test_window", "water_source". The list is based on the SVN trunk of TDM, so I guess the "test*"-folders can be left out in the conversion, as well as "editor", "fog" and "lode". But what about "water_source"? Some of the textures in the VFS-folder "water_source" are dds, so I should convert all of them to dds too, right?

 

In addition to that, I also found a couple of normalmaps stored as dds:

dds/textures\darkmod\glass\lion_glass_local.dds

dds/textures\darkmod\glass\victorian01_local.dds

dds/textures\darkmod\glass\victorian02_local.dds

dds/textures\darkmod\glass\victorian_local.dds

 

Apparently no tga files are available for these, unless I oversaw them in the highres repository. Does anyone know anything about this? lion_glas_local.dds is also double the size of the diffuse map. This could possibly be corrected too.

 

I still have a question. My script detects whether a tga is stored with 32 bpp (Bits Per Pixel) or 24 bpp (and lower) and calls TheCompressionator.exe with the appropriate parameters for DXT5 compression, DXT1 respectively. This works all just fine and the script for managing all that SVN ADD and SVN REMOVE is also done. But I am not sure whether I have to add the alphatest keyword to all materials that used those tga files. Anyone?

[Serpentine]

Ehhh, I'd prefer you left that stuff alone for now.

 

My script checks if the alpha layer is actually being used for anything useful, and if any materials that use it have any alpha-related keywords. If they arnt used it will drop and convert as 24bpp DXT1(without explicit alpha). I'd prefer this to be a low loss operation rather than rechecking DDS' post-conversion.

 

Answering the question is hard as the use of alpha in a DXT is necessary for some alpha-related operations and depends on the material in question. Most of the time it should be fine but there's no hard and fast answer.

 

My checks are also a lot more wide spread and both look at the file structure for correctness and try to actually verify and load the contained pixel data. Modules to build editor images for complex materials which have blends and such is already in place but needs my finalization of a material storage system from my current PEG parser. I'll share it all in a few days once I can add swizzle support to my DDS code and get Melans textures out of the way.

 

As for the normals, my script moans about them already. They're of bad dimensions however and need the whole material carefully changed as you can not scale it 'nicely' to power-of-two dimensions. There is no tga that I can find either, but they seemed to only require minor renormalization, so I think it'll be ok to work with them if you re-norm them before and after changes. (Tho I'll most likely just convert them back to height-maps as they're easier to work with)

Edited November 29, 2010 by Serpentine

[STiFU]

Hehe, well then this was probably just a good python coding practice for me. A little said though, to let the work go to waste. By the way, I searched for tga files that don't contain the keyword "local", to retrieve my list of files. Just saying, because your list didn't include some of the files I listed.

[Serpentine]

Ah nah, it's all good man - I had just started down this road during my past exams, I'm in the process of getting everything onto the svn and visible so that conflicts dont occur. Sorry about that, but that said, should for some reason I vanish or whatever you're most likely more capable than I at continuing down that road

 

Are you using PIL for this or FreeImage etc? I have a few fixed up PIL modules which you might be interested in. Even if the DDS one is pythonic and terribly slow(single threaded... 30 minutes to crawl through the svn ), I felt keeping everything minimal dependency wise was a good idea.

 

However there was something which I didn't quite finish up on, and that's model conversion/inspection. If that tickles you it'd be a great help. Assimp includes PyAssimp which has most of the stuff you'd need, I wanted to write generators to output to (well formed) ASE/lwo. So that we can work towards a unified model department, as well as using it to pull in materials for checking in the material/texture part of things. Adventurous use could also support some auto-optimization, which assimp seems to handle very well if you can figure out which options are best.

 

The ASE stuff might need some investigation as implementations like Rich's export a lot of redundant stuff as well as spamming sub-meshes for split material groups rather than per-material groups.

 

Anyway... just an idea

[STiFU]

I tried Python Imaging Library at first, but it couldn't open some of the TGA files, so I ended up opening the files in binary mode and looking at byte 16 in order to retrieve the bpp... For converting dds I then simply called TheCompressonator, as I said above. Other than that, the only extern module I used was pySVN, for dealing with all that SVN stuff.

 

I am not quite sure I understand what that Model conversion / inspection script is supposed to do, though. Conversion between ASE and LWO? I would bet my ass, there are already scripts that do that.

[Serpentine]

I tried Python Imaging Library at first, but it couldn't open some of the TGA files, so I ended up opening the files in binary mode and looking at byte 16 in order to retrieve the bpp...

Ahhh, yeah I fixed that up, there was a bug in how it checked headers. If a TGA contained a comment/id tag (1 at byte 0) it assumed it was PCX image... and well yeah that's not going to work. I'll kick the fixed versions out when I throw my stuff onto the svn.

 

I am not quite sure I understand what that Model conversion / inspection script is supposed to do, though. Conversion between ASE and LWO? I would bet my ass, there are already scripts that do that.

Nah, I was thinking more on the lines of 'anything to ase/lwo', Assimp as a framework loads a number of nice 3D formats and by then using this pre-prepared data, write out sane ASE/lwo's. Assimp also deals with checking and correcting problems with models during the loading process (as well as optional optimization). Allowing this portability would be quite nice to unify existing assets as well as (possibly, in an ideal world) automatically cleaning them.

 

Since it's essentially the same code, a user-end tool could also be produced from this : It would be far easier for a modeler to supply a model in say 3DS and then get a correct and healthy doom3 ready file out the other side. With a lot of tools producing pretty crap material sections and shading groups I think it'd be a great help in getting models into the game with less effort and lessen frustration with things like Blender and MAX's exporters. Correction for material paths during this would also lower the entry requirement for non-technical folk.

 

I wrote some basic code which could output doom3 loadable ASE', however I never finished it or grouped materials correctly. lwo's binary form is also something I have no idea about, but it might be an interesting bit of work

 

Anyway, just an idea

[OrbWeaver]

It sounds like a lot of the useful stuff you are doing via scripting is quite similar to what I wanted to achieve with DarkResource, which was intended to work as both a GUI-based resource manager as well as command-line tool that could be used to extract or edit resources from within the Doom 3 PK4 tree. This could be integrated with other tools such as Blender, for example to allow texturing of models using Doom 3 materials or simplifying the editing of models by automatically assigning the correct material names.

 

As you can see the project hasn't been touched in ages (it's just a few skeletal GUI dialogs with no functionality at the moment), because of other mod priorities, and it's in C++ not Python so there probably isn't much scope for code sharing, but I guess there might still be some possibility of one tool using the other; for example if the backend part of DarkResource was implemented so that the PK4 tree could be walked and individual named resources accessed, this could save redundancy in other scripts.