List of Inverted Normal Maps

[STiFU]

Thanks.

 

I have just been browsing the Dark Mod models a little bit and I noticed that some of them could do with some fixing. I am a 3ds max guy and there are no proper converters for the lwo-files besides the detour over blender, so fixing those things myself would be rather inefficient. Are you guys interested in a list of erroneous models in the form of commented screenshots and notes? That list might end up pretty big though. Already saw quite a few things in the architecture folder...

Edited January 23, 2010 by STiFU

[Springheel]

You can always make the list. I can't promise I'll have the time/inclination to fix them, however.

[Tels]

I'd say making the list is the first step to fixing things. So go ahead.

[STiFU]

Man, that was more work than I though!! Alright, so here we go...

 

Missing/wrong editor images: Most of them were easy to fix, so I just did it. However, I didn't quite know what to do with some others. Grab the package of the fixed editorimages here.

1. fixed:
   
   • junk\apple_core: Editorimages for different colors added
     
   • lights\exstinguishable\candle_01 (and other models): Editorimages generated for the materials tdm_candle_lit, bc_candle_lit and tdm_candle
     
   • furniture\clock_wall: Editorimage generated
     
   • musical\grammo3: Editorimage generated. Also, please note: All grammo2 textures are not power of two (384x512). I didn't change anything on that yet though, since it might mess up the uv-layout, I guess.
     
   • graveyard\bones\skull & dungeon\skull_on_*: Uses a default-doom3 texture, for which no editorimage was available.
     

[*]not fixed:

• nautical\bulls_eye: tdm_litbullseye has no editorimage. Didn't know how to generate one from that...
  
• lights\extinguishable\candlestand_macabre_small & lights\non-extinguishable\hooded_lantern_with_flare: Editor images for flare etc are missing, but I thought those aren't necessary.
  
• decorative\statues\statue_gargoyle: Skin "gargoyle_rock_grey_dark" references the "gargoyle_rock_grey" diffusemap, which is then darkened in the material definition. The materialdefinition references "gargoyle_rock_grey_dark_ed" as an editorimage, which doesn't exist. A new editor image has to be generated or the normal brighter version has to be used: "gargoyle_rock_grey_ed".
  
• nature\mushroom*: Most skins share the same editor image, because they also share the same diffusemap, colorized differently by shaderstages in the materialdefinition.
  

 

Missing Shaders of models or in one of their skins:

• architecture\fencing\justice_wall_fence2
  
• decorative\statues\the_hammer_medium1, skin "the_medium_hammer_darkstone"
  
• decorative\vases\cup_* (Multiple models), skin "cup_marble_shiny"
  
• decorative\wall\longbanner* & verylongbanner, skin "banners/bannerblank"
  
• decorative\wall\shields\norman_shield
  
• fireplace\fireplace_arch01
  
• furniture\tables\endtable_round01 & rtable1, defaultskin
  
• furniture\tables\pedestal_square01, multiple skins
  
• kitchen\beer_stein02 & beer_stein02_open
  
• loot\jewellery\lionhead_amulet
  
• nature\rocks\geode_base & geode_crystal_move: material crystal_light probably fails.
  
• nautical\compass_ship
  
• wearables\headgear\broadbrim_hat & woolen_cap, skin "wearables\hats\straw"
  
• wearables\shoe01
  

 

Commented screenshots of major model issues and of minor model issues. I divided the screenshots into two packages because I may have been a little nit-picky sometimes...

 

Other notes:

• industrial_towerblock_01 is the same as towerblock_industrial_01.
  
• small_house2 exists twice.
  

Edited January 25, 2010 by STiFU

[STiFU]

Now that the topic has come up in another thread, I checked if all the normalmaps in TDM are actually uncompressed (RLE disabled) and I discovered that very many (!!) of them have RLE enabled. Since there are so many, the easiest and quickest way to fix this would probably be if someone with access to the svn performed a "batch-conversion" with all images directly on the server. Simply open every file and save it in its place as TGA without RLE. Might mess up the filerevisions of the textures that weren't compressed though.

 

In a similar fashion RLE could be enabled on all TGAs in the hires repository, when you're at it anyway, to make some diskspace.

Edited February 1, 2010 by STiFU

[Tels]

Now that the topic has come up in another thread, I checked if all the normalmaps in TDM are actually uncompressed (RLE disabled) and I discovered that very many (!!) of them have RLE enabled. Since there are so many, the easiest and quickest way to fix this would probably be if someone with access to the svn performed a "batch-conversion" with all images directly on the server. Simply open every file and save it in its place as TGA without RLE. Might mess up the filerevisions of the textures that weren't compressed though.

 

Hm, I think I could do this with a script here. We should discuss this internally, because recompressing them also means we need to ship them again. Is there already an internal thread?

[STiFU]

Give me internal access and I'll tell you... Hehe nevermind, you'll have to check for yourself. Having to ship them again is of course not so nice, but at least you got it right then and people will accept a higher downloadsize for faster loadingtimes. I don't know how many files are actually affected since I only took a couple of random samples here within windows, but it could potentially be a lot. I have no proper solution for listing them until now.

Edited February 1, 2010 by STiFU

[Tels]

Give me internal access and I'll tell you...

 

Actually, that is not a bad idea Could you please post a thread in the appropriate forum?

 

Hehe nevermind, you'll have to check for yourself. Having to ship them again is of course not so nice, but at least you got it right then and people will accept a higher downloadsize for faster loadingtimes. I don't know how many files are actually affected since I only took a couple of random samples here within windows, but it could potentially be a lot. I have no proper solution for listing them until now.

 

None of the tools under linux could display the compression of a TGA file (identify comes close but on test images always says "Compression: None").

 

So I wrote that little script according to Wikipedia specs:

 

#!/usr/bin/perl -w

use strict;

my $name = shift;

 # read in first 4 bytes
 open (my $FILE, '<', $name) or die ("cannot read $name: $!");
 binmode $FILE;
 my $data;
 read($FILE,$data,4);
 close $FILE;

 my $RLE = ord(substr($data,2)) & 0x08;

 print "Warning RLE encoded: $name\n" if $RLE;

 

Then I did run it like so

 

find -name "*local.tga" -print0 |xargs -0 -L 1 ./rle.pl |wc

 

(find -print 0 prints all files zero-terminated, xargs -L 1 calls rle.pl for each one)

 

and it produced 160 (or in other words, we have 160 RLE compressed normalmaps). Oops.

 

Running it on all TGA files in the repository gives 350 as result. Oops.

[Springheel]

We'd have to see how much of an impact it had on loading times. If it is even slightly significant, it might be worth re-shipping them.

[STiFU]

I did a quick approximate timing on "Return to the City". I started the timer (on my mobile phone ) as soon as I clicked on "start mission". Every timing was done after a fresh start of TDM.

 

Initial loadtime without fix: 1:54 min

 

Initial loadtime without fix and extracted pk4: 1:40 min

 

Initial loadtime with fix (also extracted of course): 1:40 min

 

There's no difference in loadtime, so I guess the files can be kept the way they are. Throws up the question though if we should enable RLE on all TGAs, to reduce filesize. Maybe only do it for the full-install-package, but leave the update as it is... New Horizon however reported he noticed an impact of RLE compression.

Edited February 1, 2010 by STiFU

[Tels]

I did a quick approximate timing on "Return to the City". I started the timer (on my mobile phone ) as soon as I clicked on "start mission". Every timing was done after a fresh start of TDM.

 

Initial loadtime without fix: 1:54 min

 

Initial loadtime without fix and extracted pk4: 1:40 min

 

Initial loadtime with fix (also extracted of course): 1:40 min

 

Did you redo this test after freshly booting your computer? Otherwise the OS will keep the files in the cache after the first start, throwing your values off.

 

There's no difference in loadtime, so I guess the files can be kept the way they are. Throws up the question though if we should enable RLE on all TGAs, to reduce filesize. Maybe only do it for the full-install-package, but leave the update as it is... New Horizon however reported he noticed an impact of RLE compression.

 

The full-update will be replaced by the update, so after the update day, everyone will get the same package, regardless on wether they just install anew or do an update of an older install.

[STiFU]

I did not reboot. I'll do another timing later then. But forget about my suggestion about encoding all TGAs with RLE. I just realized it doesn't make sense, as they are zipped up anyway...

[STiFU]

Ok, this time I rebooted and the result was surprising...

 

Initial load time with every TGA RLE'd: 1:50 min

 

Initial load time with uncompressed TGA: 1:54 min

 

I guess we'd have to take a whole bunch of measurements for a strong evaluation. Those seem to be outliers. Or maybe it faster with compressed TGA because they are loaded faster from harddrive and extracted within RAM / VRAM.

[Tels]

Ok, this time I rebooted and the result was surprising...

 

Initial load time with every TGA RLE'd: 1:50 min

 

Initial load time with uncompressed TGA: 1:54 min

 

I guess we'd have to take a whole bunch of measurements for a strong evaluation. Those seem to be outliers. Or maybe it faster with compressed TGA because they are loaded faster from harddrive and extracted within RAM / VRAM.

 

Yeah, if your HD is slower than the decompression in ram (which I'd expect on a fast system), that might be the cases. Can you please also check the process size in both cases? Maybe one uses more memory.

 

Also, maybe another FM uses more textures and thus provides a better testbed?

 

Currently it is too early to say we should recompress them w/o RLE. Compress with RLE I wouldn't recommend (as you aid they are zipped).

[STiFU]

No noticeable difference in CPU- or RAM-usage here. I picked Melan's map for testing because it hat the biggest loadingtime of all and I just assumed it features most textures.

 

Anyway, the loading animation just extends over 1/4 of the whole loading time. I am wondering why that second phase is taking so long. Do you know which tasks are performed in that phase?

[Serpentine]

Been playing with compressing the normals, but there is no reliable way to make doom use them - if it thinks you're using a decent graphics card it will just not bother using the dds and instead use the tga, even when the dds files are placed in the correct folder etc. Its annoying as you could half the size of the 1.1gb of normal maps in TDM1.0 to around 470mb. It would be great to get 3Dc support (I have no doubt their friends at nvidia didnt want that happening tho, considering id actually did a lot of research into compressing normals but in the end 'decided' against it, you can still find the papers about it.). Would also gain a good bit of load time(since when it uses the TGA's it does its own RxGB conversion at load time) and a little performance with full mipmapping, I dont think doom3 generates them. My experiment for the day

[STiFU]

Mipmapping is not only used for better performance, but also or even mainly for eliminating Alias-effects. I can't imagine Doom 3 wouldn't create mipmaps from the TGAs in a preprocessing phase for that reason and I assume that this is the step that takes so long, after a mission has been loaded. Without mipmaps for the TGA-normalmaps, distant shading would look really weird and because I never saw anything weird about it, I am sure Doom handles them. There are also many diffusemaps stored as TGA in TDM and you'd directly notice missing mipmaps on diffusemaps from the distance...

 

But as far as you experiment in concerned, it'd be interesting to see to which amount it cuts down on loading time, especially in that second phase, to fortify my assumption. If you make sure that the dds are the only files available in your darkmod folder and all other textures are stored within pk4, it should use the dds files. If that doesn't work either, you'd have to move the TGA-normalmaps somewhere else. Have you converted all normalmaps to dds/dxt1 now?

Edited February 2, 2010 by STiFU

[Serpentine]

See, this is the interesting bug : even if you _force_ it to use the dds by specifying the exact filename and placing it in the location - it will skip them if it somehow detects it doesnt want to use them (Besides setting your gfx to low/medium I cant find any cvar that controls this reliably without influencing diffuse etc). So I had the same idea as you, Im going to force it by removing the tgas... surely it will default to the dds... surely. Well it turns out it doesnt quite like the idea of that either and rather enjoys the look of black ingame.

 

Normal's cant use DXT1/3, you had to use DXT5 (yes, unlike the others), with specific colour biases. You can find more on the wiki and Katz blog (there's also a link to the correct version of compressenator there, which you will need, the latest has removed the colour format as it is somewhat problematic) If you have any issue with the dds file not following this exactly, doom3 will not do anything with the file and will skip it. But even after all of these specifics, the results are 'ok' for most normals, but no where near 3Dc, which is finally supported by nvidia - but typically they do it in a round about way.

 

If you want to look at a comparison of filesize etc, you can just copy all of the *_local.* out of the TDM folders and run a batch conversion on them. DXT1 with mipmaps should give a comparison to the ideal method - tho the more likely option is that which uses DXT5 + mipmaps and the specifics.

[STiFU]

So you've got all normals encoded properly with dds now? I don't care about the complete filesize of the normals after compression, but I'd really be interested how loading time is affected compared to uncompressed normalmaps. Hope to see some measurements of the two load-phases with the different normalmaps from you soon...

Edited February 2, 2010 by STiFU

[Serpentine]

Hmmm, I guess I can try it out on medium and see if that changes the load time - then compare it at medium if it seems reasonable. Will do that tonight

[Springheel]

The reason we left normals as .tgas is because .dds files don't work in addnormals declarations.

[STiFU]

Ah ok. But well I was just curious. Wasn't planing on suggesting to compress the normalmaps. I just would like to know what takes so long in the second loading phase and I suppose it's the Mipmap-generation.

 

We'll see tonight hopefully...