Cleaning up textures

[Serpentine]

Hey there,

 

Since my time is really limited atm I've just finished writing up my file info/metadata parser to deal with common mistakes. I've noticed a lot of files which are just incorrectly formatted or contain a lot of data that shouldn't be there.

 

In short : remove crap from jpegs to tidy stuff up and make sure nothing uses colour profiles/whatever. Clean up and fix DDSs to remove crashes and lower hardware/load time adjustments.

 

Editor and other JPEGs : Some of them have 200% more metadata than actual image data. Would anyone object to me losslessly stripping all exif/jfif/adobe/colour profiles from these? There's no real gain but atm it's just a bit messy and shabby looking. A few files are saved with some really bad presets, I'll re-save them correctly, however there will be a sliiiight degradation due to the lossy nature, but nothing really noticeable.

 

A number of jpegs are also not power-of-two, greebo does this matter/anything for DR?

 

DDS : The list of non-"power-of-two" textures is pretty big (~50), most of them are GUI related tho. I'll have to see how those react to being resized/extended. Will be a bit tricky as some of them are non-square/power of two which means that cropping/resizing will need to take place. These are almost all DDS files which the current image checking script doesn't deal with.

 

A few DDS files are also saved in a completely non-standard format which neither ATI Comp not nVidia texture tools will read correctly, Assuming their headers are correct there will be no re-encoding/quality changes. 2-3 DDS files also have raw RGB data, not DXT encoded; these will be converted accordingly. A number of old DDS images also have what seems to be luma/alphaMask data stored in the footer, this should be chopped off too as it's most likely not used and no one has bothered with it in many years, it also has issues with newer drivers/cards.

 

Some people have also been using DXT3/5 for textures without alpha channels, they arn't an immediate problem but just a reminder for people

Also there's no advantage in using DXT3 for textures with alpha, rather use DXT5 and specify the alphaTest explicitly, DXT5 will have better color than the DXT3 equiv and a lot more future-proof.

 

So, comments/concerns. If anyone has experience resizing UI files, that'd be good too

[Tels]

Hey there,

 

Since my time is really limited atm I've just finished writing up my file info/metadata parser to deal with common mistakes. I've noticed a lot of files which are just incorrectly formatted or contain a lot of data that shouldn't be there.

 

In short : remove crap from jpegs to tidy stuff up and make sure nothing uses colour profiles/whatever. Clean up and fix DDSs to remove crashes and lower hardware/load time adjustments.

 

Editor and other JPEGs : Some of them have 200% more metadata than actual image data. Would anyone object to me losslessly stripping all exif/jfif/adobe/colour profiles from these? There's no real gain but atm it's just a bit messy and shabby looking. A few files are saved with some really bad presets, I'll re-save them correctly, however there will be a sliiiight degradation due to the lossy nature, but nothing really noticeable.

 

A number of jpegs are also not power-of-two, greebo does this matter/anything for DR?

 

Cleaning of JPG isn't that a priority (in my opinion) because:

 

* they are only for editor work.

* they sometimes interfere anyway (because since DR can now do alphamaterials etc, but the editor JPGs have no alpha, or the editor JPGs are of so ridiculous low resolution that you can't judge the quality of your level by them)

* the tiny amount of memory they save isn't probably that important compared to all the other assets (and you need the memory then later anyway, when you test your map in D3)

 

Of course, slimming them down automatically would be ok, but I wouldn't spent to much time on it.

 

Power-of-two: I can only imagine that whatever graphic driver has problems with that, will also have that in DR. It uses the same driver/hardware to render stuff.

 

DDS : The list of non-"power-of-two" textures is pretty big (~50), most of them are GUI related tho. I'll have to see how those react to being resized/extended. Will be a bit tricky as some of them are non-square/power of two which means that cropping/resizing will need to take place. These are almost all DDS files which the current image checking script doesn't deal with.

 

A few DDS files are also saved in a completely non-standard format which neither ATI Comp not nVidia texture tools will read correctly, Assuming their headers are correct there will be no re-encoding/quality changes. 2-3 DDS files also have raw RGB data, not DXT encoded; these will be converted accordingly. A number of old DDS images also have what seems to be luma/alphaMask data stored in the footer, this should be chopped off too as it's most likely not used and no one has bothered with it in many years, it also has issues with newer drivers/cards.

 

Some people have also been using DXT3/5 for textures without alpha channels, they arn't an immediate problem but just a reminder for people

Also there's no advantage in using DXT3 for textures with alpha, rather use DXT5 and specify the alphaTest explicitly, DXT5 will have better color than the DXT3 equiv and a lot more future-proof.

 

So, comments/concerns. If anyone has experience resizing UI files, that'd be good too

 

Cleaning up the DDS files (and converting the TGA files added over the last months, esp. GUI) would be worthwhile and you seem to be the one with the most experience on this I can't answer on the GUI files, tho. I think if the non-power-of-two wouldn't have worked for someone, we would have heard complaints already?

[Serpentine]

Cleaning of JPG isn't that a priority (in my opinion) because:

Of course, slimming them down automatically would be ok, but I wouldn't spent to much time on it.

Yeah it's all automatic, and copy paste code from my other stuff. I only noticed some of the files were carrying a lot of excess due to the debug I left in by mistake. I did some testing and DR doesnt freak out about non power of two even if the card has it disabled. So I'll just strip them and leave them as is in regards to res/dpi.

 

Cleaning up the DDS files (and converting the TGA files added over the last months, esp. GUI) would be worthwhile and you seem to be the one with the most experience on this I can't answer on the GUI files, tho. I think if the non-power-of-two wouldn't have worked for someone, we would have heard complaints already?

I thought the same thing, but since we have got unexplained crashes on some maps on people like Shaz' hardware and we already have pretty good informal standards, I dont see it hurting. Better avoid crashing than shrug it off and hope for best case I also think a few of these result in pink-edged paper gui objects ingame but that's just a guess.

 

As for converting new, iirc there is a script?

 

And on a tangent : about crashing on non-power of two, it's not really universal. There's a specific OpenGL extension for dealing with it and some games/editors deal with it by automatically resizing rather than hoping it's there/correct. I think that's what doom3 does and it's why I want to avoid software "fast" resizing or "suck it and see" crashing. Driver override of this is also possible in some situations but in things like Knights of The Old Republic where it was thought of as "it'll always be there" there were a lot of shaders that would react very badly with the infringing textures that made assumptions resulting in both bad performance and occasional visible issues.

 

I'll most likely commit cleaned jpegs later and fixed dds (the simple ones) tomorrow. More complex cases later as well as a list of files/issues.

[Springheel]

I think if the non-power-of-two wouldn't have worked for someone, we would have heard complaints already?

 

The main issue isn't crashing, it's that D3 resizes the non-power-of-2 textures, resulting in a blurry appearance. They should be changed. I don't *think* it will affect the actual layout of the GUI, since D3 resizes them anyway, but it's been a while since I worked with guis and I forget how much the size of the image depends on the window it's in.

[Serpentine]

The main issue isn't crashing, it's that D3 resizes the non-power-of-2 textures, resulting in a blurry appearance. They should be changed. I don't *think* it will affect the actual layout of the GUI, since D3 resizes them anyway, but it's been a while since I worked with guis and I forget how much the size of the image depends on the window it's in.

 

Ok, that's what I was looking for ; will experiment with scaling them correctly and see how it goes.

[Tels]

Yeah, I agree its better to be safe than sorry and fix the non-power-of-two issue. ATI cards had crashes (on a column model) with some textures.

 

As for JPG, if it is automatic, go for it.

[STiFU]

Sounds great generally, although I am kind of afraid, we'll run into problems with updating all that stuff. Your script will probably create numerous alterations in maaaaany pk4-archives, resulting in (a) a big update and/or ( untidyness. Maybe we should hold back on tasks like this until our updater can alter and clean up archives.

[Tels]

Sounds great generally, although I am kind of afraid, we'll run into problems with updating all that stuff. Your script will probably create numerous alterations in maaaaany pk4-archives, resulting in (a) a big update and/or ( untidyness. Maybe we should hold back on tasks like this until our updater can alter and clean up archives.

 

Nah, I'd say go for it. You never know what happens to your free time later (e.g. it will simple be a forgotten task).

 

If push comes to shove, we just push out a big update (we might just rename it v1.1 and tell people "its sooo good" or we block the editor JPGs from release.

 

However, I fear due to the long development time, we will have to ship a lot of PK4s anyway, or we risk fragmenting them even further.

[Springheel]

Your script will probably create numerous alterations in maaaaany pk4-archives, resulting in (a) a big update and/or ( untidyness

 

Yes, that's a valid concern. The texture folders are generally the biggest, and I'm not sure people are going to want to download them all again just to get .jpgs that are a few mb smaller. I wouldn't. Players never even see those .jpgs.

[Tels]

Yes, that's a valid concern. The texture folders are generally the biggest, and I'm not sure people are going to want to download them all again just to get .jpgs that are a few mb smaller. I wouldn't. Players never even see those .jpgs.

 

Yeah, but as I said, we can easily block them from the update (f.i. by simple not releasing the new PK4 files).

[Serpentine]

There are a number of textures with uppercase names for direction?/quality (ocean_NE_forward.tga etc). Quite a few skyboxes, but I see no explicit requirement for this. Some others are gui papers and such.

 

I know doom3 only spits out a console warning about them and doesn't care too much deep down, but for the sake of keeping the console cleanish, would anyone object to this? Yes, I would adjust the material file to accordingly.

 

I don't think the pk4 issue is massive, most of the files this touches will already be getting updated from other changes, and if not it's not too much to just not release the specific pk4. Keeping things clean is something that has to start some place anyway, TDM is already in great shape for the most part

 

Will link some output here soon as the DDS portion finishes the run (dxtc decoding without C, it's real slow )

 

Edit : in all its hard-to-read glory:

http://syr.taaaki.za.net/tdm/temp/imgsat2.txt

[Tels]

There are a number of textures with uppercase names for direction?/quality (ocean_NE_forward.tga etc). Quite a few skyboxes, but I see no explicit requirement for this. Some others are gui papers and such.

 

D3 on linux definitely warns about these, because "ocean_NE_forward.tga" is a different file from "ocean_ne_forward.tga" and from "ocean_NE_forward.TGA" and on linux, there is no way to find out if the file that you named doesn't exist, if it exists in any other spelling.

 

E.g. if you specify "AAb.tga" and it doesn't exist, how should the engine know that the file is really called "Aab.tga", "AAB.TGA", "aab.tga" or any other combination? The only "easy" checks are to try to load all lower-case and all upper case, but if these don't exist, all bets are off.

 

So it is alway worthwhile to convert filenames to lower-case, so that the material definitions etc. become easier.

 

I don't think the pk4 issue is massive, most of the files this touches will already be getting updated from other changes, and if not it's not too much to just not release the specific pk4. Keeping things clean is something that has to start some place anyway, TDM is already in great shape for the most part

 

Thanx, I take that as compliment (You wouldn't have guessed how much cruft we already cleaned up before release :-o

 

Will link some output here soon as the DDS portion finishes the run (dxtc decoding without C, it's real slow )

 

Edit : in all its hard-to-read glory:

http://syr.taaaki.za.net/tdm/temp/imgsat2.txt

 

Do you have a script to auto-clean the JPEGs? If not, I think "convert" (from ImageMagick) would be easily able to do so, maybe plus exiftools (both under linux).

 

Btw:

 

\textures\lode\heightmap.tga

 

Oops, my bad. But on the other hand, the LODE system doesn't care whether the file is power-of-two since ireads it directly. However, maybe, in the far future, we might push the calculations to the GPU, and then it would matter. So I am unsure if we should do something about it right now.

 

Also, pushing all editor-JPS into their own PK4 sounds like a jolly good idea, because then only mappers could download this PK4. That would allow easier updating of it, and also players don't need to download it, and their machines don't need to parse it (faster startup, slightly less memory).

[Serpentine]

Do you have a script to auto-clean the JPEGs? If not, I think "convert" (from ImageMagick) would be easily able to do so, maybe plus exiftools (both under linux).

 

Yeah, I use jpegtran for stripping and a custom pass for jfif which non of the tools really offer. ImageMagick is something I try avoid

 

Anyway, will rename the relevant stuff just need to give it some better logic for dealing with (safe)svn actions. For now it's time to learn a years worth of work I've never touched in a couple hours :/

[Serpentine]

Issue List

Updated the list after cleaning the jpegs. Also a whole lot more readable

 

Updated the TGA parser to look for a lot more things, tho very few problems were found

 

Going to start working on the non-pot textures now, starting with the easy stuff.

[Tels]

Issue List

Updated the list after cleaning the jpegs. Also a whole lot more readable

 

Updated the TGA parser to look for a lot more things, tho very few problems were found

 

Going to start working on the non-pot textures now, starting with the easy stuff.

 

Very cool. Just one question:

 

\textures\lights\church512x256.tga
[ Warn: 50 Num: 1 Autofix: False ]
+ TGA Color depth(8) is not 24/32 bpp

 

Why is this a problem? Also, my LODE code works fine with 8 bit (I am unsure if it even supports more than 8 bits), so please keep at least this in mind. Otherwise, good work!

[Serpentine]

+ TGA Color depth(8) is not 24/32 bpp

 

Why is this a problem? Also, my LODE code works fine with 8 bit (I am unsure if it even supports more than 8 bits), so please keep at least this in mind. Otherwise, good work!

 

Ah I actually forgot to comment that out, but I had noticed it before and remembered to ignore it . I remembered making a 16bit TGA for something, trying to use it would result in a warning in the console about only using/supporting 24/32bit (similar to how if you use strange color formats for DDS it moans). However I think it might be related to using them in normal/diffuse layers. Lights seem to be fine. I only added the check to remind me to investigate more. 8bit TGA's are actually great for light storage and I think if everything is happy and it was a false alarm, it might even be worth moving the light textures that don't make use of > 8bpp to 8bit RLE TGA's (could be easily automated to check/lossless convert)

 

There was also someone who had trouble with 24bit TGAs made by Paint.NET, but I need to find the sample again and cant produce one since .net framework claims it cant activate on my copy of windows, garrr. If anyone has similar stories, I'd be interested.

 

The script is now up to 36 checks excluding the implicit image validation in the format input parsers. I've added some basic checks for finding 32bit tgas which don't use their alpha chan, since those are noticeably larger. I wont make a move on those until my material script understands materials a bit better (then I can do mat->texture checks instead of guessing mat off filename). I hope to be able to then move all diffuses/speculars to their correct formats with some automation and auto-backups. Doing this will give a nice boost to TDMs performance and decrease memory usage for the most part

[New Horizon]

Ah I actually forgot to comment that out, but I had noticed it before and remembered to ignore it . I remembered making a 16bit TGA for something, trying to use it would result in a warning in the console about only using/supporting 24/32bit (similar to how if you use strange color formats for DDS it moans). However I think it might be related to using them in normal/diffuse layers. Lights seem to be fine. I only added the check to remind me to investigate more. 8bit TGA's are actually great for light storage and I think if everything is happy and it was a false alarm, it might even be worth moving the light textures that don't make use of > 8bpp to 8bit RLE TGA's (could be easily automated to check/lossless convert)

 

I wouldn't bother with RLE. The last time I checked RLE's weren't supported natively on any video card. What I mean is that when the level loads RLE textures have to be re-expanded...increasing load time.

[Serpentine]

I wouldn't bother with RLE. The last time I checked RLE's weren't supported natively on any video card.

Targa isn't supported as a hardware format for any modern graphics card, RLE isnt really worth the effort in most places but since light textures have a lot of linear repetition(think black) there's at least a little point to using them. So if anything there would be a reduction in load time

[Tels]

Targa isn't supported as a hardware format for any modern graphics card, RLE isnt really worth the effort in most places but since light textures have a lot of linear repetition(think black) there's at least a little point to using them. So if anything there would be a reduction in load time

 

Hm, theoretically, yes, but practically, the TGA is stored compressed inside the PK4, anyway, so it goes like this:

 

* load 1Mbyte compressed data from disk => uncompress to 4 Mbyte of data => end up with 4 Mbyte of data directly, => recompress as DDS => upload to graphic card

* load approx 1Mbyte of data from disk => uncompress to 1 Mbyte of RLE data => uncompress to uncompressed TGA => end up with 4 Mbyte of data directly, => recompress as DDS => upload to graphic card

 

So I think it might be that RLE inside a PK4 doesn't save us anything, it might even make things slower. However, it reduces the size of the SVN repository considerable, and it might make loading times *there* faster (which is only relevant to developers, tho).

[New Horizon]

Targa isn't supported as a hardware format for any modern graphics card, RLE isnt really worth the effort in most places but since light textures have a lot of linear repetition(think black) there's at least a little point to using them. So if anything there would be a reduction in load time

 

Like I was saying though, RLE has to be un-compressed during map load. So as Tels said...texture is expanded from pk4, then inflated back to a regular TGA, and if the player has compression turned on...compressed back to a dds.

 

See Tels post, which explains it quite well.

 

TGA without RLE is the way to go.

 

Are the pk4's compressed with 7zip, or just standard windows zip? If it's 7zip, and LZMA is being used...then a 3meg TGA compresses down to .98megs. With standard windows zip it's 1.25. So it's not worth adding another layer of compression, causing the game to un-compress the same file twice.

[Tels]

Like I was saying though, RLE has to be un-compressed during map load. So as Tels said...texture is expanded from pk4, then inflated back to a regular TGA, and if the player has compression turned on...compressed back to a dds.

 

See Tels post, which explains it quite well.

 

TGA without RLE is the way to go.

 

Are the pk4's compressed with 7zip, or just standard windows zip? If it's 7zip, and LZMA is being used...then a 3meg TGA compresses down to .98megs. With standard windows zip it's 1.25. So it's not worth adding another layer of compression, causing the game to un-compress the same file twice.

 

We use 7zip for the PK4 releases.

 

Anyway, while I agree with that assesment, I also said that RLE compressed TGA files slim down the SVN size, and if you need to recheck out SVN, you will be happy that you don't have to download a 4 Mbyte TGA instead of a 0.98 one OTOH, as a player, you don't care that the TGA is first uncompressed from RLE because that basically happens at 0 cost during load (RLE isn't so much a compression, more an encoding scheme in that rehard. The overhead is almost nil).

 

So its a hard decision I certainly would not go ahead and change existing TGA files from RLE to non-RLE (SVN updaters and the SVN server will hate you

 

However, I think compressing things to DDS would be much better, so instead of focusing RLE or not, work on this. We have accumulated quite a few non-DDS textures (GUI buttons etc) since Spring and these should be recompressed.

 

We also might finally have a look on what other textures we can compress savely as DDS. Every bit helps.

[Tels]

Edit: Nevermind, my bad.

[New Horizon]

Anyway, while I agree with that assesment, I also said that RLE compressed TGA files slim down the SVN size, and if you need to recheck out SVN, you will be happy that you don't have to download a 4 Mbyte TGA instead of a 0.98 one

True, but a full checkout probably isn't a weekly occurrence. Thank goodness for that. DDS is the way to go for most things anyway, but I'm glad to hear we use 7zip...it's far more efficient than windows zip.

[Tels]

True, but a full checkout probably isn't a weekly occurrence. Thank goodness for that.

 

Yeah, but imagine someone changes our couple hundred Mbytes of TGAs from whatever-they-are-now to non-RLE

 

Anyway, I just stored a list with warning that were generated during the extract-assets.pl script in devel/d3_assets.txt - it is mostly a list of textures and sound-shaders we are "missing", which means in almost all cases they come from D3. The list is actually rather small, so if someone feels adventurous, they could start on recreating some of these from free content or scratch

[New Horizon]

Cool. I'll take a look at that list.