Texture Compression

[SneaksieDave]

I can't find the other thread.* So, a new specialized one for just this topic. (sorry)

 

Has there been any consensus on texture compression? Even if it was "for now" and / or high res versions were kept / offered / etc. I'm wondering because I'm trying to sync up to the new CVS and I'm downloading 20 minute texture files (3, 4 Mb for each piece).

 

I don't remember the previous discussion details, but I remember that I was able to take one of our TGAs and save it to JPG with 10% compression (high quality) and the size was reduced from 4Mb to 180kb (!) with no noticeable quality reduction - at least to my eyes. So I'm wondering if there has been any decision made on this yet? It was also mentioned 'better sooner than later.' I'd happily redownload all of our textures (before a slew of new ones come) at 180kb each, as opposed to doing another dozen 4Mb ones.

 

Just wondrin'.

 

 

*Search really, really hates me. I do a search for my name, exact, all time, all forums, for "+56k +texture +jpg" and I get two pages of matches. Give me a break. I don't think I wank about 56k and textures and jpgs in 30+ threads, do I?

[Renzatic]

I believe we agreed to use RLE compressed TGA files. It's lossless, and it crushes it down to about 1/6th the size.

[sparhawk]

If we recompress them to JPG, it would mean yet another reorg with all maps not working.

[Renzatic]

as long as they shared the same name it'd still work.

 

Weirdly enough, if you list a texture as a tga in the material file and have it point to a texture that's actually a jpeg it'll still work.

 

But I'd much rather use RLE compressed TGA files. They're not much bigger, and they're artifact free.

[New Horizon]

Nice. I wonder if we should really take care of that sooner than later? The mod is only going to get bigger and bigger. Oh Renz, didn't we also agree to cut the textures down from 1024 X 1024 to 512 X 512? That should also reduce the size of the download considerably.

[Renzatic]

Sigh...OH I GUESS!

 

I'll keep the 1024x textures on the FTP, but we'll use 512x textures for the mod. Keep in mind that RLE'ing them will cut down on even the 1024x textures considerably.

[New Horizon]

I'll keep the 1024x textures on the FTP, but we'll use 512x textures for the mod. Keep in mind that RLE'ing them will cut down on even the 1024x textures considerably.

True, it will cut them down but if we can keep the quality and shrink them down to 512, I think we'll be a lot better off. With the sheer number of textures we're going to need, it's best to get the most from the least.

[sparhawk]

Weirdly enough, if you list a texture as a tga in the material file and have it point to a texture that's actually a jpeg it'll still work.

 

That's not so weird when you consider that most programmers don't rely on a name but instead on the header information, which you can use to determine what filetype it is.

[BlackThief]

True, it will cut them down but if we can keep the quality and shrink them down to 512, I think we'll be a lot better off.  With the sheer number of textures we're going to need, it's best to get the most from the least. 

well that doesn't work - you'll always loose a lot quality, when you scale textures down. I think we should keep for example wall-textures as high-res as possible - even T3 had 1024x1024 textures! it doesn't really matter anyway because, when people's hardware isn't up to date they'll have to play on medium or low settings and the textures are scaled down automatically at those settings.

[Zaccheus]

I thought RLE is only good if you have many pixels next to each other which have exactly the same colour values?

 

Perhaps one solution would be to have compressed 'debug' versions of the textures in CVS while we are devoloping, while storing the original 'production' high res versions somewhere else.

[Fingernail]

I've been using RLE on all my textures/model skins anyway.

[Renzatic]

If you want to avoid using RLE on everything (and I also recommend making a backup of the uncompressed textures since some people have shown interest in playing with the high res sources) why don't we just zip up certain directories and put them up on CVS?

 

TGA's zip really well, it'd save us alot of space and we wouldn't have to sacrifice quality in any way whatsoever.

[sparhawk]

You can also switch on compression for CVS. This way the line will be compressed. I don't know how well this will compress in comaprsion to RAR or ZIP but it should help. You can set this option in your client so I don't need to change anything on the server. I usually don't use this, so I don't know how much it will help.

 

Edit: The option for this is -zN where N is a number. Usually 3 seems to be a good tradeoff for CPU usage vs. compression. Higher numbers give more compression and more CPU load and I don't know what the maximum number is. You can play with the number and see what gives you the best results.

[SneaksieDave]

Renz - CVS or FTP? If they're zipped on the CVS, we'd (56k) still have to download them in order to sync properly. I just zipped a 3Mb texture and it only came down to 1.3 Mb. That's still pretty hefty.

 

As for compression, yep, I've had that enabled since day one on maximum, and for some reason, it only seems to work with things other than the TGAs. For instance, map files come really fast, even if they are large. But a 4Mb TGA still takes me 20 minutes to pull down.

 

Either format, all I ask is that we do something soon. It's really tough to stay synced with humongous textures, and if hopefully hundreds more are coming soon, I'm begging it be put into place before then. Batch conversion is a friendly thing.

 

I did the following just as a demonstration for whoever hasn't tried personally. I'm not pushing JPG with this, just using it as an example. I picked two particularly colorful textures, cropped them to make them manageable for my upload, and saved them as BMP so there's no loss in the new cropped version. Then, I saved them as standard encoded JPG with only 10% (high quality) compression. See the results. With a file size difference factor of 10X, and barely any reduction in quality, compression is just too effective to pass up. And remember the mantra: the high res versions won't be thrown away, and could be optionally used by anyone who wants them at any time. But for the sake of CVS sync'ing, and HD space, we should go with compressed textures by default.

 

The comparison images:

 

1.

BMP

JPG

 

2.

BMP

JPG

[Fingernail]

The thing is, RLE is lossless so there is no need to keep uncompressed TGAs, surely?

 

RLE compression works by, instead of having a piece of data arranged like:

 

AAAAA

 

you have

 

5xA

 

There's no less A there, it just takes less space to write down.

[sparhawk]

Well, what we could do is to remove the textures from the CVS main branch. We create two seperate CVS repositories. One is with the actual TGA files and the other one is with a copy of the same images but in JPG format. Names and everything would be the same for both. When you want to resync with TDM then you only sync with all files except the textures. For them you have to sync with the appropriate repository, but then you can choose which one to use. The disadvantage is that you have to download both repositories (the main TDM and the textures). Also when you want to update some maps and add some new textures you would have to update two repositories. But this way it might be easier for the modem users and you can switch between both versions just by selecting the appropriate texture repository. I guess it should be possible to write a script that generates the JPG repository automatically.

[SneaksieDave]

:thumb: !

[New Horizon]

Well, I don't think we really need two versions of CVS. We are going to compress the TGA's anyway, so it would be better use of our resources to get that job fixed up. RLE compression, as Finger suggests, sounds quite fine. Three and four meg textures are a bit much, even for high speed, so I don't think there is much need to keep them anyplace but the FTP for future release.

[sparhawk]

FTP is bad because it has to be manually synced. For all the people using the TGAs it wouldn't make much difference, except that textures have to be downloaded extra. The conversion to JPG can be done automatically by my server so there is no extra step involved. I installed imagemagick and I take a look how I can set this up. The problem is that in CVS you can not specify a full download with excluding part of the repository.

[SneaksieDave]

Any success with this? I'm sure those texture packs are coming soon. I can feel it! Yep!

 

[sparhawk]

Erm, aeh, ... No. No progress yet.