FM compression

[7upMan]

Hi guys,

 

I'm in the middle of translating an FM, and while re-packing I experimented a little with the compression settings. Turns out, the best TDM-readable compression is achieved with the following settings in 7zip:

 

Archive format: Zip

 

Compression ratio: Ultra

 

Compression method: Deflate

 

Dictionary size: 32 KB

 

Word size: 258

 

I noticed that most, if not all FMs are packed/zipped with very low compression. I don't know if there's a specific reason to this, but this way, I was able to size down Melan's "Return to the City" from 10 MB to 8.5 MB. Future FMs will likely exceed the 20 MB size (as they frequently do in Thief2) and will benefit even more from high compression.

[stumpy]

my half built map is at 6mb with the proc file at 9mb the cm file at 7mb and the aas32 file for the corpse at 6.5mb and the script file at 10k and guis at 3.5mb and xdata at 75k and am about 38percent done although 98% terrains in and 80% script, and 50% lights, 95% visportals, and doors 100%, objectives 0%, AI 2%, decorations 15% and anything I've missed at 0%. how big is my mission likely to be. It covers a city district and a big castle/mansion which isnt shown in the screenshots I posted, and some sewers.

[Fidcal]

Does Doom recognize it OK? If so, that might be useful. Generally though we just use the standard. I just drag the files into Winrar without a thought for compression type.

[7upMan]

Does Doom recognize it OK?

 

 

 Yep:  

 

Turns out, the best TDM-readable compression is achieved with the following settings in 7zip

[Serpentine]

You should most likely bench the difference on a mid range pc, I always unpack the pk4's so that I can mess around easily, but I would think that it might have slowdowns loading some map areas etc if you compress it quite heavily and it is a custom texture/model/asset map.

 

But just how much - who knows.

[Komag]

It's mostly a matter of effort versus payoff. It's very much easier to just pack it with default settings, no worries, quick and easy, no bother to set up the ideal settings.

 

And no one really cares about 10 MB vs 8.5 MB (or even 100 MB versus 85 MB), not nowadays, not when I've uploaded over 100 GB in the past few days, for instance

 

And the slower unpacking would indeed be an extra bother that makes it even more not worth it

[Ishtvan]

I thought we discussed this once before and the D3 pk4 handler (not our mission installer that just copies the pk4 to the right directory but does not expand it) couldn't handle 7zip, and we didn't want to force everyone to download it?

[Serpentine]

I dont think the OP is talking about 7zip(LZMA), rather just standard Deflate(I think thats all doom3 supports) using better settings.

 

But agreed, time vs. effort is not worth it, its an extra step for an author for such a tiny gain. If file mirrors wanted to make a script to automatically recompress the uploaded .pk4's with better ones it wouldnt be too hard either (if anyone ever worries about that)

 

Another thing is while FM's that are large are released, a large portion of them will be the textures. TGA doesnt compress very well at all, DDS even less. .maps and associated formats should compress reasonably well, so that isnt a worry. If TDM really takes off and people start getting massive bw hits, perhaps fm's coult contain a flag which tells TDM to compile the map on install... but that is highly unlikely and just food for thought

[STiFU]

Sorry I just have to be a wise ass on this one... The TGAs in TDM compress pretty darn good, because they are completely uncompressed images (Run Length Encoding is disabled), hence they are afflicted with a lot of redundancy which is removed/reduced in the compression process. The more redundancy a signal contains, the more it can be compressed, which is why DDS can't be compressed as much. It's redundancy has already been spent. To fortify my point, I just zipped a TGA and it shrinked down to one fifth of its original size.

 

So, compiling the map at the enduser, huh? How long does the compiling take on a bigger map? There is no lightmap to render or anything, so I guess it won't take up as much time as a Quake 3 map-compile by far, but still, it might be annoying. If filesizes and traffic due to a massive TDM-fancrowd eventually become an issue, we'll just have to settle for torrents and oneclick filehosters for a short period after an FM was released.

Edited February 1, 2010 by STiFU

[Serpentine]

Ah, my TGA's are all RLE'd which would make quiet a bit of difference, but that is besides the point, the reason I said it was pointless to try compressing them more is that the simple deflate with fairly minimal settings picks up on the redundant data very easily (I wasnt clear)

 

Someone asked about jpeg decoding in doom3, yes it is supported... tho avoided at all costs for things like specular/normals due to the artifacts looking really bad on scaled textures, lightmapping or something it -could- be ok if you controlled the encoding variables.

 

As for compilation time, on a 4 year old rig it takes something like return to The City about 2 minutes to compile. But yeah it was just a more conceptual thing Yes, I am a terrible person who likes to compile thier entire nix box etc, even if there is no real reason these days.

 

I will throw the tdm files through uharc with its multimedia extensions for some strange kind of amusement.

[STiFU]

As far as I know doom 3 isn't so fond of RLE'd TGAs, so be warned. Interesting info about the jpeg support though. I was in fact the one who asked that question, but in a different thread and on a different board... Over at ttlg in the "Welcome Dark Mod fans"-thread I was suggesting to use jpeg compression on AO-maps to keep the filesize down, if the specular alpha-channel is not an option.

 

Considering the jpeg artifacts: Is JPEG2000 supported as well? No problems with artifacts there... ;D Haha, can't imagine they'd implement that!!

Edited February 1, 2010 by STiFU

[Serpentine]

I havent run into any trouble with RLE, but I did see some people mention it over at doom3world. Sadly, even once opensource; adding jpeg2000 would be somewhat futile, as it stands most opensource libs for decoding it are horribly slow - and thats just in pure software, where most things dealing with jpeg get a substantial boost by hardware these days. On the other hand it does allow you to easily get hold of scaled data (think mipmaps, kinda). Its an interesting thought, a lot of current opensource and even commercial games, simulators and the likes would benefit greatly from a new optimized texture forumat taking into consideration things like alternative maps etc, I know there was meant to be some M$ one, but I havent heard anything of it lately and Im sure it isnt supported by any editor.

[New Horizon]

Any textures going into TDM shouldn't be RLE'd, this was outlined in our texture guidelines, at least it once was...so it still should be.

 

RLE saves a bit of space on the hard drive, but it increases loading time because RLE is not a format native to video cards, so at load time they have to be re-inflated as a standard TGA.

 

I tested this myself, and it does make a difference in loading time. I hope this doesn't mean we'll have to go through our repository to make sure all the textures are non RLE. If video cards began handling RLE TGA's natively, then it would be worthwhile...but as it stands there is no real benefit.

[7upMan]

After reading Komag's post, I think I need to clarify one thing: Just as not everyone here has an up-to-date gaming rig, not everyone here has an even half-decent Internet conection. Just think of the dial-up users among us, or those who have what is called DSL Light (384 kbit/s) here in Germany. For example, my gf's parents live in a village in the middle of nowhere. They have Internet either over ISDN (64 kbit/s) or UMTS (wasn't available until half a year ago). Still, even UMTS is horribly slow if compared with a 16 Mbit/s DSL line.

 

My point is that it makes almost zero difference for the mapper to enable higher compression and save a few megabytes. Granted, it increases the time to compress the FM on the mapper's side, but honestly: how often do you pack it? And you don't need to enable high compression for every test-run of your FM, just for the final version to be released.

 

Well, it's just some food for thought from my side. 

 

 

[Serpentine]

Any textures going into TDM shouldn't be RLE'd, this was outlined in our texture guidelines, at least it once was...so it still should be.

 

RLE saves a bit of space on the hard drive, but it increases loading time because RLE is not a format native to video cards, so at load time they have to be re-inflated as a standard TGA.

 

I tested this myself, and it does make a difference in loading time. I hope this doesn't mean we'll have to go through our repository to make sure all the textures are non RLE. If video cards began handling RLE TGA's natively, then it would be worthwhile...but as it stands there is no real benefit.

 

The texture guidelines are quite badly worded and do say that RLE is fine - I wont change that til its confirmed.

 

Most modern cards dont actually have great management for TGA, certainly not low level. They are most likely converted to just standard RGB which is fairly inefficient in itself.

 

I wonder if d3 supports 3Dc mmm

[STiFU]

Actually the texture guidelines do forbid RLE (see compression standards). It's just allowed for the highres repository, because these images are not used within Doom.

[Serpentine]

Since it does say diffuse is to be DDS and Normals uncompressed, Im fairly certain I meant that the archival copy would be TGA+RLE. I dont know many people that work on textures and scale down each time they save to check ingame

 

The comment was made as he was suggesting that perhaps people had put in textures which were RLE'd, while I said that its likely that people did hand over RLE'd copies as it does allow for it, if those are used in the mod is not part of the logical statement however.

 

In a world where carrots can be held in baskets while apples must always be in baskets it is the case that there could be both carrots and apples in a basket.

 

Mountain, molehill etc.

 

And the following incorrect formats are in 1.0 (normals):

gfclock_local.dds

lion_glass_local.dds

arch_scrollwork_local.jpg

corpse_local_ed.jpg

stripes_local_ed.jpg - should be redone, looks like the colours are way off

[Tels]

And the slower unpacking would indeed be an extra bother that makes it even more not worth it

 

It won't unpack slower. It might even unpack faster,because it has less data to read in (but the same amount of data to write out).

 

As for the "bandwidth" don't matter, people with slow lines certainly DO mind if it takes 16 or 20 minutes to download something.

[Tels]

Since it does say diffuse is to be DDS and Normals uncompressed, Im fairly certain I meant that the archival copy would be TGA+RLE. I dont know many people that work on textures and scale down each time they save to check ingame

 

In fact, FMs should contain textures as DDS. They are smaller, and they reduce the video memory amount used considerable (because they can be used compressed).

 

In any event, the average FM should not contain many textures, because TDM will contain them.

 

And the following incorrect formats are in 1.0 (normals):

gfclock_local.dds

lion_glass_local.dds

arch_scrollwork_local.jpg

corpse_local_ed.jpg

stripes_local_ed.jpg - should be redone, looks like the colours are way off

 

Thanx, we will fix these.