Better compression for TDM:

[peter_spy]

This is more like a wish for TDM itself, rather than DR, but what do you think about adding support for other compression formats to TDM? Right now we can only use the old "deflate" method which is rather slow and inefficient. 7-Zip offers several types of compression for zip files, and here is a small comparison for bigger TDM missions out there:

 

PK4s are original files, compression methods are in suffixes. As you see, the difference can be substantial, and those LZMA and PPMd archives also compress much faster. It took 3 minutes for Volta to compress with the deflate method vs 50 seconds with PPMd and LZMA.

[Bikerdude]

penny3 shows a massive improvement

[peter_spy]

Thanks Biker, it looks better as a separate topic. Naturally, we have to see how this affects loading times. Since first loading is usually a slog, a few extra seconds won't make that much of a difference. Anyway, I'll be happy to test that feature.

[duzenko]

I think highly-compressed LZMA is slower to decompress than zip.

[peter_spy]

Definitely, while deflate has average compression ratio, the decompression speed is pretty high. We have to see whether pros outweigh the cons.

[demagogue]

It'd be saving 100s of MB, maybe a GB or 2 if we're talking about ~100 FMs, so worth taking a serious look.

[rich_is_bored]

I'd rather piss away GB of storage than have to wait any longer at the loading screen. We should really address that first.

[nbohr1more]

Now you're giving me a guilt trip. Not too guilty though. Trying to get RGTC support working is difficult and

I will return to my attempts when 2.05 is stabilized.

[rich_is_bored]

Don't let me pressure you. Everything you contribute is appreciated.

 

I suspect the real culprit in both loading and storage space are all the plain text files that require parsing. Here's an example ripped straight out of the training mission...

 

 

model { /* name = */ "func_static_200" /* numSurfaces = */ 0

}

 

Here we have a block from a .proc file. Notice the use of C style block comments in a file that content authors are unlikely to ever inspect. And what data is being conveyed? That the model for func_static_200 contains zero surfaces. 63 bytes to say essentially nothing.

[Goldwell]

It took 3 minutes for Volta to compress with the deflate method vs 50 seconds with PPMd and LZMA.

 

I'm not too familiar with the term "deflating" but my computer can uncompress the volta pk4 archive for example in literally 2 seconds.

 

Correct me if i'm wrong but isn't TDM basically doing the same thing when it opens an archive to play the mission?

[peter_spy]

Yes, it can uncompress easily, it's the compression that takes a long time and it's not too efficient.

 

I can only guess that TDM loads the whole FM into memory, that's why it takes so long at first. Level streaming was only available around first Gears of War and UE3, so we can't rely on anything like that. Loading zones, anyone?

[Bikerdude]

Correct me if i'm wrong but isn't TDM basically doing the same thing when it opens an archive to play the mission?

The decompression engine in TDM is I image very old, compared what what you used above that took 2 seconds..

[Goldwell]

The decompression engine in TDM is I image very old, compared what what you used above that took 2 seconds..

 

I was moreso thinking hardware differences rather than software. I haven't tested if hardware makes an improvement to TDM loading times though, as I have a fairly high end PC and it still takes a long time to load.

 

It would be interesting to look into though if an improvement in how TDM decompresses PK4 files translates to improved loading times because they take quite a long time currently.

[duzenko]

I think it only makes sense to spend time on the compression changes if we can decompress in a separate thread.

It's the only way we could improve both decompression speed and compression ratio.

And improving one at the expense of the other probably leads to no added value.

[Bikerdude]

It would be interesting to look into though if an improvement in how TDM decompresses PK4 files translates to improved loading times because they take quite a long time currently.

Well one test that springs to mind, is to do a mission load time test with uncompressed versus compressed, from a fresh start of TDM. As this would show how and if the long load times are being held up by decompression of the .pk4...

[peter_spy]

I did only two tests, but the results so far are inconclusive. Volta usually needs 51-55 seconds on my laptop, Volta in uncompressed (Store mode) zip, around the same time. But with CoO: Behind Closed Doors, the difference was substantial. The compressed version takes whopping 2:29 to load. But, in a Store mode the loading time is reduced to 1:45 to (still very long, but hey). Wondering what that may be, I checked the content of both pk4s. Seems like there's a huge difference in the number of files. Volta has like less than 300 files, BCD has over 900 (!). That definitely prolongs the decompression time.

Edited January 19, 2017 by Judith

[peter_spy]

Two more missions:

 

penny3 Normal mode - 6:53!!!
penny3 Store mode - 2:49

gatehouse normal - 0:59
gatehouse Store mode - 0:31

 

Now, either my laptop's hardware isn't properly supported in TDM, or there's something very wrong here. Almost 7 minutes of loading time is just abysmal. Surely that has something to do with slower HDD (5400 vs desktop's 7200 rpm), but I think the difference won't be huge. I can see by the HDD light that it isn't used much while loading. Maybe it's a slow bus between mobo and the gpu, or gpu and vram. That said, I can comfortably play Dishonored 1 on max settings, and Thief Reboot with most options on high.

 

I will test those on my desktop too. I just upgraded to from GTX 650Ti to GTX 1060, so I got a huge boost in GPU power. But I kind of doubt it will change much in TDM though.

 

Edit: Actually, it seems it may have something to do with laptop's weird dual gfx card thing. Even though I've already chosen the nvidia card as preferred gpu in the driver menu, setting the com_videoRAM in .cfg to 2048 actually made loading this slow. Setting it back to 128 reduced PD3 loading time to ~3 minutes. The weird thing is, it only happens in some missions.

Edited January 20, 2017 by Judith

[Bikerdude]

If you upgrade your hard drive to SSD and (then use the old hdd for documents, downloads, pictures etc). You will in a nice decrease in load times.

 

In the mean time i am gonna run the same test on my pc with the dame missions to see wht sort times i get in comparison to yours.

[peter_spy]

Good idea. Also, I just read that 5400 rpm drives are only much slower than 7200s when reading multiple files, so that's definitely the case here. IMO SSD drivers shouldn't be a requirement to run TDM smoothly I'd rather teach myself, and all the mappers, how to stay below one minute of initial loading time, like in Volta or Gatehouse. I'm not ruling out loading zones as well, they make much more sense in TDM than in Deadly Shadows.

[Bikerdude]

Yeah 5400 are to be avoided if at all possible, and now that SSD'S are so cheap there is no reason not to have them as boot/system drives.

 

Regarding the test, i can run the game of my 1Tb data deives thats a 7200rpm

[Oldjim]

Behind Closed Doors using 2.05 beta 2 minutes with a i5 4670 CPU and Nvidia 1060 GPU and 7200 hard drive