Jump to content
The Dark Mod Forums

Better compression for TDM:


peter_spy

Recommended Posts

This is more like a wish for TDM itself, rather than DR, but what do you think about adding support for other compression formats to TDM? Right now we can only use the old "deflate" method which is rather slow and inefficient. 7-Zip offers several types of compression for zip files, and here is a small comparison for bigger TDM missions out there:

Zip_Compression_Rates.jpg

 

PK4s are original files, compression methods are in suffixes. As you see, the difference can be substantial, and those LZMA and PPMd archives also compress much faster. It took 3 minutes for Volta to compress with the deflate method vs 50 seconds with PPMd and LZMA.

  • Like 3
Link to comment
Share on other sites

Thanks Biker, it looks better as a separate topic. Naturally, we have to see how this affects loading times. Since first loading is usually a slog, a few extra seconds won't make that much of a difference. Anyway, I'll be happy to test that feature.

  • Like 1
Link to comment
Share on other sites

Now you're giving me a guilt trip. Not too guilty though. Trying to get RGTC support working is difficult and

I will return to my attempts when 2.05 is stabilized.

Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Link to comment
Share on other sites

Don't let me pressure you. Everything you contribute is appreciated.

 

I suspect the real culprit in both loading and storage space are all the plain text files that require parsing. Here's an example ripped straight out of the training mission...

 

 

model { /* name = */ "func_static_200" /* numSurfaces = */ 0

}

 

Here we have a block from a .proc file. Notice the use of C style block comments in a file that content authors are unlikely to ever inspect. And what data is being conveyed? That the model for func_static_200 contains zero surfaces. 63 bytes to say essentially nothing.

  • Like 1
Link to comment
Share on other sites

It took 3 minutes for Volta to compress with the deflate method vs 50 seconds with PPMd and LZMA.

 

I'm not too familiar with the term "deflating" but my computer can uncompress the volta pk4 archive for example in literally 2 seconds.

 

Correct me if i'm wrong but isn't TDM basically doing the same thing when it opens an archive to play the mission?

Link to comment
Share on other sites

Yes, it can uncompress easily, it's the compression that takes a long time and it's not too efficient.

 

I can only guess that TDM loads the whole FM into memory, that's why it takes so long at first. Level streaming was only available around first Gears of War and UE3, so we can't rely on anything like that. Loading zones, anyone? :D

Link to comment
Share on other sites

The decompression engine in TDM is I image very old, compared what what you used above that took 2 seconds..

 

I was moreso thinking hardware differences rather than software. I haven't tested if hardware makes an improvement to TDM loading times though, as I have a fairly high end PC and it still takes a long time to load.

 

It would be interesting to look into though if an improvement in how TDM decompresses PK4 files translates to improved loading times because they take quite a long time currently.

Link to comment
Share on other sites

I think it only makes sense to spend time on the compression changes if we can decompress in a separate thread.

It's the only way we could improve both decompression speed and compression ratio.

And improving one at the expense of the other probably leads to no added value.

Link to comment
Share on other sites

It would be interesting to look into though if an improvement in how TDM decompresses PK4 files translates to improved loading times because they take quite a long time currently.

Well one test that springs to mind, is to do a mission load time test with uncompressed versus compressed, from a fresh start of TDM. As this would show how and if the long load times are being held up by decompression of the .pk4...

Link to comment
Share on other sites

I did only two tests, but the results so far are inconclusive. Volta usually needs 51-55 seconds on my laptop, Volta in uncompressed (Store mode) zip, around the same time. But with CoO: Behind Closed Doors, the difference was substantial. The compressed version takes whopping 2:29 to load. But, in a Store mode the loading time is reduced to 1:45 to (still very long, but hey). Wondering what that may be, I checked the content of both pk4s. Seems like there's a huge difference in the number of files. Volta has like less than 300 files, BCD has over 900 (!). That definitely prolongs the decompression time.

Edited by Judith
  • Like 1
Link to comment
Share on other sites

Two more missions:

 

penny3 Normal mode - 6:53!!!
penny3 Store mode - 2:49

gatehouse normal - 0:59
gatehouse Store mode - 0:31

 

Now, either my laptop's hardware isn't properly supported in TDM, or there's something very wrong here. Almost 7 minutes of loading time is just abysmal. Surely that has something to do with slower HDD (5400 vs desktop's 7200 rpm), but I think the difference won't be huge. I can see by the HDD light that it isn't used much while loading. Maybe it's a slow bus between mobo and the gpu, or gpu and vram. That said, I can comfortably play Dishonored 1 on max settings, and Thief Reboot with most options on high.

 

I will test those on my desktop too. I just upgraded to from GTX 650Ti to GTX 1060, so I got a huge boost in GPU power. But I kind of doubt it will change much in TDM though.

 

Edit: Actually, it seems it may have something to do with laptop's weird dual gfx card thing. Even though I've already chosen the nvidia card as preferred gpu in the driver menu, setting the com_videoRAM in .cfg to 2048 actually made loading this slow. Setting it back to 128 reduced PD3 loading time to ~3 minutes. The weird thing is, it only happens in some missions.

Edited by Judith
Link to comment
Share on other sites

Good idea. Also, I just read that 5400 rpm drives are only much slower than 7200s when reading multiple files, so that's definitely the case here. IMO SSD drivers shouldn't be a requirement to run TDM smoothly ;) I'd rather teach myself, and all the mappers, how to stay below one minute of initial loading time, like in Volta or Gatehouse. I'm not ruling out loading zones as well, they make much more sense in TDM than in Deadly Shadows.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recent Status Updates

    • Petike the Taffer  »  DeTeEff

      I've updated the articles for your FMs and your author category at the wiki. Your newer nickname (DeTeEff) now comes first, and the one in parentheses is your older nickname (Fieldmedic). Just to avoid confusing people who played your FMs years ago and remember your older nickname. I've added a wiki article for your latest FM, Who Watches the Watcher?, as part of my current updating efforts. Unless I overlooked something, you have five different FMs so far.
      · 0 replies
    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
    • OrbWeaver

      I like the new frob highlight but it would nice if it was less "flickery" while moving over objects (especially barred metal doors).
      · 4 replies
×
×
  • Create New...