Jump to content
The Dark Mod Forums

I'd like to implement savegame compression


stgatilov

Recommended Posts

I have no problem with making stgatilov a Contributor in the first place, but at the moment the server is down (I'm working on getting a dedicated server for us anyway).

 

Read-only SVN access for the public is mainly a problem of server bandwidth (right now we have a very narrow pipe as we're biting the bandwidth off sparhawk's personal DSL line). If the server situation turns out to be more relaxed in the future I don't see a problem in handling TDM sources the DarkRadiant way (or like all the other sourceforge code projects). For the actual darkmod repository that's a different matter as it's 1) huge and 2) we don't want to open all that stuff to the public immediately.

Link to comment
Share on other sites

  • Replies 57
  • Created
  • Last Reply

Top Posters In This Topic

By the way: Am I right that idClass and all derived classes are used only inside "gamex86.dll" ?

Yes, I think so. I don't think it makes sense for the renderer to use that class, it's been designed for the game's purposes only.

Link to comment
Share on other sites

  • 1 month later...

Looked today into the matter of why entity spawn takes such a long time. First thing was to add timers that show the time when you build with TIMING_BUILD defined. Here are sample entries that are shown after loading Heart.map (with over 3600 entities). The numbers are after doing this a few times to cache everything:

 

spawned 3650 34332.45 ms (9.41 ms per entity) (copy spawnargs 5207.17 (1.43 ms per entity))
114892 msec to load heart

spawned 3650 36315.66 ms (9.95 ms per entity) (copy spawnargs 6660.76 (1.82 ms per entity))
119957 msec to load heart

 

Apart from the shocking fact that each entity spawn takes 10ms, the copy of spawnargs also takes a looong time, nearly 18% of the entire time.

 

Here are some other numbers from spawning many thousand func_statics with a boletus model (the numbers seem to be particular high during map load, this is after map loading):

 

spawned 3000 13034.41 ms (4.34 ms per entity) (copy spawnargs 3157.81 (1.05 ms per entity))
spawned 21507.48 ms (4.07 ms per entity) (copy spawnargs 3937.92 (0.74 ms per entity))

 

The numbers vary quite a bit, but eve here it doesn't manage less than 4ms per spawn, and almost 25% of that is copy of spawnargs during spawn. Ouch.

 

So I tried a first stab at improving this. Instead of:

 

* copy the spawnargs requested (usally only a few, like "classname")

* then add all the defaults (SetDefault()), including the editor_ spawnargs

* then spawn entity, and then copy all the combined from steps 1 and 2 above, but skipping the editor_ spawnargs

 

it does now:

 

* copy the spawnargs requested (usally only a few, like "classname")

* then add all the defaults (SetDefault()), excluding the editor_ spawnargs

* then spawn entity, and then use TransferKeyValues()

 

That is quite a bit faster:

 

spawned 3650 18460.42 ms (5.06 ms per entity) (copy spawnargs 4467.40 (1.22 ms per entity))
101161 msec to load heart

spawned 3650 18305.25 ms (5.02 ms per entity) (copy spawnargs 4189.34 (1.15 ms per entity))
98456 msec to load heart

 

Why the copy timer only improves by 0.5ms, but the overall spawntimer by 4..5ms is something I cannot explain, tho.

 

However, on my system Heart.map now loads in roughly 98..100s vs. 115..120s as reported by D3. Not bad :)

 

I am pretty sure this can be improved alot, because after all, loading a savegame is nearly instantanous, while loading the map takes nearly 100 seconds.

 

Stgatilov, maybe you could run some profiling on this?

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

However, on my system Heart.map now loads in roughly 98..100s vs. 115..120s as reported by D3. Not bad :)

 

I am pretty sure this can be improved alot, because after all, loading a savegame is nearly instantanous, while loading the map takes nearly 100 seconds.

 

Stgatilov, maybe you could run some profiling on this?

 

I think you are optimizing the wrong thing. I started a new game in heart map on max difficulty and it took 287 sec according to console message. And most of that time was spent on reading from hard drive. Profiling shows that 6% of CPU time was consumed by Doom3.exe and 0.5% of CPU time was consumed by gamex86.dll. 82% of CPU time was not used at all. Most of the time was spent on reading all the pk4-s from the hard drive I think, unpacking them and etc.

 

If you want to decrease this time you should think about rearranging files inside pk4 probably (though I think it is not an option). Using better coimpression on assets is one of the proper ways to decrease map load time.

 

If you need to decrease entity spawning time then you should profile another place where entity spawning is a bottleneck.

Link to comment
Share on other sites

I think you are optimizing the wrong thing. I started a new game in heart map on max difficulty and it took 287 sec according to console message. And most of that time was spent on reading from hard drive.

You need of course to profile it after the data is cached, not while the first run reads from harddrive :)

 

For me it takes 240 without caching (220 afterwards), and 120 with cached (100 afterwards). Just because some people (like me :) have a slow harddrive doesn't mean the mission should take so long to load, anyway.

 

Profiling shows that 6% of CPU time was consumed by Doom3.exe and 0.5% of CPU time was consumed by gamex86.dll. 82% of CPU time was not used at all. Most of the time was spent on reading all the pk4-s from the hard drive I think, unpacking them and etc.

 

If you want to decrease this time you should think about rearranging files inside pk4 probably (though I think it is not an option). Using better coimpression on assets is one of the proper ways to decrease map load time.

 

If you need to decrease entity spawning time then you should profile another place where entity spawning is a bottleneck.

 

Maybe you did not read my entire post - I did :) I used a SEED entity to spawn about 8000 entities and the timing there indicates that spawning a func static takes about 4ms (e.g. 1/4 of a frame!).

 

Also, did you profile the code before or after my change? If you profiled it afterwards, a lot of the time spent to spawn an entity might already have been gone.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

You need of course to profile it after the data is cached, not while the first run reads from harddrive :)

 

 

 

What do you mean by "data is cached"? Of course loading the same files from hard drive becomes a bit faster the second time, but still. 1.25Gb of data is loaded during starting the new game (measured with process explorer). No cache can handle this huge amount of data! Even if you load the second/third time the hard drive work is still very slow.

 

I run it again with profiling disabled and it got about 200 sec. What does it mean? Nothing :laugh: It means that hard drive is bottleneck. Do you have P2P/torrent clients enabled? Turn them off and you'll get less start times. Also you may try to start a copy of 4Gb file somewhere on the drive and start a new map in TDM at the same time. You'll likely get much slower loading.

 

 

Just because some people (like me :) have a slow harddrive doesn't mean the mission should take so long to load, anyway.

 

 

 

It is so. Loading thousands of files more than Gb total with random access patterns is guilty for big mission load times. Decrease size of data loaded. Or rewrite Doom3 pk4 loading routines. Probably you'll get better start times.

 

 

 

Maybe you did not read my entire post - I did :) I used a SEED entity to spawn about 8000 entities and the timing there indicates that spawning a func static takes about 4ms (e.g. 1/4 of a frame!).

 

 

 

That's much more interesting! 4ms per entity is long. Can you name any use case which leads to lots of pure entity spawnings in some times? I can start profiling easily for it then if you like. (I do not believe profiling results unless they are accumulated for several seconds of runtime :D ).

 

Also, did you profile the code before or after my change? If you profiled it afterwards, a lot of the time spent to spawn an entity might already have been gone.

Fresh trunk version! Yes, your changes are already used :)

Link to comment
Share on other sites

What do you mean by "data is cached"? Of course loading the same files from hard drive becomes a bit faster the second time, but still. 1.25Gb of data is loaded during starting the new game (measured with process explorer). No cache can handle this huge amount of data! Even if you load the second/third time the hard drive work is still very slow.

 

I run it again with profiling disabled and it got about 200 sec. What does it mean? Nothing :laugh: It means that hard drive is bottleneck. Do you have P2P/torrent clients enabled? Turn them off and you'll get less start times. Also you may try to start a copy of 4Gb file somewhere on the drive and start a new map in TDM at the same time. You'll likely get much slower loading.

 

I think you did not read my first post, here are the numbers again:

 

* uncached load: 240 seconds, from this are 36 seconds entity spawn time alone

* cached load: 120 seconds, from this are 36 seconds entity spawn time alone

 

After my optimization:

 

* uncached load: 220 seconds, from this are ~18 seconds entity spawn time alone

* cached load: 100 seconds, from this are ~18 seconds entity spawn time alone

 

Or in other words, forget the HD, forget the map loading, I care about entity spawn time.

 

It is so. Loading thousands of files more than Gb total with random access patterns is guilty for big mission load times. Decrease size of data loaded. Or rewrite Doom3 pk4 loading routines. Probably you'll get better start times.

 

I do not care about map loading time, that it got faster was a side-effect! I care about entity spawn time!

 

That's much more interesting! 4ms per entity is long.

 

Er, well, that's what I said from the begin :D

 

Can you name any use case which leads to lots of pure entity spawnings in some times? I can start profiling easily for it then if you like. (I do not believe profiling results unless they are accumulated for several seconds of runtime :D ).

 

* When you put in "combine" "0" at a seed, it manages each entity on its own, and constantly culls/spawns entities. In this case it might spawn several dozend entities during a frame, and this makes the game stutter. Technically, this can be avoided by combining entities, but atm this is only possible for func_statics. If you have for instance func_emitters (like fog), you get stuttering when they spawn when the player gets close.

* multiple things spawn entities, f.i. a broken arrow spawns at least 3: two flinders (broken back/broken end) and an arrow result. It might also spawn a smoke particle. This can lead to stuttering if many entities spawn in short time frames (f.i. break 10 bottles => 20 entities spawn).

 

If you want, I can send you a map which forces massive entity spawning, so you can profile it better.

 

Fresh trunk version! Yes, your changes are already used :)

 

That would explain why you saw so little time used during map load :)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recent Status Updates

    • Ansome

      Finally got my PC back from the shop after my SSD got corrupted a week ago and damaged my motherboard. Scary stuff, but thank goodness it happened right after two months of FM development instead of wiping all my work before I could release it. New SSD, repaired Motherboard and BIOS, and we're ready to start working on my second FM with some added version control in the cloud just to be safe!
      · 0 replies
    • Petike the Taffer  »  DeTeEff

      I've updated the articles for your FMs and your author category at the wiki. Your newer nickname (DeTeEff) now comes first, and the one in parentheses is your older nickname (Fieldmedic). Just to avoid confusing people who played your FMs years ago and remember your older nickname. I've added a wiki article for your latest FM, Who Watches the Watcher?, as part of my current updating efforts. Unless I overlooked something, you have five different FMs so far.
      · 0 replies
    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
×
×
  • Create New...