Jump to content


Photo

I'd like to implement savegame compression


  • Please log in to reply
57 replies to this topic

#51 greebo

greebo

    Heroic Coder

  • Root
  • 16053 posts

Posted 02 January 2011 - 06:30 AM

I have no problem with making stgatilov a Contributor in the first place, but at the moment the server is down (I'm working on getting a dedicated server for us anyway).

Read-only SVN access for the public is mainly a problem of server bandwidth (right now we have a very narrow pipe as we're biting the bandwidth off sparhawk's personal DSL line). If the server situation turns out to be more relaxed in the future I don't see a problem in handling TDM sources the DarkRadiant way (or like all the other sourceforge code projects). For the actual darkmod repository that's a different matter as it's 1) huge and 2) we don't want to open all that stuff to the public immediately.

#52 stgatilov

stgatilov

    Lead Programmer

  • Active Developer
  • PipPipPip
  • 892 posts

Posted 05 January 2011 - 07:34 AM

Yes, please, and I could draw pretty pictures if you like.

I added a page on wiki: http://wiki.thedarkm...d_Doom3_linkage


By the way: Am I right that idClass and all derived classes are used only inside "gamex86.dll" ?



#53 greebo

greebo

    Heroic Coder

  • Root
  • 16053 posts

Posted 05 January 2011 - 07:41 AM

By the way: Am I right that idClass and all derived classes are used only inside "gamex86.dll" ?

Yes, I think so. I don't think it makes sense for the renderer to use that class, it's been designed for the game's purposes only.

#54 Tels

Tels

    Mod hero

  • Member
  • PipPipPipPipPip
  • 15024 posts

Posted 19 February 2011 - 04:55 PM

Looked today into the matter of why entity spawn takes such a long time. First thing was to add timers that show the time when you build with TIMING_BUILD defined. Here are sample entries that are shown after loading Heart.map (with over 3600 entities). The numbers are after doing this a few times to cache everything:

spawned 3650 34332.45 ms (9.41 ms per entity) (copy spawnargs 5207.17 (1.43 ms per entity))
114892 msec to load heart

spawned 3650 36315.66 ms (9.95 ms per entity) (copy spawnargs 6660.76 (1.82 ms per entity))
119957 msec to load heart

Apart from the shocking fact that each entity spawn takes 10ms, the copy of spawnargs also takes a looong time, nearly 18% of the entire time.

Here are some other numbers from spawning many thousand func_statics with a boletus model (the numbers seem to be particular high during map load, this is after map loading):

spawned 3000 13034.41 ms (4.34 ms per entity) (copy spawnargs 3157.81 (1.05 ms per entity))
spawned 21507.48 ms (4.07 ms per entity) (copy spawnargs 3937.92 (0.74 ms per entity))

The numbers vary quite a bit, but eve here it doesn't manage less than 4ms per spawn, and almost 25% of that is copy of spawnargs during spawn. Ouch.

So I tried a first stab at improving this. Instead of:

* copy the spawnargs requested (usally only a few, like "classname")
* then add all the defaults (SetDefault()), including the editor_ spawnargs
* then spawn entity, and then copy all the combined from steps 1 and 2 above, but skipping the editor_ spawnargs

it does now:

* copy the spawnargs requested (usally only a few, like "classname")
* then add all the defaults (SetDefault()), excluding the editor_ spawnargs
* then spawn entity, and then use TransferKeyValues()

That is quite a bit faster:

spawned 3650 18460.42 ms (5.06 ms per entity) (copy spawnargs 4467.40 (1.22 ms per entity))
101161 msec to load heart

spawned 3650 18305.25 ms (5.02 ms per entity) (copy spawnargs 4189.34 (1.15 ms per entity))
98456 msec to load heart

Why the copy timer only improves by 0.5ms, but the overall spawntimer by 4..5ms is something I cannot explain, tho.

However, on my system Heart.map now loads in roughly 98..100s vs. 115..120s as reported by D3. Not bad :)

I am pretty sure this can be improved alot, because after all, loading a savegame is nearly instantanous, while loading the map takes nearly 100 seconds.

Stgatilov, maybe you could run some profiling on this?
"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

"Remember: If the game lets you do it, it's not cheating." -- Xarax

#55 stgatilov

stgatilov

    Lead Programmer

  • Active Developer
  • PipPipPip
  • 892 posts

Posted 20 February 2011 - 10:02 AM

However, on my system Heart.map now loads in roughly 98..100s vs. 115..120s as reported by D3. Not bad :)

I am pretty sure this can be improved alot, because after all, loading a savegame is nearly instantanous, while loading the map takes nearly 100 seconds.

Stgatilov, maybe you could run some profiling on this?


I think you are optimizing the wrong thing. I started a new game in heart map on max difficulty and it took 287 sec according to console message. And most of that time was spent on reading from hard drive. Profiling shows that 6% of CPU time was consumed by Doom3.exe and 0.5% of CPU time was consumed by gamex86.dll. 82% of CPU time was not used at all. Most of the time was spent on reading all the pk4-s from the hard drive I think, unpacking them and etc.

If you want to decrease this time you should think about rearranging files inside pk4 probably (though I think it is not an option). Using better coimpression on assets is one of the proper ways to decrease map load time.

If you need to decrease entity spawning time then you should profile another place where entity spawning is a bottleneck.

#56 Tels

Tels

    Mod hero

  • Member
  • PipPipPipPipPip
  • 15024 posts

Posted 20 February 2011 - 10:12 AM

I think you are optimizing the wrong thing. I started a new game in heart map on max difficulty and it took 287 sec according to console message. And most of that time was spent on reading from hard drive.

You need of course to profile it after the data is cached, not while the first run reads from harddrive :)

For me it takes 240 without caching (220 afterwards), and 120 with cached (100 afterwards). Just because some people (like me :) have a slow harddrive doesn't mean the mission should take so long to load, anyway.

Profiling shows that 6% of CPU time was consumed by Doom3.exe and 0.5% of CPU time was consumed by gamex86.dll. 82% of CPU time was not used at all. Most of the time was spent on reading all the pk4-s from the hard drive I think, unpacking them and etc.

If you want to decrease this time you should think about rearranging files inside pk4 probably (though I think it is not an option). Using better coimpression on assets is one of the proper ways to decrease map load time.

If you need to decrease entity spawning time then you should profile another place where entity spawning is a bottleneck.


Maybe you did not read my entire post - I did :) I used a SEED entity to spawn about 8000 entities and the timing there indicates that spawning a func static takes about 4ms (e.g. 1/4 of a frame!).

Also, did you profile the code before or after my change? If you profiled it afterwards, a lot of the time spent to spawn an entity might already have been gone.
"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

"Remember: If the game lets you do it, it's not cheating." -- Xarax

#57 stgatilov

stgatilov

    Lead Programmer

  • Active Developer
  • PipPipPip
  • 892 posts

Posted 20 February 2011 - 10:42 AM

You need of course to profile it after the data is cached, not while the first run reads from harddrive :)



What do you mean by "data is cached"? Of course loading the same files from hard drive becomes a bit faster the second time, but still. 1.25Gb of data is loaded during starting the new game (measured with process explorer). No cache can handle this huge amount of data! Even if you load the second/third time the hard drive work is still very slow.

I run it again with profiling disabled and it got about 200 sec. What does it mean? Nothing :laugh: It means that hard drive is bottleneck. Do you have P2P/torrent clients enabled? Turn them off and you'll get less start times. Also you may try to start a copy of 4Gb file somewhere on the drive and start a new map in TDM at the same time. You'll likely get much slower loading.

Just because some people (like me :) have a slow harddrive doesn't mean the mission should take so long to load, anyway.



It is so. Loading thousands of files more than Gb total with random access patterns is guilty for big mission load times. Decrease size of data loaded. Or rewrite Doom3 pk4 loading routines. Probably you'll get better start times.



Maybe you did not read my entire post - I did :) I used a SEED entity to spawn about 8000 entities and the timing there indicates that spawning a func static takes about 4ms (e.g. 1/4 of a frame!).



That's much more interesting! 4ms per entity is long. Can you name any use case which leads to lots of pure entity spawnings in some times? I can start profiling easily for it then if you like. (I do not believe profiling results unless they are accumulated for several seconds of runtime :D ).

Also, did you profile the code before or after my change? If you profiled it afterwards, a lot of the time spent to spawn an entity might already have been gone.

Fresh trunk version! Yes, your changes are already used :)

#58 Tels

Tels

    Mod hero

  • Member
  • PipPipPipPipPip
  • 15024 posts

Posted 20 February 2011 - 11:25 AM

What do you mean by "data is cached"? Of course loading the same files from hard drive becomes a bit faster the second time, but still. 1.25Gb of data is loaded during starting the new game (measured with process explorer). No cache can handle this huge amount of data! Even if you load the second/third time the hard drive work is still very slow.

I run it again with profiling disabled and it got about 200 sec. What does it mean? Nothing :laugh: It means that hard drive is bottleneck. Do you have P2P/torrent clients enabled? Turn them off and you'll get less start times. Also you may try to start a copy of 4Gb file somewhere on the drive and start a new map in TDM at the same time. You'll likely get much slower loading.


I think you did not read my first post, here are the numbers again:

* uncached load: 240 seconds, from this are 36 seconds entity spawn time alone
* cached load: 120 seconds, from this are 36 seconds entity spawn time alone

After my optimization:

* uncached load: 220 seconds, from this are ~18 seconds entity spawn time alone
* cached load: 100 seconds, from this are ~18 seconds entity spawn time alone

Or in other words, forget the HD, forget the map loading, I care about entity spawn time.

It is so. Loading thousands of files more than Gb total with random access patterns is guilty for big mission load times. Decrease size of data loaded. Or rewrite Doom3 pk4 loading routines. Probably you'll get better start times.


I do not care about map loading time, that it got faster was a side-effect! I care about entity spawn time!

That's much more interesting! 4ms per entity is long.


Er, well, that's what I said from the begin :D

Can you name any use case which leads to lots of pure entity spawnings in some times? I can start profiling easily for it then if you like. (I do not believe profiling results unless they are accumulated for several seconds of runtime :D ).


* When you put in "combine" "0" at a seed, it manages each entity on its own, and constantly culls/spawns entities. In this case it might spawn several dozend entities during a frame, and this makes the game stutter. Technically, this can be avoided by combining entities, but atm this is only possible for func_statics. If you have for instance func_emitters (like fog), you get stuttering when they spawn when the player gets close.
* multiple things spawn entities, f.i. a broken arrow spawns at least 3: two flinders (broken back/broken end) and an arrow result. It might also spawn a smoke particle. This can lead to stuttering if many entities spawn in short time frames (f.i. break 10 bottles => 20 entities spawn).

If you want, I can send you a map which forces massive entity spawning, so you can profile it better.

Fresh trunk version! Yes, your changes are already used :)


That would explain why you saw so little time used during map load :)
"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

"Remember: If the game lets you do it, it's not cheating." -- Xarax




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users