Bikerdude: - questions

[Bikerdude]

Ok installed win7x64 and it didnt help, but I have more mem to play with in other apps etc.

 

D3 is still crashing when reaching the 1.2-1.8Gb mark, the "/LARGEADDRESSAWARE" did not help at all! Does anyone one know how to recompile the doom3.exe so that its has that flag so I can work on large maps. The above asside I have found a work around the D3 malloc crashing by dmapping, then closing/restarting doom and then running map.

 

fyi the file sizes are as follows -

 

.ass_rat - 11.5mb

.ass32 - 9.94mb

.cm - 35mb

.map 19.3

.proc - 27.3mb

 

ps. When in-game its fine, the footprint drops too 310m. But something else I have noticed, when in the cathedral I have a 10fps drop compared with the cathedral as a separate map.

Edited October 13, 2010 by Bikerdude

[nbohr1more]

Did you also try this patching application, as was recommended previously...?

 

 

http://www.ntcore.com/4gb_patch.php

[Serpentine]

I explained this, you're not going to get more than 1.2-1.3gb allocated correctly for what doom3 wants, try anything you want, it's simply not going to happen.

Patching, changing OS and doing anything drastic like that will almost certainly not result in it being releasable anyway as it'll crash on everyone else's setups.

 

Edit:

The best thing you can do is cull enough of the map so that it loads, then profile what exactly is using the most and try to work out areas that :

• 
  Your map can be modified to reduce use
  TDM can be made more efficient
  Resources such as texture, model and such can be cleaned and optimized
  And some other stuff, perhaps doom3 related.
  

 

I can do this if you would like, but I will only try get numbers and not make changes/suggest as my time atm is limited for the next week and a bit. Tels might also have some ideas? If you'd like me to look into the usage etc, just give me a link to the package in PM nad i'll hack my way to a usable case.

Edited October 13, 2010 by Serpentine

[Bikerdude]

Did you also try this patching application, as was recommended previously...?http://www.ntcore.com/4gb_patch.php

Doh, I though this was a windows patch(Im getting old) - patching now...

[Bikerdude]

I explained this, you're not going to get more than 1.2-1.3gb allocated correctly for what doom3 wants,

 

Edit:

The best thing you can do is cull enough of the map so that it loads, then profile what exactly is using the most and try to work out areas that :

 

I can do this if you would like, but I will only try get numbers and not make changes/suggest as my time atm is limited for the next week and a bit.

 

i partly got sidetracked by all the advice and partly due to my thread being hijacked, as I have the map in layers I will save each layer as a prefab then export and remove and then have a map for each prefab so i can optimise as you suggested.

 

If I convert a lot of my very large FS to .ase models will this reduce my memory footprint...?

[Tels]

If I convert a lot of my very large FS to .ase models will this reduce my memory footprint...?

 

Since models are loaded only once, compared to brushwork-func-statics, it might.

 

But then, from your sizes above I don't see why D3 should use so much memory loading that map. I would like to try to load it under Linux here to see what it does.

[Bikerdude]

Well sweet Jesus

 

The 4GB file patched thing appeared to work - D3 peaked at 1.35GB and completed a single Dmap/Map compile session...!!! So next Im gonna have DR loaded(with said map@1Gb footprint) at the same time and see if it crashes...

 

I would still like to reduce the size of said maps to reduce load times for low ends users though. Tells shall i dop you a PM with my map so you can provide me with pointers of what to convert/reduce and where....

Edited October 13, 2010 by Bikerdude

[nbohr1more]

Congrats on pushing the boundaries Biker!

 

I hope this'll play on my rig

 

(...wonders how Dram and Greebo compile their mega-sized FM's???)

 

(...Linux probably doesn't have the same boundary issues...)

Edited October 13, 2010 by nbohr1more

[lost_soul]

If you guys are looking for places to trim the memory usage, you may want to re-encode some of the sound effects. I notice that many of the Ogg Vorbis ambiance are encoded at 499 kbps. That is way past any point at which you'll be able to hear a difference in sound quality. I've got all my Ogg Vorbis music at Q5 and it sounds just beautiful. That is roughly 160 kbps. Some of the mono Ogg files in TDM are also at 239 kbps. Perhaps if you use 192 kbps for ambient music and 160 for other sounds, it would trim memory usage quite a bit. I certainly wouldn't re-encode the existing sound files, but if you've got the WAVs it may be worth a try.

 

Also keep in mind that memory usage will only increase. If females are about to enter the TDM world, that is more voice sets you'll have to load at runtime, which means more memory will be used. Many commercial games use 64 kbps or so for vocals and sound effects. Doom 3 only used 35 kbps, which was pushing it a bit too far. You can hear compression artifacts in some of the sounds.

 

Bikerdude: if you're looking for a Linux tester, I would be happy to do it.

 

A re-encode of the resources could be made very simple by a Linux scripter. I'm not that guy, but I bet it would go something like

 

for $FILE in (ls *.wav)

do

oggenc -q 5 $FILE

done

Edited October 13, 2010 by lost_soul

[Bikerdude]

Ok, several things have transpired from this little experiment -

 

The mapper:

• D3 only peaks at 1.36GB when doing a dmap/map compile in a single session with a D3 cold start.
• D3 peaks at 800MB when doing a dmpa/map compile in a single session with D3 warm (already loaded)

The Player:

• D3 only peaks at 700Mb when doing a map, eg loading the mission

Questions:

 

Why does D3 use less memory on the second compile session..?

Why does the FPS drop all over the map when both maps are combined..? eg I am seeing a 10+ fps drop in the cathedral.

On the file sizes -

 

.ass_rat - 11.5mb

.ass32 - 9.94mb

.cm - 35mb

.map 19.3

.proc - 27.3mb

 

do the sizes indicate anything ..?

Edited October 13, 2010 by Bikerdude

[Serpentine]

If you guys are looking for places to trim the memory usage, you may want to re-encode some of the sound effects. I notice that many of the Ogg Vorbis ambiance are encoded at 499 kbps.

 

Yeah, I already noticed most of this.

I have a basic system that is now checking indexing and looking for issues with all images/audio/roq vids/models. It just needs and overhaul with regards to managing the metadata efficiently and finish the webUI for it, but this will only happen late Nov/Early Dec. It's aimed so that I/interested people can look for outliers in quality, either too low, too high or just badly formed (i,e we have a 20kb jpeg with 340kb of meta data...).

 

But for now these are not huge concerns, I just like to have ways to see 'everything' and try keep quality balanced

 

Re: BD

 

Sorry I didnt see that you were having trouble only with the dmapping, if you've managed to patch that and have it working so that dmap completes and its loadable on a normal executable that's fairly damn good my apologies!

 

As for the degraded performance, rememebr that with both maps joined, AI and such are still active even in visportaled areas, there's still a load of things that need to be taken care of even if you cant see them, most of this is CPU bound stuff which is Doom3's real weakness with large maps, memory and cpu. We should work out a list of doom3 commands and such to be able to get rough performance information that you can maybe get clues from. Perhaps the Quake4 performance HUD can be hacked to work with Doom3... Hmmm

[Tels]

Yeah, just PM me. I won't have time tomorrow night, tho, but I might have a look on friday night.

[Bikerdude]

Yeah, just PM me. I won't have time tomorrow night, tho, but I might have a look on friday night.

Ok I`ll pm you a link friday during that day, that gives me time to wkr on the map a little

[Bikerdude]

As for the degraded performance, rememebr that with both maps joined, AI and such are still active even in visportaled areas

 

So I assume there are no commands or ags I can use to deactivate Ai once Im in a certain section of the map..?

[nbohr1more]

Are you already using Interleaved Thinking optimizations?

 

 

Distance = 3000 example

max_interleave_think_dist 3000

 

[Bikerdude]

ye, what would be handy is to have an arg that could use the info location_settings zones to activate/deactivate Ai when I player enters/leaves a info zone etc...

Edited October 13, 2010 by Bikerdude

[nbohr1more]

I saw a thread somewhere where they mentioned scripting AI to path_sleep (?) if you are on the other side of a door... I will have to look again...

[Bikerdude]

Distance = 3000 example

max_interleave_think_dist 3000

I currently have them at -

 

max_interleave_think_dist - 1000

min_interleave_think_dist - 240

 

What are the recomended ranges to use in this instance where I am trying to reduce cpu overhead even more..?

 

Also what overheads to ragdoll Ai use..? or are they just treated as a model..?

Edited October 13, 2010 by Bikerdude

[Fidcal]

Remember you can set some AI neverDormant 0 which means they will freeze if not alert and they are outside the player area. This is suitable for things like static sentries but also many patrols. You just need to test them and if any look silly because they are always in the same place when you enter then remove it or possibly modify the path. It's surprisingly unnoticeable in many cases unless the player is deliberately looking for it. There were seven "neverdormant" "0" in Heart and nobody noticed - or at least, nobody commented.

[Baddcog]

yeah, unless the ai have a patrol that covers huge areas, having them freeze is no biggie. If it's a small patrol its likely nobody will be watching them, leaving, timing them, coming back... weird way to play the game.

 

and if it's a small patrol the player wont think its odd if they keep seeing the guard in that close vicinity.

 

1,000 units is a pretty decent distance, most views in my city aren't much more than that. you just have to be careful that the player is outside visibly bounds at any point in that patrol. I'd probably add a few hundred units to max sight distance just to give the ai some 'float' room.

[Bikerdude]

ah yes I forgot about that..

 

Adn regarding my files sizes -

 

.ass_rat - 11.5mb

.ass32 - 9.94mb

.cm - 35mb

.map 19.3

.proc - 27.3mb

 

do the sizes indicate anything bad ..?

[nbohr1more]

Your collision file is the largest so:

 

Collision Model Removal on some func_static Entities

For models that neither the player nor any AI nor any object can reach or interact with then some memory can be saved by setting the property:

 

noclipmodel 1

 

All moving game elements would then pass through the model so only normally use on inaccessible model features.

 

 

Any places that the player or AI cant reach yet have collision set?

[Bikerdude]

I have been doing that a lot in the cathedral, but it didnt occure to me, to do that in the other map as I had started that before cathedral and had been learning thing in cathedral that i didnt know to do in the previous map - so I shall get about doing that in the others parts of the maps etc.

 

I assume then the the .cm file is the collision models file and that's why is so bloody big at 35MB etc..

[Baddcog]

the sizes actually look pretty standard to me. cm is always the largest and i found with rift that 'over monster clipping' make it larger (i thought having fewer places for ai to go might shrink it, but i guess it still has to keep track of those areas. more like an on/off thing. being off is still data. 0 isn't smaller than 1, just different.

-----------

 

cm isn't collision models. those are in the models themselves afaik.

 

but it is level geometry collision model. so all brushes, monster clip... you might be abvle to shrink it some by say:

replace a picket fence (func_static-10 brush collision) with a monster clip and player clip brush (2 brush collision), but that's speculation.

 

whether or not model collision mesh data is counted i don't know.

[nbohr1more]

The town will be tougher 'cause you've got a lot of free-roam roofs etc... Good luck!

Edited October 13, 2010 by nbohr1more