Jump to content
The Dark Mod Forums

What are current max limits for the following:


Bikerdude

Recommended Posts

Excellent summary. I'll contribute some research on the options too.

 

  1. [*]
Remove unnec. entities after map loading. This might be difficult, because some script could refer to any entity, and then it must be kept. But likely candidates are portal info, difficulty info, shop info, starting inventory (I believe these are already removed). Another difficulty is that removing the entity might not be enough, the entity must not even be spawned - otherwise it uses up one entity slot during frame #1 and the map would still reach the limit.

 

We need to take a look again at the code that's meant to inline func statics. That one sticks out for me as a potentially huge gain with little effort. The code is already there, but it needs fixing. It's not needed for existing maps, just new ones.

 

Does entity spawning not re-use empty index slots then? I'm sure we could make it do so if not.

Link to comment
Share on other sites

  • Replies 63
  • Created
  • Last Reply

Top Posters In This Topic

I vote for this as its been done before, but that said once I have had TDM open for a while with the above map and done a few DMAP runs I occasioanly get a malloc crash whioch on smallers maps I don't. maybe at the same time we make darkmod.exe 64bit native with 32bit legacy mode.

 

Yeah, that would be the way forward. However, larger maps, larger entity counts and so will also put a strain on the entire editing pipeline (you need a beefy PC just to open the map in DR) and the player. 64bit doesn't help if you have less then 4Gb memory in the PC, it might actually hut, because it uses a bit more memory.

 

So, we also need to somehow try to reduce the memory footprint as well as the loading size.

 

I remember a few years back stgatilov removed the "editor_" spawnargs from entities in memory, this massively reduced the savegame size (because our documentation effort added thousands of these to entities :D

 

Maybe we can look at slimming down entities in general in the engine. If one entity uses a few Kbyte less memory, and the parts of the engine that try to find entities don't loop through all of them multiple times per frame, then doubling or trippling the limit won't be a problem.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Cycling through entities:

 

There are 3 lists of entities maintained in gameLocal:

  • entities -- this is a fixed-size array, currently 8192 entities
  • activeEntities -- a variable-sized linked list
  • spawnedEntities -- also a variable-sized linked list

Most code that cycles through every entity does it through the first one, and it means it checks 8192 slots every time even if you have only 100 entities in your map. But that's fast because it's just a single block of 8k pointers. Cycling through the linked lists is slower because the data you're checking will probably be scattered in different blocks of memory, but it involves fewer steps... and come to think of it, if you want to do anything or check anything with the entities in the fixed-size list, again you have to access the scattered memory that holds the details for each entity.

 

There are lots of places that allocate a temporary 8k block of memory to store lists of entities, for example the check for touching entities that you mentioned earlier. Those would all become 16kb allocations if we increase the entity limit. That'd probably be the main impact of increasing the entity limit, and (for all I know) it might not turn out to be a barrier at all. TDM doesn't tend to take up too much memory on my system even with a big map -- it certainly doesn't push anywhere near the 3gb range where we'd start to get problems.

 

Fixing the "inline" spawnarg for func statics would also give Biker an immediate relief from this problem.

Link to comment
Share on other sites

We need to take a look again at the code that's meant to inline func statics. That one sticks out for me as a potentially huge gain with little effort. The code is already there, but it needs fixing. It's not needed for existing maps, just new ones.

 

I'm not sure it would really help. Sure, it removes the entities, but it makes them world_spawn, which is a different beast and I'm not sure that would work for huge maps. It depends on how they are converted to world_spawn. Just added to the "render list" or do they divide world_spawn faces, too?

 

Does entity spawning not re-use empty index slots then? I'm sure we could make it do so if not.

 

I think it does, but I'm not sure. This routine is the likely place and it looks like it doesn't do wrap around idGameLocal::UnregisterEntity does indeed set firstFreeIndex back.

 

void idGameLocal::RegisterEntity( idEntity *ent ) {
int spawn_entnum;

if ( spawnCount >= ( 1 << ( 32 - GENTITYNUM_BITS ) ) ) {
	Error( "idGameLocal::RegisterEntity: spawn count overflow" );
}

if ( !spawnArgs.GetInt( "spawn_entnum", "0", spawn_entnum ) ) {
	while( entities[firstFreeIndex] && firstFreeIndex < ENTITYNUM_MAX_NORMAL ) {
		firstFreeIndex++;
	}
	if ( firstFreeIndex >= ENTITYNUM_MAX_NORMAL ) {
		Error( "no free entities" );
	}
	spawn_entnum = firstFreeIndex++;
}

entities[ spawn_entnum ] = ent;
spawnIds[ spawn_entnum ] = spawnCount++;
ent->entityNumber = spawn_entnum;
ent->spawnNode.AddToEnd( spawnedEntities );

// this will also have the effect of spawnArgs.Clear() at the same time:
ent->spawnArgs.TransferKeyValues( spawnArgs );

if ( spawn_entnum >= num_entities ) {
	num_entities++;
}
}

 

Interesting is the support for "spawn_entnum", I'd say we should remove that. Setting the entity number via spawnargs - ugh. That exposes internal stuff to the outside world without real need.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Nice one guys and on a related note is there a console command I can use to see how many entites are being used in-game..?

I have one that gives a count and breakdown by class that I use in testing. I'll commit it to svn this eve so you can use it. It's only an extra console command, should be an uncontroversial feature proposal :)

Link to comment
Share on other sites

Cycling through entities:

 

There are 3 lists of entities maintained in gameLocal:

  • entities -- this is a fixed-size array, currently 8192 entities
  • activeEntities -- a variable-sized linked list
  • spawnedEntities -- also a variable-sized linked list

Most code that cycles through every entity does it through the first one, and it means it checks 8192 slots every time even if you have only 100 entities in your map. But that's fast because it's just a single block of 8k pointers. Cycling through the linked lists is slower because the data you're checking will probably be scattered in different blocks of memory, but it involves fewer steps... and come to think of it, if you want to do anything or check anything with the entities in the fixed-size list, again you have to access the scattered memory that holds the details for each entity.

 

There are lots of places that allocate a temporary 8k block of memory to store lists of entities, for example the check for touching entities that you mentioned earlier. Those would all become 16kb allocations if we increase the entity limit. That'd probably be the main impact of increasing the entity limit, and (for all I know) it might not turn out to be a barrier at all. TDM doesn't tend to take up too much memory on my system even with a big map -- it certainly doesn't push anywhere near the 3gb range where we'd start to get problems.

 

Yes, I should have mentioned that there are different problems when raising the entity limit:

  1. maps with lots of entities (> 8000) will use a lot of memory. But only a part of that might be due to the entities itself, textures&models probably eat a lot more
  2. maps with lots of entities (> 8000) will run through XK slows instead of only 8K. That might be no problem at all (16K index accessed might be nothing for a modern CPU).
  3. maps with few entities (<1000) might end up with a lot of temp. allocations (I wasn't aware of that problem, and it might be fixable - allocated only HIGHEST_CURRENT_ENTITY_ID slots instead of the full block)
  4. maps with little entities, but constant entity spawning and respaning might have num_entities higher than it actually is - that is why the code seems to always run from 0 .. num_entities,but then checks that each entry is actually existing and valid.

 

Interstingly enough, RegisterEntity does num_entities++ but UnregisterEntity does not --? Likewise, I'm not sure RegisterEntity handles the increment always properly.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

I'm not sure it would really help. Sure, it removes the entities, but it makes them world_spawn, which is a different beast and I'm not sure that would work for huge maps. It depends on how they are converted to world_spawn. Just added to the "render list" or do they divide world_spawn faces, too?

I think it's a purely a spawn-time thing so not used during dmap but I'll check.

 

I think it does, but I'm not sure. This routine is the likely place and it looks like it doesn't do wrap around idGameLocal::UnregisterEntity does indeed set firstFreeIndex back.

Cool, so there's hope for "inline".

 

Interesting is the support for "spawn_entnum", I'd say we should remove that. Setting the entity number via spawnargs - ugh. That exposes internal stuff to the outside world without real need.

!!!!! Yes, a recipe for disaster that one. I'll check maps and defs to make sure we don't use it for anything. A few things like world do have fixed entity slots, but it'd only work if you can guarantee the specified entity will spawn first else it mightoverwrite an existing entity.

Link to comment
Share on other sites

void idGameLocal::RegisterEntity( idEntity *ent ) {
int spawn_entnum;

if ( spawnCount >= ( 1 << ( 32 - GENTITYNUM_BITS ) ) ) {
	Error( "idGameLocal::RegisterEntity: spawn count overflow" );
}

 

Oh boy... increment, but never decrement (or at least my grep-foo didn't find it):

 

   entities[ spawn_entnum ] = ent;
spawnIds[ spawn_entnum ] = spawnCount++;

 

So, constantly spawning entities, then removing them, then respawning (what SEED can do) would after a while crash the game?

 

Plus, if we increase GENTITYNUM_BITS from 13 to 14 or 15, the code above does (32 - GENTITYNUM_BITS), which means it would have a lower limit? (e.g. now it is 1 << (32 -13) == 1<<19, afterwards it is 1 << (32 -14) == 1<<18?

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Cool, so there's hope for "inline".

 

Not sure if these two things are related. If the entities are inlined at "map load", then they are never spawned in the first place, so wrap-around does not matter. If they are inlined after spawn, the map would still crash unless the inlined entities + everything else > LIMIT.

 

So, yeah, I think inlining could help. OTOH, care must be taken to not inline func_statics that are refered to by something later. E.g. anything with a LOD stage or watched over by SEED or having a link (e.g. another entity uses it as "target spawnarg") or it is refered to by a script. I'm not sure it is that easy to discern.

 

Maybe if the func_statics have an "inline" spawnarg, but then it would only be for new maps.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

That's what I was thinking too. We only need it for new maps, existing ones already fit in the limit. We could do a good job of finding out automatically whether an entity is tweaked after spawn, but not a perfect one because of the dynamic FindEntity functions. But enabling it for new maps only would solve the problem.

 

New shadow lights on existing lights is the only exception. We'll have to make a point of testing the maps that are close to the limit in 2.03, and if worse comes to worst, stop it happening for those.

Link to comment
Share on other sites

btw, we fixed inlining in our engine. In stock Doom 3 inlining works, but it doesn't remove inlined entity and it doesn't keep collision model for it. As I recall the fix was trivial since the functionality is already in the engine. I wish I was a programmer to tell you more :)

 

The upside of inlining is obvious - less entities. The downside is that you can no longer reference the model. So it's useful for static, non-animated models, such as architectural detail, rocks, etc.

Link to comment
Share on other sites

btw, we fixed inlining in our engine. In stock Doom 3 inlining works, but it doesn't remove inlined entity and it doesn't keep collision model for it. As I recall the fix was trivial since the functionality is already in the engine. I wish I was a programmer to tell you more :)

 

The upside of inlining is obvious - less entities. The downside is that you can no longer reference the model. So it's useful for static, non-animated models, such as architectural detail, rocks, etc.

Glad to hear you got it working, and yes that's how I expected it would work. Inlining all genuine statics like lamp posts and bookcases would solve the problem. Does your project have a public repo where I can check out the patch?

Link to comment
Share on other sites

Nice one guys and on a related note is there a console command I can use to see how many entites are being used in-game..?

There was already a "listEntities" command that print a table of entities and a total at the bottom.

 

I've added a "countEntities" to go with it, which prints the counts by class. Probably more useful in this situation than "listEntities".

Link to comment
Share on other sites

Oh boy... increment, but never decrement (or at least my grep-foo didn't find it):

 

entities[ spawn_entnum ] = ent;
spawnIds[ spawn_entnum ] = spawnCount++;

 

So, constantly spawning entities, then removing them, then respawning (what SEED can do) would after a while crash the game?

 

Plus, if we increase GENTITYNUM_BITS from 13 to 14 or 15, the code above does (32 - GENTITYNUM_BITS), which means it would have a lower limit? (e.g. now it is 1 << (32 -13) == 1<<19, afterwards it is 1 << (32 -14) == 1<<18?

 

I've been puzzling over this too. It seems this weird rule is in place because idEntityPtr wants to be a single 32-bit int like a proper 32-bit pointer, but it has to hold two pieces of information: the entityNum AND the spawnID. So a certain number of bits get allocated to the entityNum (currently 13, enough for 8192 entities) and the rest get allocated to the spawnID (currently 19, enough for 500k spawnings).

 

If we were to double the entity limit, we'd have 16384 entity limit but only a quarter of a million spawnings allowed during a play session. That wouldn't be a problem in most maps, but SEED or imaginative scripts could end up exceeding it.

 

The merged entityNum-spawnID field is used only for synchronising the game over a network, i.e multiplayer mode. The rest of the game uses gameLocal.spawnIds whch devotes a full 32 bits to each spawnID.

Link to comment
Share on other sites

I've been puzzling over this too. It seems this weird rule is in place because idEntityPtr wants to be a single 32-bit int like a proper 32-bit pointer, but it has to hold two pieces of information: the entityNum AND the spawnID. So a certain number of bits get allocated to the entityNum (currently 13, enough for 8192 entities) and the rest get allocated to the spawnID (currently 19, enough for 500k spawnings).

 

If we were to double the entity limit, we'd have 16384 entity limit but only a quarter of a million spawnings allowed during a play session. That wouldn't be a problem in most maps, but SEED or imaginative scripts could end up exceeding it.

 

The merged entityNum-spawnID field is used only for synchronising the game over a network, i.e multiplayer mode. The rest of the game uses gameLocal.spawnIds whch devotes a full 32 bits to each spawnID.

 

Maybe we should get rid of the crazy netsync scheme (at least for spawnids) and replace it by two 32 bit ints. It's not that it would be used, anyway :D

 

The spawn limit was formerly 2 million when we had 2048 entities as limit (that seems outright tiny now). I guess very long running network play sessions would still be safe with a 2 million limit, but as we are now down to 0.5mill and would be at 250K, or even 120K, this seems dangerous.

 

The interesting thing is to me, that the spawnId could also just wrap around. But I guess back then it was simpler to code and worked (unless you played more than a few hours on a map with constant respawning items).

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Speaking of feature proposals:

 

Should I try adding the posibilty of a having particle at an idLight? I guess a few spawnargs with offset, rotation and .ptr name would be enough.

 

In fact, I think we could even do something like a "multimodel" instead. Add support for "model2" and "model2_ofs" and "model2_rotate" and model3 and so on.

 

Each light with a model would then simply either present the model to the renderer, or add the particle to its particle emitting-list (I know that multiple particles per entity work, I've used it for SEED).

 

That way you'd had one entity with multiple models.

 

Lights also have "broken" models (I'm not sure if TDM ues this, but in D3 you could shoot lights out) and an "off" model (the candle flame f.i. has a repeating "model" particle with the flame, and a "smoking for a while" run-once particle. (see, the "model" if the flame is a particle actually)

 

If we add the same support for the other model spawnargs, then you could have a ligh with a streetlamp and a haze when on, and streetlamp and no haze when off.

Edited by Tels

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

If we add the same support for the other model spawnargs, then you could have a ligh with a streetlamp and a haze when on, and streetlamp and no haze when off.

I think a mapper could do that anyway by targeting the func emitter from the light. I might be wrong.

 

But sure, supporting multiple "models" for a single entity sounds useful to me, especially as you say for entity lights that are currently made up of model+light+particles. 3+ entities would collapse to 1.

 

For my part, I'm going to take a look at "inline" and try to figure out how it was meant to work and why it's broken. That could be a quick fix for the entity limit, if it turns out that that the visible model simply gets absorbed into the world model and that it doesn't take part in any way in dmapping.

Link to comment
Share on other sites

I think a mapper could do that anyway by targeting the func emitter from the light. I might be wrong.

 

Yes, that would work. But it still uses two entities :)

 

But sure, supporting multiple "models" for a single entity sounds useful to me, especially as you say for entity lights that are currently made up of model+light+particles. 3+ entities would collapse to 1.

 

For my part, I'm going to take a look at "inline" and try to figure out how it was meant to work and why it's broken. That could be a quick fix for the entity limit, if it turns out that that the visible model simply gets absorbed into the world model and that it doesn't take part in any way in dmapping.

 

Yes, that sounds worth investigating. Even if we would only make it work for func_statics with a special spawanrg "inline" "1", it would help huge maps. Does the inlining happen at dmap stage?

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Yes, that sounds worth investigating. Even if we would only make it work for func_statics with a special spawanrg "inline" "1", it would help huge maps. Does the inlining happen at dmap stage?

"inline" "1" is exactly how it's implemented. And yes it happens during dmap but long after the bsp generation so it won't cause problems with splitting up the map. The only problem I see so far just from reading the code is that at spawn time the entity doesn't get removed, it just gets hidden. Perhaps it's working already except for that bit! Time to test.

Link to comment
Share on other sites

"inline" "1" is exactly how it's implemented. And yes it happens during dmap but long after the bsp generation so it won't cause problems with splitting up the map. The only problem I see so far just from reading the code is that at spawn time the entity doesn't get removed, it just gets hidden. Perhaps it's working already except for that bit! Time to test.

 

An easy way would be to inhibit the spawn of the entity when "inline" "1" is set.

 

A good place to add this would be:

 

/*
================
idGameLocal::InhibitEntitySpawn
================
*/
bool idGameLocal::InhibitEntitySpawn( idDict &spawnArgs ) {

   // Consult the difficulty manager, whether this entity should be prevented from being spawned.
   return m_DifficultyManager.InhibitEntitySpawn(spawnArgs);
}

 

or in m_DifficultyManager.InhibitEntitySpawn. This routine already checks for difficulty related spawn and for "random_remove".

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

/*
================
idGameLocal::InhibitEntitySpawn
================
*/
bool idGameLocal::InhibitEntitySpawn( idDict &spawnArgs ) {

// Consult the difficulty manager, whether this entity should be prevented from being spawned.
return m_DifficultyManager.InhibitEntitySpawn(spawnArgs);
}

 

or in m_DifficultyManager.InhibitEntitySpawn. This routine already checks for difficulty related spawn and for "random_remove".

 

Looking again at the code, the second part in DifficultyManager.InhibitEntitySpawn should probaby be moved to idGameLocal::InhibitEntitySpawn - that would be a bitmore logical. No functional change, but keep the code in DifficultyManager only for difficulty, and put the other query (random_remove, inline) in idGameLocal. Looks cleaner to me. (And mea culpa, should have done this the first time round :)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

^^ ok, will do if I end up touching / using it.

 

"inline" works for the visible model, but not the clipmodel. The visible model gets added to the worldspawn model triangle by triangle just the way we'd want it to, but it's ignored during the clipmodel generation. Now taking a look at that...

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recent Status Updates

    • The Black Arrow

      Hope everyone has the blessing of undying motivation for "The Dark Mod 15th Anniversary Contest". Can't wait to see the many magnificent missions you all may have planned. Good luck, with an Ace!
      · 0 replies
    • Ansome

      Finally got my PC back from the shop after my SSD got corrupted a week ago and damaged my motherboard. Scary stuff, but thank goodness it happened right after two months of FM development instead of wiping all my work before I could release it. New SSD, repaired Motherboard and BIOS, and we're ready to start working on my second FM with some added version control in the cloud just to be safe!
      · 2 replies
    • Petike the Taffer  »  DeTeEff

      I've updated the articles for your FMs and your author category at the wiki. Your newer nickname (DeTeEff) now comes first, and the one in parentheses is your older nickname (Fieldmedic). Just to avoid confusing people who played your FMs years ago and remember your older nickname. I've added a wiki article for your latest FM, Who Watches the Watcher?, as part of my current updating efforts. Unless I overlooked something, you have five different FMs so far.
      · 0 replies
    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
×
×
  • Create New...