Jump to content
The Dark Mod Forums

hide_distance broken again?


grayman

Recommended Posts

6 hours ago, cabalistic said:

I'm afraid you're not. You are showing a standard scene and just hide a lot of its lights. It's very clear that this gives a performance boost, nobody doubts that. It's just also completely meaningless, because doing so alters the entire scene. And so it becomes a completely different scene and not merely a "performance optimized" scene. As such, it is *not* suitable to determine whether culling lights by distance is an actual usable performance optimization.

As I already explained on the Discord, the influence of lights on performance correlates with the amount of pixels it touches, because the most expensive parts of lights are the calculations done in the pixel shader. As such, the amount of tris is not as important, so you are looking at the wrong numbers. However, this is also the issue: when a light is sufficiently negligible to a scene that culling it could actually be an acceptable performance optimization, it will invariably also be touching a lot fewer pixels to begin with. Certainly a lot fewer than in your test scene. And that changes the whole equation, because if it's touching fewer pixels to begin with, it will have less benefit to not render the light.

All we are asking here is a demonstration that in a situation where a light is barely noticable (visually), it is still sufficiently impacting performance that implementing a distance-based culling system for lights would be worth the (potentially considerable) development effort.

To be clear, it is my understanding that lights still perform the following actions:

1. If a light is visible in the view frustum, visleaf, portal view, and scissor calculations

2. Gather all vertex data within the bounding box of the light

3. Skin all vertices gathered in step 2 ( expensive CPU wise )

4. Upload all skinned triangles to the GPU ( expensive regarding bus data )

5. Fill pixels for all triangles uploaded in step 4 via a depth pass

6. Calculate object silhouettes on the CPU

7. Extrude the shadow vertices on the GPU

8. Fill the stencil buffer

9. Fill the pixels that are now visible after steps 1 through 8 via the light shader

 

If so, it is your assertion that step 9 is more expensive that steps 3, 4, 5, 6, 7, 8 ?

Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Link to comment
Share on other sites

I think I found a way to settle this. Managed to put together a script / addon which emulates this functionality. The advantage is you can use it in any FM: Drop the pk4 directly inside your TDM folder and it will automatically work. Two versions are provided: One turns the light off entirely, the other only disables its shadows... don't use both at once but pick one.

Just remember this is a testing script and will break your run! For one it doesn't know when a light is meant to be turned off, thus will enable even lights that started or were toggled off once they're in range... meanwhile the nowshadows version enables self lighting on torch models which looks wrong. They also seem to cause infinite loop crashes on occasion, though it should take long enough for them to occur in order to obtain a result.

light_dist.pk4

light_dist_noshadows.pk4

// Test script for hiding lights with distance
#define DIST_TORCH 1024
#define DIST_GAS 1536
#define DIST_ELECTRIC 2048

void light_dist()
{
	while(1)
	{
		// sys.waitFrame();
		sys.wait(0.5);

		player ent = $player1;
		entity ent_find;

		// Do torch lights
		do
		{
			ent_find = sys.getNextEntity("lightType", "AIUSE_LIGHTTYPE_TORCH", ent_find);
			if(ent.distanceTo(ent_find) > DIST_TORCH)
				ent_find.Off();
			else
				ent_find.On();
		}
		while(ent_find != $null_entity);

		// Do gas lights
		do
		{
			ent_find = sys.getNextEntity("lightType", "AIUSE_LIGHTTYPE_GASLAMP", ent_find);
			if(ent.distanceTo(ent_find) > DIST_GAS)
				ent_find.Off();
			else
				ent_find.On();
		}
		while(ent_find != $null_entity);

		// Do electric lights
		do
		{
			ent_find = sys.getNextEntity("lightType", "AIUSE_LIGHTTYPE_ELECTRIC", ent_find);
			if(ent.distanceTo(ent_find) > DIST_ELECTRIC)
				ent_find.Off();
			else
				ent_find.On();
		}
		while(ent_find != $null_entity);
	}
}
// Test script for hiding lights with distance
#define DIST_TORCH 1024
#define DIST_GAS 1536
#define DIST_ELECTRIC 2048

void light_dist()
{
	while(1)
	{
		// sys.waitFrame();
		sys.wait(0.5);

		player ent = $player1;
		entity ent_find;

		// Do torch lights
		do
		{
			ent_find = sys.getNextEntity("lightType", "AIUSE_LIGHTTYPE_TORCH", ent_find);
			ent_find.noShadows(ent.distanceTo(ent_find) > DIST_TORCH);
		}
		while(ent_find != $null_entity);

		// Do gas lights
		do
		{
			ent_find = sys.getNextEntity("lightType", "AIUSE_LIGHTTYPE_GASLAMP", ent_find);
			ent_find.noShadows(ent.distanceTo(ent_find) > DIST_GAS);
		}
		while(ent_find != $null_entity);

		// Do electric lights
		do
		{
			ent_find = sys.getNextEntity("lightType", "AIUSE_LIGHTTYPE_ELECTRIC", ent_find);
			ent_find.noShadows(ent.distanceTo(ent_find) > DIST_ELECTRIC);
		}
		while(ent_find != $null_entity);
	}
}

With the defaults I balanced it at, I admit I'm not all impressed with the result on my FM: The popping is noticeable and the performance improvement doesn't feel as radical as I would had hoped even if it's there. With r_showprimitives it does seem to kick about 300.000 triangles off when fully disabling the light which doesn't seem too bad... not giving any more verdicts this way though, I'll let the devs play with the script and decide what to make of it.

Link to comment
Share on other sites

2 hours ago, cabalistic said:

All we are asking here is a demonstration that in a situation where a light is barely noticable (visually), it is still sufficiently impacting performance that implementing a distance-based culling system for lights would be worth the (potentially considerable) development effort.

Yeah this is tricky, frivolous shadow rays *used* to be a big issue, but with all the renderer and GPU updates maybe not?  I wonder if  transparents, alpha tests, and FXs are bigger culprits now. Having a light touching a glass window with fog and trees behind it might be a worse case scenario. 

Would switching to a "less expensive" lighting model at a distance  be beneficial or even feasible? Are there less accurate, but cheaper lighting methods outside of just  shadow/ no shadow? What about texture resolutions? Would rendering less pixels at distance do anything for us? Sorry just kind of troubleshooting here.

@MirceaKitsune @duzenko I think the proper approach here would be to open up a heavy map like TPW or the opening shot in your FM Mircea, and look into the impact  lights, emitters, transparents, ect are having on it. I'm sure turning off lights will have an impact, but doing it in an elegant manner might not feasible. 

I'll look into this a bit more this weekend and see if I can really prove that dropping lights out can A: give back enough performance in enough situations to warrant the change and B: actually be done in such a way that isn't glaring to the player. 

Link to comment
Share on other sites

11 minutes ago, kingsal said:

What about texture resolutions? Would rendering less pixels at distance do anything for us?

Note that mipmapping still applies here, although that still means keeping the base (highest resolution) texture in memory. Otherwise you can set different materials for different LOD levels, so they could use lower resolution textures, but IIUC, TDM would need to have texture streaming to use that effectively.

Edited by peter_spy
Link to comment
Share on other sites

19 minutes ago, peter_spy said:

Note that mipmapping still applies here, although that still means keeping the base (highest resolution) texture in memory. Otherwise you can set different materials for different LOD levels, so they could use lower resolution textures, but IIUC, TDM would need to have texture streaming to use that effectively.

Does mipmapping apply to light calculations as well, including the normal map pixels? @cabalistic should know best about this one, it's one of the questions I didn't get to ask last time.

There was also another one, may as well drop it here: If you have a large func_static with a highpoly mesh, and a light only touches part of it, does that light calculate each of the faces of that func_static model and waste resources on it all? Or are single models pruned to have triangles not used by that light / shadow removed from its calculation?

Asking this as a mapper too: If every model touched by a light will have all its vertices calculated for that light, it then becomes important to move meshes which are only partly touched outside the area of that light.

Link to comment
Share on other sites

20 hours ago, nbohr1more said:

To be clear, it is my understanding that lights still perform the following actions:

1. If a light is visible in the view frustum, visleaf, portal view, and scissor calculations

2. Gather all vertex data within the bounding box of the light

3. Skin all vertices gathered in step 2 ( expensive CPU wise )

4. Upload all skinned triangles to the GPU ( expensive regarding bus data )

5. Fill pixels for all triangles uploaded in step 4 via a depth pass

6. Calculate object silhouettes on the CPU

7. Extrude the shadow vertices on the GPU

8. Fill the stencil buffer

9. Fill the pixels that are now visible after steps 1 through 8 via the light shader

 

If so, it is your assertion that step 7 is more expensive that steps 3, 4, 5, 6, 7, 8 ?

I'll answer this if you don't mind

1, 2, 3, 6 happen on CPU and 5, 8, 9 on GPU, so they don't play the zero sum game together

But see my post a few pages back - it's cheap enough to draw a distant light when it just touches a couple of triangles of wall brush - agree?

It's been reiterated many times now in this thread that a light is only as expensive as the entity models it's touching

You should worry about entities first.

18 hours ago, kingsal said:


@MirceaKitsune @duzenko I think the proper approach here would be to open up a heavy map like TPW

A very good point

The only heavy street scene I found in my quick fly over in TPW was inside tree gardens, caused by many transparent tree leave surfaces overdrawing up close.

Link to comment
Share on other sites

41 minutes ago, duzenko said:

I'll answer this if you don't mind

1, 2, 3, 6 happen on CPU and 5, 8, 9 on GPU, so they don't play the zero sum game together

But see my post a few pages back - it's cheap enough to draw a distant light when it just touches a couple of triangles of wall brush - agree?

It's been reiterated many times now in this thread that a light is only as expensive as the entity models it's touching

You should worry about entities first.

A very good point

The only heavy street scene I found in my quick fly over in TPW was inside tree gardens, caused my many transparent tree leave surfaces overdrawing up close.

In all previous Doom 3 discussions, one of the main areas of concern was that this renderer is "not deferred" meaning that regardless of whether the entities are visible the engine will attempt to render them. This means that if a light touches entities that are obstructed by geometry but are not culled via visportal then the engine will perform all the setup of all that unseen geometry. This is generally offered as one explanation for why a scene with good visportal design performs dramatically better than one with no visportals that is visually identical. So? Of course we would wish that all maps have perfect visportal design but even if this comes true what about the case where shadow casters exist behind obstructive geometry? As I recall, vanilla Doom 3 treats lights as having possibly infinite shadow casting range ( I believe Doom 3 BFG constrains the potential shadowed area to the light volume cube ). This means that many shadows that are invisible but extend away from the player behind an occluder can consume resources. It has been said that there is no "solution" to shadows behind occluders since there is the potential for incorrect shading on the unobstructed sides behind the occluder but I would offer that mappers might want to try that tradeoff? Especially if the difference is a negligible sliver of light or shadow being removed. ( See a previous discussion about some mappers wanting "antiportals" \ "func_occluder" entities. )

Does this make sense?

Of course, Shadow Maps also play a part in removing some of these factors but since shadow map mode is still hybrid the stencil shadow concerns still arise.

Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Link to comment
Share on other sites

5 hours ago, nbohr1more said:

If so, it is your assertion that step 7 is more expensive that steps 3, 4, 5, 6, 7, 8 ?

No, I would argue that in most instances, steps 8 and 9 (or the alternative GPU operations for shadow maps) are more expensive than the rest, unless perhaps you have a very strong GPU or use a very low rendering resolution. Yes, you might potentially be able to create scenes which are CPU-limited rather than GPU-limited due to poor visportaling or some very particular circumstances, but in the majority of cases that I have seen, GPU time at least matches and usually overtakes CPU time at the moment. Therefore, if we want to find ways to reduce overall frametime that is as generically applicable as possible, then right now we need to find ways that reduce GPU time at least as much as CPU time.

Link to comment
Share on other sites

15 minutes ago, cabalistic said:

No, I would argue that in most instances, steps 8 and 9 (or the alternative GPU operations for shadow maps) are more expensive than the rest, unless perhaps you have a very strong GPU or use a very low rendering resolution. Yes, you might potentially be able to create scenes which are CPU-limited rather than GPU-limited due to poor visportaling or some very particular circumstances, but in the majority of cases that I have seen, GPU time at least matches and usually overtakes CPU time at the moment. Therefore, if we want to find ways to reduce overall frametime that is as generically applicable as possible, then right now we need to find ways that reduce GPU time at least as much as CPU time.

Sorry, forgot one critical edit. Still this answer makes sense. Doom 3 is a fillrate monster and continues to be one in the TDM engine era.

Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Link to comment
Share on other sites

3 hours ago, kingsal said:

Would switching to a "less expensive" lighting model at a distance  be beneficial or even feasible? Are there less accurate, but cheaper lighting methods outside of just  shadow/ no shadow? What about texture resolutions? Would rendering less pixels at distance do anything for us? Sorry just kind of troubleshooting here.

Less expensive, possibly, but again, it would depend how many pixels actually benefit from the less expensive model and what kind of effort you'd have to take to make the blend between the two not pop out too much. And exactly how much the difference between the lesser and the full model really is.

As for rendering fewer pixels in the distance, that's not really possible, because GPUs traditionally don't really work that way. Or rather, it would require a very new feature called variable rate shading that is only supported by RTX and RDNA 2 cards. And only RTX cards support it for OpenGL. Don't think it's worth to implement for those cards alone :D

Link to comment
Share on other sites

16 minutes ago, cabalistic said:

Less expensive, possibly, but again, it would depend how many pixels actually benefit from the less expensive model and what kind of effort you'd have to take to make the blend between the two not pop out too much. And exactly how much the difference between the lesser and the full model really is.

As for rendering fewer pixels in the distance, that's not really possible, because GPUs traditionally don't really work that way. Or rather, it would require a very new feature called variable rate shading that is only supported by RTX and RDNA 2 cards. And only RTX cards support it for OpenGL. Don't think it's worth to implement for those cards alone :D

Right on, thanks for the clarity. Yeah it looks like just reducing object count /light interactions the good ole fashion way will have the most impact. Ill see if I can put something together.

Link to comment
Share on other sites

3 hours ago, kingsal said:

Right on, thanks for the clarity. Yeah it looks like just reducing object count /light interactions the good ole fashion way will have the most impact. Ill see if I can put something together.

What I'm starting to learn from this is that, perhaps, DarkRadiant and / or default prefabs should make better use of hide_distance for meshes. As well as making it easier to configure LOD and better encouraging mappers to do so.

I actually made a thread suggesting a LOD tab in the entity inspector, similarly to how there's an AI tab. Figured it might make it easier to configure this system and gain the best from it. Feel free to share your thoughts there!

 

Link to comment
Share on other sites

By the way, I guess you'll see that in the code, but the way the lod_fadeout_range works right now makes it unusable with anything but LOD 0 models. Maybe it's because it uses hide_distance as a starting point instead of fading out to hide_distance?

In any case, you can have a model with e.g. LOD 0 that is 3k tris and cast shadows, then LOD 1 that is 1k tris and noshadows, and when you reach the hide_distance, the model switches to LOD 0 again.

Not only can this look bad, with shadows suddenly popping in before the model fades out, but I guess it can also generate unnecessary performance spikes, going from properly optimized models to full quality in the last stage.

Edited by peter_spy
Link to comment
Share on other sites

The LOD and LOD Bias wiki pages fail to explain a little detail I'm interested in; When you set only the hide_distance or lod_#_* spawnargs on an entity, do these follow the menu setting for LOD automatically?

So let's say I set a func_static to have "hide_distance 1000": Does this mean it will be 1000 for players with the normal setting, but 500 for players with the low setting or 2000 for players with the high setting? This is the functionality I'd prefer BTW! If not I'd like to know if I need to set other spawnargs manually, so that the base range can be set by the mapper but made lower or higher from the menu based on each player's hardware and preference.

Link to comment
Share on other sites

I took a quick look at @MirceaKitsune's test map

While I generally retain my entity vs light LOD opinion, at the same time I can see some merit in per-light LODing as well

This particular test map seems to include many small entities (details) which can be tedious to manually mark one-by-one

I.e. 7 lights vs 469 entities

I don't think I'll be actually hiding lights but maybe skipping some interactions and/or bulk-hide entities grouped by light

Still, @MirceaKitsune how did you imagine this? You didn't think scenes like this will actually turn off big nearby lights?

 

Link to comment
Share on other sites

OK, sorry to be a pain but I did ponder one other sort of ugly use-case.

The mission "Sir Talbot's Collatoral" uses many small and subtle lights referred to as "bounce lights" to improve the look of ambient lit areas. If some author went crazy with this lighting method, performance could be very bad in a large open area such as a warehouse, courtyard or foyer. Turning off these bounce lights at a distance would be far less noticeable since they only add subtle effect anyway.

Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Link to comment
Share on other sites

6 minutes ago, nbohr1more said:

OK, sorry to be a pain but I did ponder one other sort of ugly use-case.

The mission "Sir Talbot's Collatoral" uses many small and subtle lights referred to as "bounce lights" to improve the look of ambient lit areas. If some author went crazy with this lighting method, performance could be very bad in a large open area such as a warehouse, courtyard or foyer. Turning off these bounce lights at a distance would be far less noticeable since they only add subtle effect anyway.

I use a ton of bounced lights as well, even in large spaces. Its true they can really add up if you arent careful, but at least in my case turning them to noshadow is what keeps the frame rate from tanking :) Also from my understanding its moreso about the amount of surfaces those lights touch. So if you have 2 direct lights and 2 bounced lights on a pile of objects, there will be a noticeable drop in perf. 

I can't say  that turning those bounce lights off will be less noticeable, in fact it might be the opposite in my case. I guess it all depends on how the author uses them. 

Link to comment
Share on other sites

That's what I've been wondering recently as well

Why not merge nearby ambient lights into one mega-light? We already use textures for projection falloff. It makes perfect sense to slap a few lights into a single texture and voila.

Link to comment
Share on other sites

1 hour ago, duzenko said:

That's what I've been wondering recently as well

Why not merge nearby ambient lights into one mega-light? We already use textures for projection falloff. It makes perfect sense to slap a few lights into a single texture and voila.

Isn't that what r_shadowMapSinglePass 2 currently does for normal lights? Maybe create a cvar that does the same but only for ambient lights?

Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Link to comment
Share on other sites

2 hours ago, nbohr1more said:

Isn't that what r_shadowMapSinglePass 2 currently does for normal lights? Maybe create a cvar that does the same but only for ambient lights?

Sadly it's broken, for me "r_shadowMapSinglePass 2" turns everything into white textureless geometry. Also @cabalistic confirmed it's going to be removed 😭 Can only hope the improvement being lost isn't going to be significant.

At least "r_shadowMapSinglePass 1" works very well though. Hope that's going to be defaulted on for 2.10.

Link to comment
Share on other sites

10 hours ago, nbohr1more said:

Isn't that what r_shadowMapSinglePass 2 currently does for normal lights? Maybe create a cvar that does the same but only for ambient lights?

I did not follow that change but from the name of it unlikely

I meant merging lights on the DR level

Link to comment
Share on other sites

Just found something really funny: hide_distance already works for torch lights in the latest dev snapshot, including turning off the light source as desired! It only doesn't work on electric and standalone light entities. So we actually have this half implemented now, if anything it's an inconsistency... it just needs to be fixed for the remaining light types and that's it :D

Link to comment
Share on other sites

7 hours ago, duzenko said:

Cool, how can I test this?

I'm seeing it with atdm:torch_gothic02_wall: Give it "dist_check_period 0.5" with "hide_distance 1000" and it will pop out together with its light source. Electric lights though, like atdm:streetlamp_round_lit, will not... it likely has to do with how their light source is defined or attached by comparison.

Link to comment
Share on other sites

I've been optimizing my map to boost fps from in a lot of cases 25-30 to 45-65, doing this systematically.  I have a newbie perspective, this being my first FM - so   ....  be kind.

First off, when building the .map I began by dropping in shadow/no_shadow lights with zero care - just looking to light the areas being worked on.  Same for entities, dropping them in with little care except for aesthetics and creating shadows for the player to hide in. That procedure made for a very unoptimized map. 

So, what's been the optimizing plan?  First, clean up the geometry.  Second, go thru' the map section by section, setting shadow/no_shadow on entities.   There's a tradeoff:  complexity of geometry multiplied by shadows gets out of hand, so a lot of care needed here.  With a first pass on that done, redo the lighting.  I eliminated all the small ambients I'd put in, whether shadowcasting or not, leaving just those lights attached to fixtures of some kind, which I wanted to be in general shadowcasting.  I reset these shadowcasters to not overlap so far as viable.  I was amazed to find that this made for a much much better looking map coupled with much more acceptable frame rates.  

I use hide_distance on entities sparingly as I don't like the pop-outs.  Not at all!   My map has a lot of transparent windows/doors.  I use func_portals with hide_distance set on every one of these.  Controlling pop-outs caused by func_portals is fairly simple.  The func_portal pop-outs tend to be marked by abrupt lighting changes.  These are very ugly and getting rid of them or making them unobtrusive is a priority.  

I cant see using any kind of light pop-out/pop-in spawnarg.  Rather, I could see using a shadow/no_shadow hide_distance setting, which could be used more universally and quite unobtrusively.  If that's possible now, please tell me how!  Also, I could see using a fade-to-black hide_distance on lights, so there'd be no lights a popping.

 

  • Like 1
Link to comment
Share on other sites

Guest
This topic is now closed to further replies.

  • Recent Status Updates

    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
    • OrbWeaver

      I like the new frob highlight but it would nice if it was less "flickery" while moving over objects (especially barred metal doors).
      · 4 replies
    • nbohr1more

      Please vote in the 15th Anniversary Contest Theme Poll
       
      · 0 replies
×
×
  • Create New...