hide_distance broken again?

[MirceaKitsune]

Okay... what I can provide in terms of a performance comparison for now. This is a small closed area so it may not be the ideal scenario, but distant lights do the same thing and have the same effect just at a different scale.

I have that basement / jail cell where I get a pretty big performance drop due to torches hitting some high-poly models. I took a screenshot of the area in its initial state... I had 6 water arrows placed on the map so I used them to turn off 6 of the torches then took another screenshot. First image shows roughly 800000 tris, the second shows it dropping to nearly 500000 after the torched go out. Performance noticeably improves a little as this happens though not radically much.

[peter_spy]

Ok, so you turned off some overlapping and shadowcasting lights, which obviously got you some performance gains, but drastically changed the look of the environment. So, what's that got to do with an LOD system?

[duzenko]

@MirceaKitsune

There seems to be communication breakdown here and on the bug tracker page

As said above, I'm not planning to spend any time on hiding lights until I'm convinced it's practical

By repro case I mean either an existing mission and a setviewpos for it, or a dedicated test map package file

No repro case - no nothing

Best regards

[peter_spy]

FWIW, you can use the stop time cvar to see how the LOD system works even with lights being rendered.

So you start here and stop time:

As you go further, the number of objects and stats go down:

And you finally end up beyond the hide distance range:

So even if there are lights there, they don't do much to lower the performance. Lights are not expensive until they hit something expensive.

[MirceaKitsune]

It might be best to wait until I have more FM's public, or find one that acts as a proper test case. I understand duzenko's request, from a development standpoint it makes sense to properly understand the need for a feature first. So far the testing I did has been estimative and inexact, even if I know there's a performance improvement in some form. Perhaps I should make a dedicated test map for this? Will think about it and get back to the tracker issue with that if I do.

[MirceaKitsune]

I spent the day creating a miniature city from scratch just to offer the required test case for this, I hope it was worth it and will help in some form. It's probably not perfect, no test case is for something as fiddly as map design is, but I tried getting as close as I could; It's meant to replicate a city area done in my typical style, containing what you can expect to be the top level of detail an author would use (myself included): Not insanely overboard but pretty detailed in what you can see at once.

It differs from Peter's test with the trees in that it uses the default prefabs (some of which already have a hide_distance) containing many smaller models and skins, which should better replicate what you get when you detail a FM in practice. If I was to add an objective this could technically be turned into a super tiny FM you can play... if anyone feels like expanding it you have my permission, I don't plan on it unless it's to improve the testing process. The map is free to do what you want with and its darkradiant file is included, prefabs are grouped so you can easily select and manipulate them.

The player starts at a position from which they see the vast majority of lights and objects. There's a pulpit with two levers in front: The left one disables electrical lights the right one torches: It's the easiest way I could think of to toggle them and see the difference with and without lights. Most of the far away lights should be at a distance of over 1000 to the player, which should replicate what would happen if you'd use a reasonable "hide_distance 1000" on a light.

The numbers with r_showprimitives indicate exactly what I've been seeing in my normal FM: Once you have enough meshes on your map and enough lights around them (even separated enough so they don't overlap much) it gets quite a lot for the engine to manage. The tris with this dedicated test case as you toggle the levers are:

• All lights enabled: 1.300.000
• Only torches or electric electric: 1.000.000
• All lights disabled (except for candles): 670.000

So turning off all important light sources on the map literally halves the total triangle count. Obviously we wouldn't be downright halving it in a practical scenario, but this should show there's a significant difference: If even close to a quarter of the total triangles processed get removed thanks to light hiding, it's worth considering in my book.

disttest.pk4

[MirceaKitsune]

A little something I wanted to add separately from the more technical post above which I wrote for the bug tracker: This map is literally one road with a T dead end at the ending, a design that should be normal for a FM and close to what you'll see from me too. It's not something you can cramp any further and improve with portaling: You will see most of it at once as you stand at the other end of the road and face alongside it, it's unavoidable; All you can do is simply refuse designing any roads longer than some 500 units... which many FM's do and that's okay, but I find it horribly limiting to have to force one's self to towns that are unrealistically claustrophobic for this reason alone.

Just to prove my point I actually added the few logically possible visportals between the buildings in this map, press the n key by default to see them. As one can tell they will not do much, there's not much of a helpful position to put them in.

Another point I heard was that we can use hide_distance on the meshes instead. Technically true, but look at how many objects we have: What mapper is going to ungroup the prefabs then carefully judge based on the scale of each mesh what the distance should be? You obviously won't want to select that entire chunk of entities with different models and sometimes different classes, then blindly set the parameter on them all... remember that using the dist_check_period call can have its own impact too, so it only needs to be set on those models that are both high-poly and used repetitively.

[peter_spy]

Again, you kind of did the same as in the previous example: you have multiple overlapping lights hitting tons of objects, and everything gets better when you turn them off. Perhaps you should give us a hint on what you understand an LOD system is, because I suspect it might vastly differ from what it usually means.

Edited July 28, 2021 by peter_spy

[peter_spy]

...And to give you some perspective on that:

https://developer.valvesoftware.com/wiki/LOD

Quote

Level of Detail (LOD) is the principle of having lower-detail versions of models in memory and switching to them when the object is small/distant. This saves a lot of unneeded processing power at the cost of a not too big memory/disk size increase.

https://docs.unrealengine.com/4.26/en-US/WorkingWithContent/Types/StaticMeshes/HowTo/LODs/

Quote

When a player is up close to a Static Mesh you have placed in your level, you want the mesh to look very detailed. However, you do not need that mesh to be very detailed and complex once the player moves far away from the mesh. There is no point in having a very complex and detailed mesh if it is only taking up a few pixels on the screen and the player can barely see it. However, the mesh needs to be detailed when the player is up close to the mesh and can see it clearly. In UE4, you can place a mesh in your level that switches to a less complex mesh as the player moves away from it in order to make your level perform better. You can do this through the use of Level of Details, or LODs. This How-to will show you how you can import a less detailed version of a mesh into UE4 and have your mesh seamlessly swap from one mesh to the other as the player moves closer or farther away.

http://wiki.polycount.com/wiki/LOD

Quote

A Level of Detail model (LOD) is a lower-resolution version of a game model. The lower-res version is used to improve performance by being swapped in whenever the higher-resolution of the game model is not apparent anymore, for example when it is a certain distance from the player. Usually LODs have a lower vertex count (50% or so) and simpler shader (diffuse only, for example).

https://docs.unity3d.com/Manual/LevelOfDetail.html

Quote

Level of detail (LOD) is a technique reduces the number of GPU operations that Unity requires to render distant meshes.

When a GameObject in the Scene is far away from the Camera, you see less detail compared to when the GameObject is close to the Camera. However, by default, Unity uses the same number of triangles to render it at both distances. This can result in wasted GPU operations, which can impact performance in your Scene.

The LOD technique allows Unity to reduce the number of triangles it renders for a GameObject based on its distance from the Camera. To use it, a GameObject must have a number of meshes with decreasing levels of detail in its geometry. These meshes are called LOD levels. The farther a GameObject is from the Camera, the lower-detail LOD level Unity renders. This technique reduces the load on the hardware for these distant GameObjects, and can therefore improve rendering performance.

 

And in general:

https://docs.unity3d.com/Manual/OptimizingGraphicsPerformance.html

https://docs.unrealengine.com/4.26/en-US/TestingAndOptimization/PerformanceAndProfiling/Guidelines/

These guidelines^ can be applied to idtech4 as well. As you see, there are performance optimization tricks for lighting, but there's literally nothing there on using LOD for lights.

 

By the way:

12 hours ago, MirceaKitsune said:

remember that using the dist_check_period call can have its own impact too

Perhaps in the old days, but having a few thousand of LOD entities in a scene costs CPU next to nothing, while providing substantial savings for the GPU. It's more likely that you'll run into any other bottleneck much sooner.

[MirceaKitsune]

I think we're talking about two different kinds of LOD overall: One is using a lower polygon version of a model after a certain distance, which obviously applies only to meshes... the other is hiding and derendering an item after a certain distance, which can be applied to anything and I'm not sure to what extent LOD is even the best term to describe it. From a technical standpoint I can see why it's included in the same category, theoretically though it's hiding something entirely not changing the detail level.

A bit of overlap will exist on any map: You'll have little stuff like a candle being located near an electric light here and there. Further more you have cases where for example, you'll want two torches on opposite sides of a wall or around a corner, as if you put one only on one side the other will remain dark due to shadows... technically they overlap but it may be necessary.

In this case torches are alternated with electrical lights: If you turn off just one or the two from the levers, you'll get rid even of that bit of overlap. Yet that's still the difference from 1.300.000 to 1.000.000, turning off the other group gets you under 700.000.

[MirceaKitsune]

Just updated my test case to ensure it's as representative as possible to what we're looking at. I namely removed all lights near the starting position of the player, leaving only those that are at least 512 units away which would be the minimal sane distance to start hiding some lights like torches and candles. I also repositioned them to avoid even that bit of overlap between electrical lights where the edges were touching slightly. Overall I left in only 3 street lamps and 2 torches in between them, disabling even the prefab candles for consistency: This should be as accurate as it gets to a realistic scenario.

How the numbers changed: With all 5 lights enabled we have 810.000 tris, with them turned off it's 480.000. Once more it's almost halved, more than a third of the tris calculated are gone. And with this updated version it's representative of the exact effect the player would get with such distance hiding, nothing generic or estimative: All lights are +512 units away, with "hide_distance 500" it would be the exact same thing as toggling the lever from behind the pulpit.

Extra info: I'm on the latest development version available in the installer (9507). I'm using shadow maps not stencil shadows, they're the fastest option as of the newest dev snapshots. OS is Linux Manjaro on KDE, amdgpu module.

disttest_onlydistant.pk4

[cabalistic]

But these screenshots look nothing alike. Surely that's not what a player would want to happen just by moving a certain amount beyond some arbitrary distance?!

Any LOD effects are usually meant to be deployed where a reduction in graphical detail gives a performance boost without being too noticeable to the player. Those lights being turned off is as far from not being noticeable as you can get.

[peter_spy]

41 minutes ago, MirceaKitsune said:

I think we're talking about two different kinds of LOD overall: One is using a lower polygon version of a model after a certain distance, which obviously applies only to meshes... the other is hiding and derendering an item after a certain distance, which can be applied to anything and I'm not sure to what extent LOD is even the best term to describe it. From a technical standpoint I can see why it's included in the same category, theoretically though it's hiding something entirely not changing the detail level.

I think it's ok to end this part of discussion, as it becomes noise. You're just making stuff up now; none of this has any confirmation in any systems used in game engines, past or present.

As for overlapping shadowcasting lights, this is what causes performance issues, in any game engine, and it's on you to optimize the lights.

And you still don't understand that your map demonstrates nothing.

So, to reiterate: LOD system changes model geometry when it's away, in order to save performance, but while maintaining the visuals relatively unchanged.

In order to demonstrate that lights with LOD system make sense, you'd have to make a map where they're toggled on and off, perhaps with scripts and triggers placed at certain distances, and in such an environment, e.g. with certain fog settings, where this would also maintain the visuals relatively unchanged.

So far you have done none of that.

[MirceaKitsune]

A little extra test, since I know what some are going to say next and maybe rightfully so: "But Mircea, you still have those two torches overlapping the electrical lights, many mappers won't do that". Also the street lamps are long range and we wouldn't be crazy to turn those off at a distance of 512 (I'd probably have them at 1024 - 2048), only torches would be disabled at 512.

So let's try the updated map without the torches. Not gonna post another version just for that, simply delete the two torches and load the map. This way we have only 3 electrical lights that don't touch.

For the first debate: Disabling the electric lights alone and having no more lights (except ambient and fog) still takes you from 639655 tris and 3552 draws to 476101 tris and 2474 draws, removing roughly a third for both.

On the second issue: Between street lamps + torches and only street lamps, we seem to get 811.636 tris and 4525 draws versus 639.655 tris and 3552 draws. So even if we were to hide just the torches at +512, we get about a quarter of the tris and draws trimmed down. I definitely think this says something honestly.

[MirceaKitsune]

See my last reply, was just typing it in the meantime. It shows how even removing only the torches and leaving the electric lights on has a notable improvement. Which would count as leaving the visuals relatively unchanged, especially if we had fading as well. Torches are lower range and acceptable... we'd also have even smaller lights like candles.

The lever example is the best I can use right now: I can't test a feature that isn't yet implemented. But it's close to simulating what would happen. The map is easy to edit, you can make it toggle just the torches or something and see to what extent you feel that makes a visual difference. I did my best to provide a useful test case as much as I saw possible.

Edited July 28, 2021 by MirceaKitsune

[peter_spy]

Nope, that's still making stuff up. Show us a map with lights that can act as standard LOD system, as described above (without impacting visuals), and present us the hard data on performance improvements. Otherwise, you're derailing the conversation.

[MirceaKitsune]

11 minutes ago, peter_spy said:

Nope, that's still making stuff up. Show us a map with lights that can act as standard LOD system, as described above (without impacting visuals), and present us the hard data on performance improvements. Otherwise, you're derailing the conversation.

I'm doing exactly that: Showing you a map designed specifically for understanding this improvement, with precise data on how it's influencing the amount of triangles being rendered. You don't want to see that it's there, but rather wish to ask for impossible things. There are two logical issues with your stance which I simply can't satisfy.

One is that you're asking me to show an exact test case of something that isn't implemented yet, or to implement it myself in order to show that it's worthy of being implemented. That's a bit of a paradox. I got as close as possible with current features.

The second is arguing what one may or may not find visually disruptive, which is highly subjective. I for one think two groups should decide on that: The mapper who sets those spawnargs for the lights on their map, which they should do based on what they and the players feel looks alright on that particular map. Also the player theirself via the LOD setting in the menu, which can ideally affect the hide distance so they can set it higher or lower based on their taste for performance versus popping... this is a discussion of its own and we already have enough going on here.

Edited July 28, 2021 by MirceaKitsune

[peter_spy]

You don't see the difference between making assumptions and extrapolations in your head and doing actual experiments because you're not doing anything with these maps. Read the above posts until you get them, and make a map with actual lights toggled with distance, not with a switch and everything going dark, because this way you prove absolutely nothing.

Also, to make it even more literal for you, why do you think noone in gamedev spent time on including lights in the LOD systems? Maybe they experimented with that, and found out there is no way to use it without it looking awful? And/or the performance gains were minimal? Do you think you're better or more experienced than people making Unreal, Unity etc.? Then feel free and supply us an actual proof, not a map that is 50% there, and 50% your feelings, preconceptions and extrapolations.

Edited July 28, 2021 by peter_spy

[MirceaKitsune]

1 minute ago, peter_spy said:

You don't see the difference between making assumptions and extrapolations in your head and doing actual experiments because you're not doing anything. Read the above posts until you get them, and make a map with actual lights toggled with distance, not with a switch and everything going dark, because this way you prove absolutely nothing.

Also, to make it even more literal for you, why do you think noone in gamedev spent time on including lights in the LOD systems? Maybe they experimented with that, and found out there is no way to use it without it looking awful? And/or the performance gains were minimal? Do you think you're better or more experienced than people making Unreal, Unity etc.? Then feel free and supply us an actual proof, not a map that is 50% there, and 50% your feelings, preconceptions and extrapolations.

Again: You're asking me to implement this feature myself in order to show that it should be implemented. That's like saying "build a car to prove that someone should go through the effort of building a car". Really?

There are no assumptions and extrapolations: I shared screenshots with exact numbers and data. By now I suspect that no matter what data I'd post here you still wouldn't accept it for some reason.

This argument isn't getting us anywhere. At this point I'd like to stop filling the thread with it and wait for other developers to look at this and try the map I shared and decide for themselves. duzenko asked for a test case and I partly made this in an attempt to offer what I could in that regard.

[cabalistic]

10 minutes ago, MirceaKitsune said:

I'm doing exactly that: Showing you a map designed specifically for understanding this improvement, with precise data on how it's influencing the amount of triangles being rendered. You don't want to see that it's there, but rather wish to ask for impossible things. There are two logical issues with your stance which I simply can't satisfy.

I'm afraid you're not. You are showing a standard scene and just hide a lot of its lights. It's very clear that this gives a performance boost, nobody doubts that. It's just also completely meaningless, because doing so alters the entire scene. And so it becomes a completely different scene and not merely a "performance optimized" scene. As such, it is *not* suitable to determine whether culling lights by distance is an actual usable performance optimization.

As I already explained on the Discord, the influence of lights on performance correlates with the amount of pixels it touches, because the most expensive parts of lights are the calculations done in the pixel shader. As such, the amount of tris is not as important, so you are looking at the wrong numbers. However, this is also the issue: when a light is sufficiently negligible to a scene that culling it could actually be an acceptable performance optimization, it will invariably also be touching a lot fewer pixels to begin with. Certainly a lot fewer than in your test scene. And that changes the whole equation, because if it's touching fewer pixels to begin with, it will have less benefit to not render the light.

All we are asking here is a demonstration that in a situation where a light is barely noticable (visually), it is still sufficiently impacting performance that implementing a distance-based culling system for lights would be worth the (potentially considerable) development effort.

[peter_spy]

5 minutes ago, MirceaKitsune said:

Again: You're asking me to implement this feature myself in order to show that it should be implemented. That's like saying "build a car to prove that someone should go through the effort of building a car". Really?

No, noone cares about you solutioning developers on how a feature should be implemented, you giant confused dummy Since you're so good at extrapolation, how come it didn't occur to you that you can simply emulate the LOD system for lights with trigger brushes placed at certain distances, to toggle lights on and off?

Edited July 28, 2021 by peter_spy

[MirceaKitsune]

16 minutes ago, cabalistic said:

I'm afraid you're not. You are showing a standard scene and just hide a lot of its lights. It's very clear that this gives a performance boost, nobody doubts that. It's just also completely meaningless, because doing so alters the entire scene. And so it becomes a completely different scene and not merely a "performance optimized" scene. As such, it is *not* suitable to determine whether culling lights by distance is an actual usable performance optimization.

As I already explained on the Discord, the influence of lights on performance correlates with the amount of pixels it touches, because the most expensive parts of lights are the calculations done in the pixel shader. As such, the amount of tris is not as important, so you are looking at the wrong numbers. However, this is also the issue: when a light is sufficiently negligible to a scene that culling it could actually be an acceptable performance optimization, it will invariably also be touching a lot fewer pixels to begin with. Certainly a lot fewer than in your test scene. And that changes the whole equation, because if it's touching fewer pixels to begin with, it will have less benefit to not render the light.

All we are asking here is a demonstration that in a situation where a light is barely noticable (visually), it is still sufficiently impacting performance that implementing a distance-based culling system for lights would be worth the (potentially considerable) development effort.

This explains it better. Yes, there's pixel calculations and shadow calculations, was still confused on the impact between the two but aware they each have their own impact.

Which numbers do you suggest I test, if there's a cvar to print other useful data? One of the issues here is that r_showprimitives is the only one I'm aware of and am used to referring myself to. I can noticeably feel the performance difference, and it seems to correlate with what showprimitives prints out (noticeable drops occur once it prints past one million triangles), thus I got used to resorting to it.

14 minutes ago, peter_spy said:

No, noone cares about you solutioning developers on how a feature should be implemented, you giant confused dummy Since you're so good at extrapolation, how come it didn't occur to you that you can simply emulate the LOD system for lights with trigger brushes placed at certain distances, to toggle lights on and off?

I could do that. If anyone else asks me to I'll update my test case again and use trigger brushes. Won't do it just for you because you've been a meanie Ok not really, but I feel that if I did I'd be told once more it's not representative either; Your suggestion is good actually, I'll see if I will... just don't want to waste another 30 minutes / hour only to be told "no".

And again: If anyone thinks the r_showprimitives data is meaningless, please let me know what other performance statistics I can use instead. I have this test case open and handy to edit if there's something else to try. I want to help in determining if this is worth it or not, but need to understand clearly what I must do.

Edited July 28, 2021 by MirceaKitsune

[peter_spy]

I'm only mean because my god you're annoying  

To be fair, I much rather see such proof of concept from @kingsal who also showed interest in this feature; he's in touch with modern gamedev, and he knows the proper level design tips and tricks. Plus he already has experience with making such lights in his maps, so this way we'd be sure we're on the same page and his example is relevant.

[duzenko]

25 minutes ago, peter_spy said:

I'm only mean because my god you're annoying  

@peter_spySorry for interference, but I think that's the kind of language that can get you in trouble in this forums

The silent committee is watching you

I personally don't mind a little internet drama, but better me try to warn you than one day you just wake up and find out you're banned. Not that it would be end of the world, but still.

Let's be cool.

[MirceaKitsune]

I wouldn't mind seeing Kingsal or another mapper make a test case too. Mine isn't necessarily perfect, it's just the one I could come up with in the meantime.

Just tried simulating distance with triggers but it doesn't want to work unfortunately. trigger_multiple acts weird when you make large brushes and target lights with them... plus it toggles them every 0.5 seconds as you touch the brush so they just blink, I need a proper relay and trigger_inactivity fails so far. trigger_once wouldn't work either as it runs once.

If that's alright I'm leaving it with the lever. @duzenko I hope that's good enough. If not I don't mind waiting on someone else to design a better test or bring mine up to standards; This feature isn't a top emergency even for me if it's so much trouble... I want it as I feel it improve performances after everything I tried, but even my FM's will survive without it.

Edited July 28, 2021 by MirceaKitsune