My idea for a reflection / global illumination system

[MirceaKitsune]

Last night we chatted on Discord about Vulkan support and PBR, bringing up a system for adding proper reflections once more. I suggested screenspace reflections but it was argued reflection probes would still be better than SSR in our case, a ticket for that is already open but I'm uncertain whether it's the best way: A manual approach would need new entities to be placed by the mapper... this requires extra effort and would exclude old FM's that are no longer updated, while the result will also be inaccurate and static meaning you won't see an AI reflected as they walk on a shiny metal plate for instance. If PBR with realistic graphics can be a hope for 2.12 or later, we'll definitely want to do it right rather than using a limited / limiting system.

A technique came to mind that might just work for our engine and setup. I wanted to share it here before I forget the specifics; This might already be a common practice and even have a definition, for the purpose of this thread I'll just describe it as I originally imagined it and feel this would work for our engine.

The idea is we'd use reflection probes but in an automated fashion: A probe is automatically spawned in every valid area (within bounds) in the player's view, at a given grid unit size. For example: If the grid scale is 16, a probe may exist at position '0 -48 16' another at ''0 -48 32' and so on. Every point projects its result on all surfaces in its radius which contain a specular channel masking it, the best alternative presently available till we were to convert all textures for PBR support. The cool thing is the same cubemap can also be projected as a light source, allow for global illumination in addition to just reflections! This would be similar to how the Irradiance Volume works in Blender / Eevee except each dot renders a little cubemap from its perspective.

I already know what everyone is rightfully thinking: This is going to kill performance! After all each probe needs to produce a render from its perspective, and being a 360* panorama it will open portals in all directions. Normally that would be insane, but I thought of various ways in which the impact could be greatly minimized to very bearable amounts.

• The frame buffer of each probe will be at a very small resolution by default since much detail shouldn't be needed. Even 64x64 per cube face might do.
• Each probe only needs a draw distance double its grid size, given it only has to see as much as is necessary to fill the gaps between it and its neighbors. So if the grid size is set to 64, each probe would only have a draw distance of 128 to cover the space in between its neighbors, nothing beyond that would exist to it.
• Only probes the player can see would ever be spawned and calculated; If the view frustum doesn't overlap the virtual cube who's corners touch that probe's neighbors, the probe is dropped from memory. Probes are also only spawned in a valid visible space, never out of bounds including rooms culled by portals.
• A draw distance after which probes are removed or not spawned can also be included. This would further help by making any probe further than X distance be ignored, slowly fading away as to not noticeably pop in and out of existence. Reflections / GI are a discrete effect you'll only see up close.
• Similar to lights and shadows, the result of a probe should be cached and not calculated unless necessary. This means that unless something moves within radius of that probe its cubemap won't be rendered again. Probes would only be updated either when they first come into the player's view, or if something touching their cube has moved. Note that particles and lights with animated textures would have to count as you may see them in a detailed reflection, candles and torches would force constant updates per-frame for probes they intersect.
• If with all that performance is still affected, frame skipping is also a solution: Reflection probes can update at a lower frame rate to further decrease their impact. If you have a 60 Hz monitor and are running TDM at 60 FPS max, reflections could run at 30 / 20 / 10 FPS without looking out of place. They could in fact be defined as a fraction of your average framerate, so for the FPS you get you can decide whether it's going to be 1/2 or 1/4 or so on of that... this would have the added advantage of exponentially gaining back FPS the lower your FPS goes.

There are several reasons why I believe this would be better than mappers manually placing new probe entities:

• Extra work is required for the mapper, who needs to figure out how to cover each area in probe lights. Every piece of the map would need to be encompassed in a reflection / GI probe otherwise you won't get shiny surfaces or bounce lighting which will look out of place.
• Most existing FM's will never be updated to use this: Only maps created or updated after the feature is introduced would benefit, anyone playing old missions will get boring visuals without reflections / GI which will be inconsistent to new ones. I strongly believe this should be done as an universal effect like SSAO.
• Single large probes will produce inaccurate results; The larger a cubemap is, the more drift and a fake results you get with distance from its center. This can be mitigated by using parallax corrected cubemaps which should be used for automated cubemaps too... none the less you get a single point of vision for a large room which makes the result inaccurate the further you go... with an automated approach you could have many probes in a dense grid (if your hardware allows it) for a much more accurate result at any position and angle.

What are your thoughts on this solution, do you think it's realistic and can work out? I do believe it should be either this or screenspace reflections the way they're done in Godot or Blender / Eevee. If SSR isn't the right choice for our engine, reflections and global illumination alike could be captured using a global grid of capture points shining within their respective areas.

Edited January 31, 2023 by MirceaKitsune

[MirceaKitsune]

Just realized a flaw in my proposal: Each probes view range can't be just its distance from the neighboring probe, as that distance will also be the maximum distance of the reflection itself. At a grid size of 64 for instance, a probe in front of a wall would only reflect at most 64 units behind that wall. So reflection distance should be minimum the grid distance doubled to avoid gaps, but also higher if you want your reflection to fade further. This would mean some bad news for performance unfortunately.

An similar yet different alternative would be for each point to capture and retain a single color, acting as a 3D pixel / point of light altogether. This would automatically emanate light and show up in specular channel without requiring material changes, tackling both GI and reflectivity at once... the grid size would be the resolution of the result in this case. But now we run into the issue that points behind the player need to be updated... in this case it would be best to calculate and remember all voxels inside a room the moment its portal is opened, only updating ones within radius of a light or object when there's movement near it.

[nbohr1more]

I don't know where you came to the conclusion that any "plan" involved manual probe placement?

If you are discussing a "probe system" you are naturally discussing some sort of automated spawning mechanism

and most likely a jittered grid. RBDoom3BFG does roughly what you are describing, auto generated grids of probes.

In concept, we would probably incorporate probe generation into dmap but it's certainly possible to also allow mappers to manually place probes.

All this has been true of probe systems going back to (arguably) the first probe system "precomputed radiance transfer" in HL2.

TDM's cubemap based ambients already allow manual placement but notably are deficient in comparison to a proper probe system and have been this way since v2.05.

[MirceaKitsune]

43 minutes ago, nbohr1more said:

I don't know where you came to the conclusion that any "plan" involved manual probe placement?

If you are discussing a "probe system" you are naturally discussing some sort of automated spawning mechanism

and most likely a jittered grid. RBDoom3BFG does roughly what you are describing, auto generated grids of probes.

In concept, we would probably incorporate probe generation into dmap but it's certainly possible to also allow mappers to manually place probes.

All this has been true of probe systems going back to (arguably) the first probe system "precomputed radiance transfer" in HL2.

TDM's cubemap based ambients already allow manual placement but notably are deficient in comparison to a proper probe system and have been this way since v2.05.

Got it, and that's great news! I thought the initial plan was to create a new type of map light that captures a cubemap at map start and reflects it. I was thinking that would be better than existing skybox lights, but very inaccurate and unable to capture moving objects in reflections. A global grid based solution will be complex but I do believe worth it in the end.

Also am I correct to assume this can tackle both reflections and global illumination? Technically you can project the captured cubemap mirrored in global space for a reflection, as a cube light for GI. Will definitely be amazing if both could be tackled at the same time with a common implementation.

Edited February 1, 2023 by MirceaKitsune

[nbohr1more]

I believe the plan was to first get better reflections working then move towards better surface shading and finally incorporate the probes into the ambient_world ( aka probe GI ). Hard to say but it may just be easier to examine what can be ported from RBDoom3BFG and add the full feature-set in one big change.

If someone wanted to examine the shaders from that project and try to make them comply with our design requirements ( OpenGL GLSL 3.3 ) that would make it easier to begin such an evaluation. For non-team members, I think that only hmart, jonri, and totallytubular ( maybe Filizitas ?) have the skill-set to do this type of thing so it will probably wait until a team member has time to begin experimenting.