Shadow maps behavior

[stgatilov]

As everyone know, right now we have both stencil shadows and shadow maps in the engine, and the player is mostly free to choose whichever he likes more. Due to the reasons I provide below, stencil shadows will most likely be dropped in some future, but that future is definitely not here yet.

Unlike shadow maps, stencil shadows do not support:

1. Alpha-tested shadow occluders.
2. Shadows on translucent surfaces.
3. Volumetric lights with account for shadows.
4. Soft shadows with contact hardening.

Point 4 can be ignored, point 3 has an automatic workaround by forcing lights with volumetrics to shadow maps.
Points 1 and 2 result in major difference in behavior between the two modes.

For the reference, we have one more technical difference:

• Front faces cast shadows in shadow maps mode, but don't cast them in stencil shadows mode.

I recall how initially volumetric lights had their shadows disabled in stencil shadows mode, and mappers were not happy because that's a big difference in behavior. One issue was that every mission has to be beta-tested in both modes.

That's why right now I mostly follow the same strategy:

• Alpha-testing is disabled when rendering shadow maps (since 2.12), unless you enable r_shadowMapAlphaTested.
• Shadows are disabled on translucent objects (since today's 2.13 dev build), unless you enable r_shadowMapOnTranslucent.

I wonder if this is still what majority of the mappers think is best?

[nbohr1more]

I think this will always be a divide.

Engine coders will want a universal behavior that acts the same way across all missions. So if we want to offer the additional benefits of Shadow Maps, we will need to break lots of missions and then work to fix them.

Mappers will probably want material or entity flags that force these behaviors so that they can force usage where they prefer.

I think defaulting to consistent with Stencil rules is a good idea. Maybe this is a good case for using mission.cfg and have it force the cvars to enable the features?

[datiswous]

14 hours ago, stgatilov said:

Due to the reasons I provide below, stencil shadows will most likely be dropped in some future, but that future is definitely not here yet.

Unlike shadow maps, stencil shadows do not support:

1. Alpha-tested shadow occluders.
2. Shadows on translucent surfaces.
3. Volumetric lights with account for shadows.
4. Soft shadows with contact hardening.

Point 4 can be ignored, point 3 has an automatic workaround by forcing lights with volumetrics to shadow maps.
Points 1 and 2 result in major difference in behavior between the two modes.

For the reference, we have one more technical difference:

• Front faces cast shadows in shadow maps mode, but don't cast them in stencil shadows mode.

With VL, uses shadow maps in Stencil Shadow mode. Why is it not possible to enable all features that are specific to shadow maps by forcing shadow maps in those situations? Settings menu could have an extra option to force Shadow maps features when using Stencil Shadows.

Stencil shadows look much better and give much better performance, but shadow maps have more features. I would use both to get best of both.

What are Alpha-tested shadow occluders?

[HMart]

I know you didn't asked for non TDM mappers opinion but man this decision makes me really sad,   imo after so many years, with both shadow systems and still deciding to hard limit shadow maps, to only stencil can do, is like cutting the wings of a bird.

Has a player, I can't wait for the day you guys finally decide to remove stencil shadows from this engine.

EDIT: Ups I didn't saw the poll at first so my comment was a tad precipitated sorry about that. 

 

Spoiler

For example, we could have add trees leaf's casting shadows by now, in many missions, making for some cool and moody shadow casting scenes, like for example, a courtyard with a big tree in the middle, casting a big shadow unto everyone and everything passing beneath, or a volumetric light casting thought the leaf's of a big tree and throwing a cool spooky volumetric shadow unto the scene, etc. We don't have such scenes in any mission, imo only because stencil shadows are the default. And if they, even if not removed from the engine, were just not thought has the default shadow system anymore, I believe things would change fast.

Every time I play a TDM mission and see a cool tree but only the trunk cast shadows, I get really disappointed, because I know is a artificial limitation now, not a impossibility of the tech like it was before.

The majority of the gaming industry has changed to shadow maps, ages ago. And I would bet money that if you guys, did the same, no one will miss the stencil shadows, not when they finally realize, what they have been missing out.

Enough said, you guys decide.

 

Edited March 8 by HMart

[nbohr1more]

3 hours ago, datiswous said:

With VL, uses shadow maps in Stencil Shadow mode. Why is it not possible to enable all features that are specific to shadow maps by forcing shadow maps in those situations? Settings menu could have an extra option to force Shadow maps features when using Stencil Shadows.

Stencil shadows look much better and give much better performance, but shadow maps have more features. I would use both to get best of both.

What are Alpha-tested shadow occluders?

I think the short answer is "it is possible... but" with the "but" meaning that we have some serious performance challenges. The first RBDoom3BFG release that supported Shadow Maps only applied them to transparent surfaces ( which stencil cannot do ) but the use of both systems in the same scene carried a heavy toll. As I recall, hybrid stencil mode was eventually dropped there too.

Still, stencil can look so much cleaner than shadow maps for large volumes and large casters it's a shame to lose them. And hybrid stencil \ maps rendering is still much cheaper than ray-traced shadows or penumbra wedges so it might be worth it to expand the hybrid mode anyway as a sorta "ultra quality" mode.

[datiswous]

47 minutes ago, nbohr1more said:

but the use of both systems in the same scene carried a heavy toll.

Are you saying using (only) shadow maps in missions with a lot of VL gives better performance than using (both) stencil shadows in combination with shadow maps in those scenes?

[nbohr1more]

5 minutes ago, datiswous said:

Are you saying using (only) shadow maps in missions with a lot of VL gives better performance than using (both) stencil shadows in combination with shadow maps in those scenes?

Yes, but there are qualifiers to that statement. Stencil is more CPU bound ( and fillrate bound ) and Maps are more GPU memory bound and rely more on texture bandwidth. If lighting mechanisms were perfect the hybrid might not be so bad but both systems often overdraw and waste resources on unseen geometry so you end up with significant amounts of scene duplication. Maybe moving to a deferred model would reduce that duplication but then you end up with having to pack data into buffers and the inevitable quality loss and artifacts of compressing everything into a buffer. Trade offs in all directions.

[Daft Mugi]

On 3/7/2024 at 2:59 PM, stgatilov said:

Due to the reasons I provide below, stencil shadows will most likely be dropped in some future, but that future is definitely not here yet.

Losing the option for stencil shadows would be extremely sad. TDM has the best stencil shadows implementation I've ever seen, and they look far better than shadow maps. They also perform wonderfully.

Stencil shadows not supporting shadows on translucent surfaces is a bummer, but that's a fine tradeoff.

Personally, I'd rather not see alpha-tested shadow occluders, because they look awful and are distracting. Shadow maps in general look bad and pixelated -- quite immersion breaking.

Another issue with shadow maps is that they substantially increase the usage of the GPU. For those using laptops or integrated GPUs, it could cause performance or heat issues. For those with decent dedicated GPUs, increased wattage (and therefore heat) can and does turn on its GPU fans. The wattage on my AMD RX 6700 XT can become tripled when using shadow maps. That's undesirable.

The advice I give players is: There really isn't a "best" shadows implementation these days. Some devs and players prefer one over the other. Regardless of which is chosen, stencil shadows and maps are used where needed. So, I'd try out both and pick the one you prefer.

Here's Fen complaining about shadow maps.

Complaint #1 (7:28):

Complaint #2 (23:46):

In his next video (Written in Stone - 4 - Chocolate Coated Brendel Bar), he turned on stencil shadows, and as far as I know, he never complained again.

 

The following area in Written in Stone shows a good example of alpha-tested shadow occluders. Yes, technically the moving leaves cast shadows, but it looks awful in game, in my opinion. I know others like it. The same scene using stencil shadows looks amazing in game. (The screenshots don't do a good job demonstrating how it looks in game.)

 

The current hybrid system is fantastic. It gives people choice. Those who prefer stencil shadows can use them; those who prefer shadow maps can use them. People become attached to the "art" and "style" of a mission, and changing the shadow implementation changes the art and style.

If one must be dropped, I'd say shadow maps should be dropped from the menu settings. Use shadow maps for volumetric lights and stencil shadows everywhere else.

And, I agree with nbohr1more that defaulting to rules consistent with stencil shadows is a good idea.

[datiswous]

8 hours ago, HMart said:

And I would bet money that if you guys, did the same, no one will miss the stencil shadows, not when they finally realize, what they have been missing out.

If shadow maps would look good I would agree, but they just don't.

[HMart]

1 hour ago, datiswous said:

If shadow maps would look good I would agree, but they just don't.

ok fair enough but you have still not seen TDM using shadow maps to the full potencial, so I would wait until then before making a final decision but ok. Also unlike stencil that looks crisp at any quality level shadow maps only look good at res beyond 1024 and up and also dependes on the tech used, some have noisy edges by nature, because of low res penumbra effects.    

But to me today they look good enough and the fact they may look less than equal to stencil, maybe the particular TDM implementation, with all respect for those that implemented them. 

I have played a ton of games with shadow maps that look good at lest to me.

Stalker for example: 

Metro 2033

Dead Space

Call of Juarez

Arkane Prey made on Cryengine

and many many more. 

[HMart]

And those pics that Daft Mugi posted do show the problem with TDM maps, but I also have to say, that is a problem with TDM shadow maps, not with maps in general, I never seen such ugly issues in Stalker for example or Call of Juarez and many other games. 

Perhaps those shadow maps issues arise from people not thinking of shadow maps at all, when creating their lights, shadow maps do require a little more tweaking to look good, specially at grazing angles, because they are literally textures, so perhaps moving the lights a little solves those problems or even increasing the shadow map res.

Just think about this, if shadow maps where that ugly, the gaming industry would never deprecate stencil shadows and we would still be seeing a ton of games using them, but is totally the contrary, games with stencil shadows, are the minority, even idSoftware removed them, I think since Rage (i'm not totally sure about Rage...) and up (besides obviously Doom 3 BFG). 

For example: 

Wolfenstein II

Dishonored 2 

Evil within 2

(this particular shot is impossible with stencil shadows, because of the blood that is a particle effect with alpha)

etc.

For obvious reasons TDM fans have a good opinion of stencil shadows and I comprehend that, they do look very good, TDM and Wolfenstein 2009 are the games with the best soft stencil shadows that I haver seen, but maps can also look good, if given the chance and used to their full potencial. 

another game with good stencil soft shadows

Thou tree shadows on this game seem to use something else or literally stencil shadows because sometimes they look like a blob on the floor instead of seeing the individual leaf. 

 

Edited March 9 by HMart

[HMart]

And btw I'm not against using both shadow systems, I just think that supporting two systems, is more work, but if the TDM engine team is okay with that, then go for it they have my full support, for what is worth.

I just personally don't like that shadow maps are being pushed back is all.

But you guys do what you want, this should be a mission makers decision, I'm just a "customer" I will have no choice but accept what you guys decide.   

Edited March 9 by HMart

[chakkman]

"Ignorant" opinion: I doubt that 99,9% of the general population would notice the difference in game, so, this is a bit of a geeked out discussion. Also, if there are some significant downsides which can't be overcome, then I'm always for the solution which offers more and causes less trouble, regardless whether or not it looks 0.001% worse.

Edited March 9 by chakkman

[Frost_Salamander]

I'm no graphics nerd, but I can barely tell the difference between the two.  What I can tell from messing around a bit:

• soft shadows of low quality look like garbage with both maps and stencils (EDIT: wrong, I didn't realize I was looking at shadow maps for a volumetric light shadow.  Stencil definitely looks better).
• increasing soft shadow quality decreases performance in both implementations.  I think the CPU/GPU of the end user would influence which is gives better performance
• both maps and stencils can produce a pixelated shadow if you look close enough. (EDIT: wrong.  Again, was looking at a volumetric light shadow).

I tend to use maps, because for whatever reason I seem to get a few more FPS out of them.

As a mapper, I am certainly NOT interested in endlessly tweaking a scene to make the shadows look perfect.  I just don't care enough.  If they look shockingly bad I will put some effort into it though (which will probably mean just disabling them for the offending entity).  It's rare that I feel this is necessary though.

So I guess I don't really get the argument that stencils are amazing and maps are crap.  I just don't see it or am too dense to notice. (EDIT: indeed I was being dense.  I was comparing shadow maps with shadow maps because I was looking at a volumetric light shadow).

Also, in my last couple of missions I had graphical bugs that only showed up with stencil shadows enabled.  It would be nice to not have to deal with that all the time.

 

Edited March 9 by Frost_Salamander
correct misconceptions about stencil shadows

[HMart]

On 3/9/2024 at 12:45 PM, Frost_Salamander said:

both maps and stencils can produce a pixelated shadow if you look close enough.

Not to start a discussion on this but just for correctness, that is not true at all, unless you are using a really low polygon shadow mesh, if you see a pixelated shadow with stencil, then that is a shadow map or a mix of both stencil and shadow maps. 

Stencil shadows are crisp and well defined, they have no resolution setting or quality setting (besides turning off shadows), because they are literally geometry (triangles) being projected unto surfaces, while shadow maps are pixels (textures) so you can control how sharp they are.

TDM has a mix shadow system, where some effects even in stencil shadow mode, will use shadow maps, for example volumetric lights shadows, are shadow maps.

 

Stencil shadows

 

Shadow maps

Edited March 16 by HMart

[Frost_Salamander]

21 minutes ago, HMart said:

... where some effects even in stencil shadow mode, will use shadow maps, for example volumetric lights shadows, are shadow maps.

Apologies, this is where I went wrong.  It happened to be a volumetric light shadow I was looking at 

[stgatilov]

I'd like to remind that the core question is: what should we do now?

Stencil shadows don't support a lot of things, but they are pixel-perfect in exchange.
Indeed, stencil shadows won't be removed while shadow maps look like crap here and there (and are not even self-sufficient).

[HMart]

27 minutes ago, stgatilov said:

... (and are not even self-sufficient).

What do you mean?   Btw I don't want to continue with my "complaint" or push the subject further, I've said enough already and is your guys decision.  I'm just genuinely curious what do you mean by that. 

[datiswous]

I thought I've read somewhere that if you enable shadow maps, that on some lights, stencil shadows are still forced, for some reason. Correct?

I'm not sure if I should vote. This seems more for fm-bakers.

Edit: I would go for option 1

Edited March 9 by datiswous

[Frost_Salamander]

1 hour ago, stgatilov said:

I'd like to remind that the core question is: what should we do now?

Sorry - my response was a long way of saying I'd be fine with going to maps only as a strategy.  I voted option 1.

But this is without understanding some things in your top post.

What is an 'Alpha-tested shadow occluder'  and how does it affect me as a mapper?

What is an example of a translucent object that won't cast shadows?

[nbohr1more]

Both mean basically the same thing. Stencil is incapable of shading anything other than geometry so if you want textures that have transparent parts to cast shadows from the non-transparent part of the texture you need to accept that stencil mode will look different to maps mode. Likewise if you want shadows to render onto the semi-transparent part of textures.

[stgatilov]

1 hour ago, Frost_Salamander said:

What is an 'Alpha-tested shadow occluder'  and how does it affect me as a mapper?

You can't make trees/grass cast proper shadows with stencil buffers, because they are rendered as bulky quads with alpha-tested texture.

Quote

What is an example of a translucent object that won't cast shadows?

Imagine shadows falling on the surface of water, or on the glass.
You will never see it with stencil shadows: the shadows are computed only on single solid surface closest to viewer at each pixel.

[Frost_Salamander]

I feel I should add - due to my blunder with the volumetric lights shadows I didn't realize I that I was looking at soft shadows for maps instead of stencil when I was comparing them.  After some more playing around I definitely see that soft shadows with stencil looks better.  So I get what all the fuss is about now.

 

 

[MirceaKitsune]

If shadow maps support more detail that stencil shadows can't offer, I'm absolutely in favor of enabling them: It makes sense to use them for obtaining new details and effects that weren't possible previously. Geometry based shadows are technically a legacy thing anyway, I imagine that once every player and their hardware work well with shadow maps then stencil shadows may someday be deprecated.

[HMart]

17 hours ago, MirceaKitsune said:

If shadow maps support more detail ...