New shadow leaks in 2.12 dev builds

[Daft Mugi]

There's a lighting issue in "William Steele 1: In the North" at

-1153.92 3523.22 -955.75 29.8 -145.6 0.0

There's a candle on the other side of the wall. In 2.12 dev, that candle's light leaks through the wall at the base of the door as seen in the video, comparing 2.11 and 2.12-r10451.

 

[Daft Mugi]

The lantern in the Training Mission has a shadow casting issue. The lantern is at viewpos:

1550.89 -2142.79 354.25   26.8 36.3 0.0

Sometimes it casts a shadow, and other times it does not. Which is the correct behavior?

The video first shows it NOT casting a shadow and then shows it casting a shadow:

 

[stgatilov]

18 hours ago, Daft Mugi said:

There's a lighting issue in "William Steele 1: In the North" at

-1153.92 3523.22 -955.75 29.8 -145.6 0.0

There's a candle on the other side of the wall. In 2.12 dev, that candle's light leaks through the wall at the base of the door as seen in the video, comparing 2.11 and 2.12-r10451.

I wrote about it here: https://bugs.thedarkmod.com/view.php?id=5172#c16126

This is broken by my shadows optimization.
Basically, this map breaks Doom 3 assumptions by replacing some parts of world geometry in the doorway with caulk.
No idea how to fix it in engine (aside from disabling all the optimizations).

UPDATE: We had a major performance issue with shadows.

Imagine we have a small fully closed house with a light source inside it. The light has huge light radius, which covers a lot of stuff outdoors.
Previously, all that stuff did cast shadows, despite everyone agrees it is not necessary. This is no longer the case: the latest dev build won't cast shadows for the entities outdoors.

Now imagine that part of the wall of the house is replaced with caulk. The engine still thinks it is a solid wall because it's fully enclosed with opaque surfaces, so it drops shadows of outdoors entities. But outdoor objects might get lit.

This map does the same.

 

[MirceaKitsune]

From what I understood last time, lights now use portal culling and solid walls to dump unreachable entities, correct? So if a torch is located in a room and its radius box covers entities in a neighboring room but the light doesn't also touch any portals through which it can reach that room, those entities are ignored.

I'm curious if lights also use a projection check like the player view. So even if a room is opened by portals from the perspective of the light, an entity is still ignored if the light's origin can't shine over the bounding box of the entity through any open portals leading to it. Hopefully this is the case by now, hope it's planned if not as it sounds optimal.

Last time I asked, it couldn't be determined if spotlights still calculate entities in an omnidirectional 360 field like normal lights, instead of only looking at portals and entities within their cone. If this is the case I support making some of the default hooded lights targeted as an optimization, given some are omnidirectional and self shadowed in a way that the upper area is covered but still calculated. An obstacle I remember encountering is at the moment, there's a limitation in both DarkRadiant and TDM engine that prevents targeted lights from having an origin offset, even if the origin spawnarg is set the center falls back to the entity's position making this unusable on some lamp models.

Edited October 12, 2023 by MirceaKitsune

[Daft Mugi]

2 hours ago, stgatilov said:

No idea how to fix it in engine (aside from disabling all the optimizations).

What are our options?

1. Let old missions be broken.
2. Find a workaround for old missions, such as disabling the optimizations.
3. New missions set a flag for the new optimizations.
4. Other?

[thebigh]

I suppose option 5 would be to edit the mission itself to patch the caulk. Something like this was done to In a Time of Need to fix the LOD on some torches that a TDM update had turned into game-ruining proximity detectors. I'd be very wary of making changes to other peoples' missions though.

 

Does the stray light in WS1 affect the light gem if you walk on it? If not, I'd say just leave it. It's a minor cosmetic issue.

[stgatilov]

7 hours ago, Daft Mugi said:

New missions set a flag for the new optimizations.

This would be almost the same as "disable new optimizations completely".

Perhaps first see how many missions are broken.
 

[Daft Mugi]

9 hours ago, stgatilov said:

Perhaps first see how many missions are broken.

Here's another instance found in "The Painter's Wife" at viewpos:

688.45 -205.07 -127.75 57.2 -124.4 0.0

The following video compares 2.11 and 2.12-r10451:

 

The new optimizations are a good thing as they allow mappers to build more, so I think they should be kept. Also, I've heard that some missions are currently being built with the assumption that the optimizations will be available.

Without knowing much about this, would two code paths be doable: one for old missions and one for new missions? Perhaps DarkRadiant can add a flag by default for the new optimizations when making a new mission?

If that doesn't seem like a good strategy, what other solutions can be tried?

[snatcher]

I understand and assume @stgatilov's optimizations are a leap forward (or go in that direction).

Having two or more ways of doing things means mappers can still get away with the "old ways" (the ball keeps rolling). Although deprecated, the devs must take into consideration alternatives that were left behind. Maintaining backwards compatibility can sometimes hinder development in the long run (I rather not go down the rabbit hole). Letting things break sometimes is the best solution: somehow, someday, someone will fix it.

We must assess the impact and then assess potential solutions. I am impressed by your capabilities to detect these issues @Daft Mugi! How do you do it? Do you somehow sense where or what to look for?

[Daft Mugi]

6 minutes ago, snatcher said:

I am impressed by your capabilities to detect these issues @Daft Mugi! How do you do it? Do you somehow sense where or what to look for?

I guess I'm highly sensitive to visual and auditory distortions.

I found these while testing code I'm writing for something entirely unrelated to graphics rendering. I test my code in several scenarios in several missions, so sometimes I find other issues along the way.

[snatcher]

I see what you see (red), plus something else (yellow). Major or minor issues can be anything, anywhere.

Is there any way we can detect an author / pattern and somehow check missions / locations where the issue can be truly relevant to the point of making a scene look completely broken? I would like to help.

[Wellingtoncrab]

The issue with the bounds of visportals "glowing" has been quite apparent in my WIP:

If it is helpful in resolving the issue they are only visible when soft shadows are enabled.

Using the "Maps" lighting mode also seems to reduce how visible they are.

The performance improvements are quite significant so it would obviously be good to keep as many of the optimizations as possible.

[stgatilov]

On 10/13/2023 at 8:37 AM, Daft Mugi said:

Without knowing much about this, would two code paths be doable: one for old missions and one for new missions? Perhaps DarkRadiant can add a flag by default for the new optimizations when making a new mission?

If that doesn't seem like a good strategy, what other solutions can be tried?

It is possible to add:

1. Entity spawnarg --- means that this entity should casts shadow even if it's beyond walls according to area/portals.
2. Worldspawn spawnarg --- same meaning as p.1, but applies to all entities in the map.

On 10/13/2023 at 1:37 AM, thebigh said:

I suppose option 5 would be to edit the mission itself to patch the caulk. Something like this was done to In a Time of Need to fix the LOD on some torches that a TDM update had turned into game-ruining proximity detectors. I'd be very wary of making changes to other peoples' missions though.

This is OK if the mission is rather up-to-date.

The older missions (like before 2.08 maybe) were compiled with older dmap. While I tried to add some compatibility cvar for dmap, it is still likely that fresh dmap will behave differently. You'll have to test the whole map thoroughly, and probably fix dmap-related issues everywhere.

On the other hand, adding spawnarg is risk-free, since it does not require fresh dmap.

12 hours ago, snatcher said:

I see what you see (red), plus something else (yellow). Major or minor issues can be anything, anywhere.

Yes, the major part of the problem is that these caulky walls don't manifest any issues until you carry a light source into specific position. The original game would be buggy only if simple static light is in that position, which is much easier to test.

Quote

Is there any way we can detect an author / pattern and somehow check missions / locations where the issue can be truly relevant to the point of making a scene look completely broken? I would like to help.

Maybe filter away entities in DarkRadiant and look for visible caulk.

I think mappers put caulk on surely-invisible part as an optimization, since otherwise this brush surface would actually be rendered (although only to depth buffer, which is cheap). But some of these surfaces should cast shadow, so it is no clear which can be replaced with caulk and which not.

 

My suggestion is:

1. Create a dedicated thread regarding light/shadow issues. Move the messages from this thread into it too.
2. Analyze cases one by one. It can easily happen that some issues are caused by other things.
3. Decide what to do when we have enough info.

[stgatilov]

On 10/13/2023 at 8:37 AM, Daft Mugi said:

Here's another instance found in "The Painter's Wife" at viewpos:

688.45 -205.07 -127.75 57.2 -124.4 0.0

The following video compares 2.11 and 2.12-r10451:

This is certainly a different issue.

It renders bright lines along scissor rectangle of some visportal area (happens near visportals).
It might be the same as 4660, but much more likely to appear after the optimizations.

UPDATE: No, they are not related.
I'll soon commit a fix for this scissor issue, and those two issues from 4660 still happen.
In fact this problem is clearly distinguished by the fact that bright lines are always horizontal or vertical (go along light scissor rectangle), and they move as player moves even if light + shadow casters are still.

UPDATE: Hopefully fixed in svn rev 10463.

[Daft Mugi]

Found another light leak in "William Steele 1: In the North". I imagine there are more in WS1, so I won't post anymore examples in that mission. I wonder if the other "William Steele" missions have doorways built in the same manner.

viewpos:

-664.04 2894.9 -635.75 10.2 9.9 0.0

[MirceaKitsune]

I'm on dev16829-10455: With a new map I'm working on I keep noticing lights leaking through walls. The walls are thin caulk brushes (8 units thick) as I'm using the building modules. Sometimes a lamp on the other side will shine straight through the wall over the floor or ceiling. In one case I noticed this happen even if the room where that lamp is located was behind a closed door / portal, meaning the engine should know the light doesn't reach you and should be disabled entirely yet it still worked.

I've reworked my map since so I no longer have it. But in this screenshot I took, the brightness you see on the ceiling in the middle is from a gas lamp mounted on the wall on the other side. In this case closing the door to the right while standing in the same spot makes the glow go away once the door is shut, but like I said there was another case where the light shined through a caulk room even when its door was closed and I was standing outside.

[AluminumHaste]

Make sure the gas lamp light is not set to no shadows. Same with the wall texture, make sure you didn't accidentally use _ns version of a texture

[MirceaKitsune]

25 minutes ago, AluminumHaste said:

Make sure the gas lamp light is not set to no shadows. Same with the wall texture, make sure you didn't accidentally use _ns version of a texture

It's not that no: I didn't modify the properties of the default entity or its flame, in this case it's the standard atdm:lamp_oil_wall_lit entity... also I have player shadows enabled, the player as well as other architecture elements cast shadows fine. Walls are the building modules, eg: model models/darkmod/architecture/modules/interior_set01_corner.lwo with skin diamond_wallpaper as a test.

This is the closest to the setup I still have: The origin of the light is well beyond the face of the module surface for shadow casting. Though this shouldn't even matter since the light is in the other room and the caulk brush should itself mark this.

I think I noticed this on other maps too while playing, but only now saw it obviously enough to realize there's likely an issue somewhere. I remember seeing the glow of a light from another room shining on the floor / ceiling when it shouldn't, though I didn't document it at the time.

Edited November 4, 2023 by MirceaKitsune

[Frost_Salamander]

@MirceaKitsune try something other than caulk on that brush and see if it goes away. Make sure it's not a no shadows or transparent material. 

[AluminumHaste]

Maybe I misunderstood what you wrote, but caulk does not block light.

[stgatilov]

The modules are comprised of models, they are completely ignored by the area-portal graph. And now they don't cast shadows if it is proved that light beams travelling though visportals don't touch them.

If you use caulk to create areas, then there is nothing that would cast shadows: brushes don't cast shadows because they don't exist (caulk), and modules don't cast shadows because according to area-portal graph they are not hit by light to begin with.

There is internal material called "shadowcaulk".
As far as I understand, it is not drawn as ordinary surfaces, but it does cast shadows. I think in order to work properly with new optimization, shadowcaulk should be preferred to caulk for the brushes.

[MirceaKitsune]

28 minutes ago, stgatilov said:

The modules are comprised of models, they are completely ignored by the area-portal graph. And now they don't cast shadows if it is proved that light beams travelling though visportals don't touch them.

If you use caulk to create areas, then there is nothing that would cast shadows: brushes don't cast shadows because they don't exist (caulk), and modules don't cast shadows because according to area-portal graph they are not hit by light to begin with.

There is internal material called "shadowcaulk".
As far as I understand, it is not drawn as ordinary surfaces, but it does cast shadows. I think in order to work properly with new optimization, shadowcaulk should be preferred to caulk for the brushes.

Aha, thank you for this clarification! I see what's happening now: I was expecting that the building modules still cast shadows by default, which I was in fact hoping got disabled since their performance impact can be felt... in addition I thought texturing the walls with normal caulk would still work as they create closed rooms, but indeed caulk isn't a shadow caster so I shall replace them with shadowcaulk as suggested.

There's another bug in latest dev I was waiting to report: I noticed I can place electrical lights in DarkRadiant where they show up just fine, but when opening the map in TDM they do not spawn any longer. The console provides a warning which suggests an invalid newline causing the engine not to load the def though DR doesn't mind it. I don't override the file in my FM nor had any errors while running the installer and updating so I'm presuming something got broken.

WARNING:file def/tdm_lights_static_electric.def, line 612: newline inside string

Edited November 4, 2023 by MirceaKitsune

[stgatilov]

5 minutes ago, MirceaKitsune said:

There's another bug in latest dev I was waiting to report: I noticed I can place electrical lights in DarkRadiant where they show up just fine, but when opening the map in TDM they do not spawn any longer. The console provides a warning which suggests an invalid newline causing the engine not to load the def though DR doesn't mind it. I don't override the file in my FM nor had any errors while running the installer and updating so I'm presuming something got broken.

WARNING:file def/tdm_lights_static_electric.def, line 612: newline inside string

Yes, this will be fixed in next dev build.

[MirceaKitsune]

Though this is a more general question and less related to light leaking bugs per say, I'd still like to ask to be sure: Does using shadowcaulk instead of caulk teach lights to better detect rooms and ignore entities behind such brushes, thus texturing walls with it improves performance? At first I thought that's the case but likely not since I think the recent light optimizations use visportals to check rooms outside of the light's field of vision.

[stgatilov]

20 hours ago, MirceaKitsune said:

Though this is a more general question and less related to light leaking bugs per say, I'd still like to ask to be sure: Does using shadowcaulk instead of caulk teach lights to better detect rooms and ignore entities behind such brushes, thus texturing walls with it improves performance?

No, it does not.

Caulk, shadowcaulk, or any other opaque material are the same for the sake of area/portal structure.
All brushes with such textures are blockers of the same kind.

The difference is that caulk is not rendered, is not lit, and does not cast shadows.
Shadowcaulk is supposed to cast shadows (while still not being rendered and not lit).