Jump to content
The Dark Mod Forums

r_showShadows + shadow volumes


SteveL

Recommended Posts

My turn for a weird problem. Until recently, r_showShadows 1 drew the outline of shadow volumes for me in blue wireframe lines, visible through solid surfaces, which is correct. r_showShadows 2 drew them as pale blue solid shapes, which get hidden by solid surfaces, also correct. Now, option 1 draws them as black lines which are harder to see, and option 2 doesn't show anything at all.

 

It's not any recent change we've made to the engine. I tried it in 2.02 TDM release build and got the same problem.

 

I wondered if the new catalyst beta driver was at fault, so I reinstalled it. No change. I did a clean remove and reinstall of microsoft .NET, no change. I removed all ATI sofware and reinstalled the stable release version instead of the latest driver, no change.

 

I can't think what else to test. Would someone do me a favour please and confirm whether they can see any shadows with r_showShadows 2 turned on? This end, I'm seeing nothing: I'm losing both the shadows and the debug drawings of the shadow volumes.

 

??? :huh:

 

Edit: It turned out the blue colour wasn't correct after all.

 

Edit2: Updated the thread title because this has become a more general discussion since we got past the bug fix.

Edited by SteveL
  • Like 1
Link to comment
Share on other sites

If I'm not mistaken, one of the Sikkmod fixes was to give these shadows color in that mode. Perhaps that broke when those changes were backed out? (Shadow.vp)

Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Link to comment
Share on other sites

If I'm not mistaken, one of the Sikkmod fixes was to give these shadows color in that mode. Perhaps that broke when those changes were backed out? (Shadow.vp)

 

I was wondering whether it worked differently only in the 2.03 development code. That's why I asked for someone to try it for me, to find out whether it's a driver problem on my machine or a problem in the code. But if it's a code problem, that must mean that r_showShadows2 never worked.

 

Does r_showShadows 2 show anything on screen for you? (Anyone?) If I can get confirmation from someone else that it's not just my machine, I'll look into fixing it.

Link to comment
Share on other sites

You're right, it looked better with the Sikkmod shadow.vp, but it was still broken.

 

Somewhere along the way, we must have lost a line from shadow.vp:

MOV  result.color, vertex.color;

 

Our shadow.vp doesn't set the vertex colour at all, which is why my lines were black and why option 2 (which uses an additive blend to paint the shadow volume "surfaces") didn't show up at all.

 

Sikkpin's shadow.vp set it to a constant RGB 0-0-1 which is why everything came out blue.

 

But the engine is set up to set the colours itself, using different colours for front faces and back faces, and for different shadow optimisation routines, so you can tell how a given shadow got drawn.

 

It looks much better when you let the engine color it:

post-29566-0-44809200-1417937844_thumb.jpg

  • Like 1
Link to comment
Share on other sites

I added the missing line.

 

If you are visualizing shadow volumes it's helpful to set r_useDepthBoundsTest to 0 too, otherwise the engine will cull away parts of the shadow volume (a performance optimisation).

 

One small thing: maybe you can multiple the engine color by 0.5 or 0.7 - in sikkmod they are transparent, while the engine colored once seem almost solid. It might be better visuals when they are not that solid.

 

But you might want to test if that is actually an improvement, its hard to tell from static images.

 

Good work, tho!

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

It's just the picture and model that I chose making it look opaque. The drawnings are transparent, but the colour builds up if you have several layers of shadow volume to look through. I agree it'd probably work better if the colour got reduced a bit. I'll try tweaking it up and down while I'm experimenting.

 

The sikk version looks transparent no matter how many layers you have because it saturates the blue channel but leaves the others alone. That's not an improvement, because you can't see any detail... in the pic above for example, the two prongs of the generator look like a single shadow. In the engine-coloured one you can see the layers.

 

I can now see in my test map that some shadows are drawn using different routines. I have a mixture of red and yellow shadows. The red ones are "turboshadows" whatever that might mean. I plan to find out this morning...

Link to comment
Share on other sites

While we're on the subject, I've added some documentation to shadow.vp because it was cryptic, to say the least.

 

!!ARBvp1.0
# Vertex program for projected shadow volumes
#
# How this works [notes from SteveL #3958]
#
# The shape of the model (which can be any shadowcasting surface) is stretched
# out to infinity in the direction opposite to the light. This stretched shape is the
# shadow volume.
#
# The engine creates *two* vertices for every vertex in the shadowcasting surface, and
# passes them to this program to be drawn as a shadow volume.
# Each pair of vertices has the same xyz coordinates (using the model's axis), but one
# from each pair has 0 in its 'w' (4th) slot and the other has 1 in there.
#
# The 'w=0' coordinate is the one that ends up being projected to infinity by this program.
# The 'w=1' coordinate stays where it is.
#
# program.env[4] is set by the engine. It's the light vertex position, relative to the model origin.
#
# Step 1 below works out the vector from the light to the vertex, by subtracting
# the light's coordinate from the model vertex's coordinate. That, plus a
# w-value of 0, is enough to represent a point infinitely far away in the opposite direction to the
# light, because the w-value represents scale when the model coords are transformed into view coords
# in step 3. The xyz coordinates are scaled by 1/w.
#
# Step 2 performs a trick using the w=0 or 1 values: for the w=0 vertex,
# there's no change to the value in R0. The w=1 coordinate gets the light origin added back on,
# so it gets put back at its original vertex.position.
#

# declare the temporary variable
TEMP R0;

# STEP 1: RO = original position - light, this assumes light.w = 0
SUB	R0, vertex.position, program.env[4];

# STEP 2: RO -= original position.w * light
MAD	R0, R0.wwww, program.env[4], R0;

# STEP 3: transform the coordinates from the model axis to the view projection as normal
DP4	result.position.x, R0, state.matrix.mvp.row[0];
DP4	result.position.y, R0, state.matrix.mvp.row[1];
DP4	result.position.z, R0, state.matrix.mvp.row[2];
DP4	result.position.w, R0, state.matrix.mvp.row[3];

# For debug coloring ( r_showShadows )
MOV	 result.color, vertex.color;
END

 

Link to comment
Share on other sites

It's just the picture and model that I chose making it look opaque. The drawnings are transparent, but the colour builds up if you have several layers of shadow volume to look through. I agree it'd probably work better if the colour got reduced a bit. I'll try tweaking it up and down while I'm experimenting.

 

The sikk version looks transparent no matter how many layers you have because it saturates the blue channel but leaves the others alone. That's not an improvement, because you can't see any detail... in the pic above for example, the two prongs of the generator look like a single shadow. In the engine-coloured one you can see the layers.

 

Yeah, I got this from your post, but I didn't see that there were so many shadows making it opaque due to the many layers. But maybe you can still tweak it a bit, even a single shadow doesn't need to be that strong in the color.

 

I can now see in my test map that some shadows are drawn using different routines. I have a mixture of red and yellow shadows. The red ones are "turboshadows" whatever that might mean. I plan to find out this morning...

 

Please post the findings :) And thanx for the work!

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

One small thing: maybe you can multiple the engine color by 0.5 or 0.7 - in sikkmod they are transparent, while the engine colored once seem almost solid. It might be better visuals when they are not that solid.

 

I'm multiplying it by 0.2 currently. Yes, less colour definitely makes it easier to see detall and the number of layers of shadow volume. I'll probably drop it to 0.1 or less when I get out of my test map and into a proper one.

 

There are 7 different colours used in the engine for drawing shadow volumes, and the colour tells you how the shadow volume got calculated. Quite cool. 4 of them are probably never used any more, they're for GPUs that don't support shader programs. The ones we're likely to see are:

  • Green: worldspawn shadows generated and optimized during dmapping. These are independent models in their own right, they don't "belong" to the worldspawn that generated them in the same way that dynamic shadows belong to the model that casts them.
  • Orange: turbo shadows, which are dynamically generated shadows that got drawn by the shadow.vp shader program.
  • Red: turbo shadows where the player's view is partly or fully inside the shadow volume, so they need to draw extra shadow tris (end caps) to avoid errors in working out where the shadow hits. This is the same complication that was behind the z-pass / z-fail patent controversy.

If we are going to improve the way shadows are drawn, we should push for a single solution instead of 7. I plan to test ignoring the dmapped shadow volumes and force all shadows to be turboshadows, calculated by the shader program. The dmapped shadows might actually be a hindrance nowadays. Doing extra stuff in a vertex program is free for us performance-wise because we use it so little, and those dmapped shadow volumes don't benefit from the culling controlled by r_useDepthBoundsTest, which is more of a concern. Also, they have to calculate and draw end caps far more often than turbo shadows.

 

Another opportunity I've spotted is that turbo shadows draw way too many surfaces. shadow.vp doesn't identify which edges are silhouette edges, it relies on the engine for that, and the engine isn't very good at culling them. I mean this kind of thing:

 

post-29566-0-55729100-1417970552_thumb.jpg

 

That's a single func_static with the same texture everywhere. There's no need for the parts of the surface on the sticky-out bit to be casting shadows. That little shadow is completely covered by the shadow of the bigger box. Not sure yet whether it's fixable -- it will depend on whether the sticky-out bit is part of the same surface or not -- but I think it is and if so, then it can probably be fixed in shadow.vp to reduce the number of shadow surfaces that have to be drawn.

 

* Ignore the red colour in that pic. That was due to me messing round with the code. The shadow volume would normally be orange in this situation.

  • Like 2
Link to comment
Share on other sites

Steve, you probably know more about shadows than me, but I remember that the reason for the dmap-shadows was that multiple shadows can be combined, e.g. if you have a stack of static crates, the shadows drawn can be simpler than if you have individual crates.

 

But from your pic, it looks like that is no longer the case. Maybe this optimization got broken somewhere in between? Or it only worked for certain cases?

 

It definitely would be good to test if just having a unified shadow calculation to be faster. (Likewise, the way finishSurfaces() recalculates shadows might be expensive...)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

The pic above was a turbo shadow, so not related to dmapped shadows.

 

Like you, I assumed that dmap shadow optimisation would merge the smaller shadow into the bigger one, but for whatever reason it doesn't. Here's the same box reverted to worldspawn with its dmapped (green) shadows:

post-29566-0-05067600-1417995096_thumb.jpg

Link to comment
Share on other sites

I tried it but it didn't fix the above example. The default shadowOpt is apparently 1. I also tried 3, 4, and 5. The only ones that made a visible difference were 4 and 5, which added an extra shadow surface to the setup!

 

I won't work on dmapped shadows unless we find that using only turboshadows is a problem... It would mean trying to implement penumbra wedges twice.

Link to comment
Share on other sites

I plan to test ignoring the dmapped shadow volumes and force all shadows to be turboshadows, calculated by the shader program. The dmapped shadows might actually be a hindrance nowadays. Doing extra stuff in a vertex program is free for us performance-wise because we use it so little, and those dmapped shadow volumes don't benefit from the culling controlled by r_useDepthBoundsTest, which is more of a concern.

 

I look forward to the results of this test. It would seem surprising to me that calculating all shadow volumes dynamically would be faster than pre-calculating them at the map compilation stage, especially if you have lots of lights and geometry, but then our graphics cards do have a lot more processing power these days than when Doom 3 came out (I first started playing Doom 3 on a Radeon 9800XT with eight pixel shaders. My latest acquisition is a R9 290 with 2560 unified shaders).

Link to comment
Share on other sites

Cool. We can test it without a special engine build. Dmapping with shadowopt 0 will turn off dmapped shadows completely, and there's another option: you can make the game abandon its dmapped shadows and to start using turbo shadows by using a brief script to move all light origins a tiny amount. That will invalidate all the dmapped shadow volumes.

 

If the map has lots of complicated worldspawn in view, a load of verts will get loaded immediately after the light move for shadow.vp. But then they should sit there in the cache from frame to frame.

 

I haven't yet worked out whether sillhouette detection is happening every frame for turbo shadows, or whether that result is cached in the vbo too. That's one thing that could go wrong.

 

In the most advanced maps it's nearly all turboshadows anyway. Func_Static walls are turbo shadow users. On the other hand, we all know that moving lights are a big perf hit. I'm not sure why that should be. It shouldn't be down to shadow verts being uploaded -- they just sit in the vbo. Perhaps it's sillhouette detection that's the problem.

 

My next step is to figure out how the sillhouette detection works. I occurs to me that the way I've read about of doing sillhouette detection on the gpu would produce the same result we see in the pictures above. The method is to compare the normals of the two faces either side of a given edge, and if one normal faces the light and the other doesn't then it's a sillhouette edge. In my example above, that would leave the little sticky-out bit of the box casting its own pointless shadow. Perhaps that's what's going on.

Link to comment
Share on other sites

I look forward to the results of this test. It would seem surprising to me that calculating all shadow volumes dynamically would be faster than pre-calculating them at the map compilation stage, especially if you have lots of lights and geometry, but then our graphics cards do have a lot more processing power these days than when Doom 3 came out (I first started playing Doom 3 on a Radeon 9800XT with eight pixel shaders. My latest acquisition is a R9 290 with 2560 unified shaders).

 

I thought I read somewhere that the way GPU manufacturers count "cores" is a bit different now and that the Radeon 9800's cores count for 4 or 6 cores in

the modern context (obviously less flexible because the architecture is divided between pixel and vertex shaders). So, 640 unified shaders compared to the

Radeon 9800 if my recollection is correct. Still not too shabby. Yeah, I honestly don't see how much benefit dmap shadows have since that will probably be

recalculated once any dynamic object intersects them. The only place I see them having a benefit is where "noDynamicInteractions" is flagged (Doom 3 monorail level).

Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Link to comment
Share on other sites

Yeah, I honestly don't see how much benefit dmap shadows have since that will probably be recalculated once any dynamic object intersects them.

 

My understanding is that each shadow volume is a separate piece of geometry. The engine would submit the pre-calculated shadow volume associated with static geometry, then submit the dynamically-generated volume for a moving object in a separate operation. The dynamic shadow volume doesn't invalidate (or even interact with) the pre-calculated one; they are just rendered independently into the stencil buffer with the stencil operation making the final decision whether a fragment is in shadow or not.

 

It is the fact that shadow volumes are separate geometry that makes me skeptical that creating them can be done dynamically "for free" on the GPU, unless this functionality is moved into a geometry shader (which, unlike a vertex shader, is allowed to generate new geometry). Otherwise, the shadow volumes will need to be calculated on the CPU (on every frame) based on the relative positions of the light and the shadow-casting triangles.

Link to comment
Share on other sites

My understanding is that each shadow volume is a separate piece of geometry. The engine would submit the pre-calculated shadow volume associated with static geometry, then submit the dynamically-generated volume for a moving object in a separate operation. The dynamic shadow volume doesn't invalidate (or even interact with) the pre-calculated one; they are just rendered independently into the stencil buffer with the stencil operation making the final decision whether a fragment is in shadow or not.

That's my understanding too.

 

It is the fact that shadow volumes are separate geometry that makes me skeptical that creating them can be done dynamically "for free" on the GPU, unless this functionality is moved into a geometry shader (which, unlike a vertex shader, is allowed to generate new geometry). Otherwise, the shadow volumes will need to be calculated on the CPU (on every frame) based on the relative positions of the light and the shadow-casting triangles.

That's not what happens. Most shadow volumes are already computed on the GPU. To get around the lack of a geometry shader that can create vertices, the engine puts two copies of the verts of shadowcasting meshes into the vbo. For each original vert there's one vert with in its w-component set to '0' and one with the w-component set to '1'. The shadow.vp program above projects the '0' vert out to infinity and the edge between it and its twin (that doesn't get moved) is what makes up one face of the the shadow volume.

 

What does happen on the cpu right now is sillhouette detection. All the verts necessary to costruct the shadow volume are kept permanently in the vbo, but new sets of indexes can be constructed to identify a particular subset of those verts as the sillhouette. At least, that's what I think is happening, but I didn't get as far as stepping through the sillhouette code yesterday.

Link to comment
Share on other sites

That's not what happens. Most shadow volumes are already computed on the GPU. To get around the lack of a geometry shader that can create vertices, the engine puts two copies of the verts of shadowcasting meshes into the vbo. For each original vert there's one vert with in its w-component set to '0' and one with the w-component set to '1'. The shadow.vp program above projects the '0' vert out to infinity and the edge between it and its twin (that doesn't get moved) is what makes up one face of the the shadow volume.

 

Ingenious; I would never have thought of doing that. The vertex shader can't create lines, so the engine gives it degenerate "lines" consisting of cloned vertices and tells the shader to move the vertices into position.

 

So I guess the benefit of pre-calculated shadow volumes depends on whether this GPU operation can be skipped entirely for the static volumes, or if it needs to be performed regardless.

Link to comment
Share on other sites

Ingenious; I would never have thought of doing that. The vertex shader can't create lines, so the engine gives it degenerate "lines" consisting of cloned vertices and tells the shader to move the vertices into position.

 

So I guess the benefit of pre-calculated shadow volumes depends on whether this GPU operation can be skipped entirely for the static volumes, or if it needs to be performed regardless.

Exactly. And the GPU operation in shadow.vp is indeed skipped for dmapped shadow volumes, but I'm thinking that that doesn't matter because it's a trivial operation, vanishingly insignificant compared with the effort of rasterizing the shadow volumes into the stencil buffer. The bit that won't be trivial on the CPU is sillhouette detection and the constant loading of sillhouette indexes into the vbo. But if we can move that to shadow.vp too... tris whose verts don't get moved and that leave a degenerate edge won't get rasterized will they?

 

Looking at this code, I'm wondering whether the problem with moving lights eating performance is down to a moved light recreating all its interactions, including replacing the verts in the vbo, shadow and solid alike, which don't need to be changed for turbo shadows... maybe moving lights weren't a factor in the original design.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recent Status Updates

    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 1 reply
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
    • OrbWeaver

      I like the new frob highlight but it would nice if it was less "flickery" while moving over objects (especially barred metal doors).
      · 4 replies
    • nbohr1more

      Please vote in the 15th Anniversary Contest Theme Poll
       
      · 0 replies
×
×
  • Create New...