TDM Engine Development Page

[Springheel 4469]

Most of the lights there are noshadows already. The few remaining lights are relatively small and don't intersect with much of the foliage.

 

 

That was one example picked at random. Are you confident about all other places foliage textures are used in existing maps?

 

Let's use some specific numbers, because I could be totally wrong about how this works. The hedge01_square_long.lwo model has a 96 tris shadow mesh, but 5128 tris of an alphamap foliage texture set to noshadows. Let's imagine a short path with these hedge models lining each side of the path-say 10 in total, hit by two lights. Currently that's maybe 2000 shadow-casting tris at most, which isn't an issue. Are you saying that if that foliage texture was suddenly forced to be shadow-casting, the 100,000 extra tris wouldn't have more than a 5% effect on the drawcalls of that scene?

[duzenko 588]

 

That was one example picked at random. Are you confident about all other places foliage textures are used in existing maps?

 

Let's use some specific numbers, because I could be totally wrong about how this works. The hedge01_square_long.lwo model has a 96 tris shadow mesh, but 5128 tris of an alphamap foliage texture set to noshadows. Let's imagine a short path with these hedge models lining each side of the path-say 10 in total, hit by two lights. Currently that's maybe 2000 shadow-casting tris at most, which isn't an issue. Are you saying that if that foliage texture was suddenly forced to be shadow-casting, the 100,000 extra tris wouldn't have more than a 5% effect on the drawcalls of that scene?

Any model with separate shadow geometry must stay as it is - i.e. keep the noshadow flag on the main mesh material.

No argue there.

I'm absolutely 'pro' all and every speed trick there is - this one especially.

 

OK, let's just imagine we did just that. We'd still have 100K tri's in depth render, 100K for ambient light, 200K for point lights - plus the added 200K. Even on a system clearly limited by tri numbers it's only 200/600 = 33%. Now the big question is, are we actually having tri-limited scenes in TDM? Because if we do, that's already a big problem of its own. A mapper can target 60fps on his GTX 1070 but the average Joe's Radeon 7670 will barely squeeze 15 fps. We have lots of graphics options in TDM that can allow smooth gameplay on wide range of hardware but there is no option to alleviate the tri-count bottlenecks.

 

That means that 3d scenes can never get even close to limiting FPS by tri count. That's going to be causing massive problems with or without shadows on transparent materials. Putting very-high-poly models in maps is the best way to make the game unplayable for a good half of our users. Forcing alpha shadows is just making the margin to be a bit higher e.g. 50% vs 60%. The right way is to make the low-poly LOD versions for models like that.

 

But again, I don't suggest to lose the shadow meshes, or to force shadows on all perforated materials.

[peter_spy 1389]

AFAIK, you really have to try hard to make your FPS go down because of excessive geometry. It scales with hardware very well. Last time I tried, I needed like 4 million tris per scene to make the FPS go down. Drawcalls or shadow tris were always first to bring FPS down.

[Springheel 4469]

shadow tris were always first to bring FPS down.

 

 

Yes, adding new shadow tris can't be done casually...that's why I'm confused about how 100,000 new shadow tris to a scene (in the example above) wouldn't be a performance issue.

[duzenko 588]

 

Yes, adding new shadow tris can't be done casually...that's why I'm confused about how 100,000 new shadow tris to a scene (in the example above) wouldn't be a performance issue.

Because it's not stencil shadows

[peter_spy 1389]

With shadow maps, complex shadow scenes will now rely on GPU memory. Not sure how we'll measure these things, I'll have to make some new test maps to see how it works on my system now.

[HMart 270]

I'm also of the opinion shadow meshes should still be included, they are a performance trick even for shadow maps and yes, not all alpha materials should cast shadows but that is obvious. The ability to control shadow mapping on/of per material and or entity should also be a most.

 

Btw i'm almost afraid to say it, because this is really a radical suggestion... but what would solve many of the problems i'm seeing coming up is...removing stencil shadows all together.

 

Wait! Don't come with the pitchforks right away! Hear me out ;P ...By doing that, you can then remove noshadow from the materials with no fear of breaking what is not there, and imo logically, removing shadows or not from a object, should be in the hands of the mission maker, not TDM forcing everyone that uses the oficial materials to not have shadows (apart from the obvious ones that should never cast shadows), this way there's no need for a extra keyword either. Anyone wanting to control the shadows, for existing materials per mission, just use a material skin override with noshadow on it, like always, missions where shadow anomalies are detected, because someone faked shadows, should be updated by the mission makers or by a third party if the author is nowhere to be found. Yes this is a radical change indeed but would save many headaches and pretty much all engines i know, use one or the other, supporting two radically different shadow systems in the same game is not easy.

 

If this is still not the time to think of that then, i may suggest not removing noshadows, let that keyword control only stencil shadows, make another to control shadow maps, that way you don't mess with stencil, for shadow maps, editing the missions at least to remove faked shadows wouldn't be necessary (nor do you want that if playing with stencil shadows), why, because you would only need to stop those particular materials from casting shadow maps and this could be done like i already said, by implementing that new "noshadowmap" keyword on a skin and then just make a script that at map/mission start time would detect the materials you want and swap them.

 

I can't think of anything better unfortunately.

 

Because it's not stencil shadows

 

Yes shadow maps are just textures they do not create extra geometry like stencil does.

[peter_spy 1389]

I think Duzenko already mentioned that stencil shadows are a dead end in terms of development, so you can expect that at some point, r_shadows 2 will be default / mandatory setting.

[duzenko 588]

removing stencil shadows all together.

Is not on the table ATM, for a number of reasons

 

I think Duzenko already mentioned that stencil shadows are a dead end in terms of development, so you can expect that at some point, r_shadows 2 will be default / mandatory setting.

It's all about what the current hardware offers. In 2004 stencil made a lot of sense on that time GPU's with relatively limited shaders and powerful ROPs.

Since then, nVidia has been investing a lot of effort in adding more stuff in their hardware that would allow more and more benefits for shadow mapping. I don't know if it was the best course but they chose it and here we are now - with lots of hardware optimization and algorithm RnD leveraging each other.

Maybe in a few years the RTX technology will make shadow maps obsolete again but until then we'd be an odd duck trying to cling on to stencil. There have been fundamental issues in both techniques but for shadow maps they're largely resolved while stencil mostly remains where it was 15 years ago.

Throw the constantly increasing abuse of stencil shadows by mappers on top of that.

It's just too aged now.

[nbohr1more 1924]

It seems that PCSS approaches a proper ray-traced solution as long as you have "infinite shadow map resolution"

so the newest shadow solutions like HFTS and RTX ray tracing are sold as "solutions" to this last resolution hurdle.

 

I'd still like to see this horse race:

 

1) Stencil Shadows with PCSS blurring

2) Penumbra Wedges

3) HFTS Shadows

4) RTX raytraced shadows

 

I'll bet the result would be 1 as the clear winner as long as GPU skinning is also done.

HFTS and RTX waste more resources than Stencil Shadows ever dreamed of to achieve that look.

 

Though, ideally, we'd just have better shadow map resolution management that isn't too expensive

for real-time moving lights. I'm not sure how CSM or Parallel Split Shadow Maps, etc perform in dynamic

scenes.

 

 

Also, I like the idea of a new noStencilShadows flag. It's clear that there are cases where enabling

shadows will only be possible in shadow map mode.

[nbohr1more 1924]

Whoa!

 

I went back to looking at Trapezoid Shadow maps and what recent work has been done in similar

areas of research.

 

Look at this stuff!

 

https://marciocerqueira.github.io/#2

[nbohr1more 1924]

Robert Beckebans is looking into getting his GL conventions to use Direct State Access:

 

https://github.com/Fendroidus/Guide-to-Modern-OpenGL-Functions

 

I think we might've made some of these changes already but this might come in handy for further modernization efforts.

[duzenko 588]

Whoa!

 

I went back to looking at Trapezoid Shadow maps and what recent work has been done in similar

areas of research.

 

Look at this stuff!

 

https://marciocerqueira.github.io/#2

Much of that is too complicated for my feeble mind.

However the revectorization PDF got me thinking in a particularly exciting direction.

My first attempt at shadow AA, unsurprisingly, ended with a puff, however the second time came out better - see the attached

This is obviously very much WIP, but the next version should add enough quality to get rid of shadow edge corners almost completely

You can try it with current svn trunk - multi light shader

[nbohr1more 1924]

I just read through the slides and the papers to get a fuller picture.

 

Initially, I thought this was some sort of space convolution trick akin to Trapezoid Shadow maps

since it was one of the first things that came up for a search for that.

 

I now see what this really is...

 

The algorithm treats all the silhouette pixels like graph plot points then reconstructs

the silhouette by plotting a curve through the points. It kinda reminds me of post process

AA like MLAA or FXAA... or (more fittingly) like a pixel art up-scaling algorithm:

 

https://en.wikipedia.org/wiki/Pixel-art_scaling_algorithms

 

You can see in a slide where the shadows from a thin tree branch produces scattered pixel

dots with breaks and discontinuities that the algorithm turns those scattered pixels into

curved blobs rather than restoring the original branch silhouette. Thus this algorithm

requires a certain minimum amount of shadow map resolution to be effective.

[duzenko 588]

I just read through the slides and the papers to get a fuller picture.

 

Initially, I thought this was some sort of space convolution trick akin to Trapezoid Shadow maps

since it was one of the first things that came up for a search for that.

 

I now see what this really is...

 

The algorithm treats all the silhouette pixels like graph plot points then reconstructs

the silhouette by plotting a curve through the points. It kinda reminds me of post process

AA like MLAA or FXAA... or (more fittingly) like a pixel art up-scaling algorithm:

 

https://en.wikipedia.org/wiki/Pixel-art_scaling_algorithms

 

You can see in a slide where the shadows from a thin tree branch produces scattered pixel

dots with breaks and discontinuities that the algorithm turns those scattered pixels into

curved blobs rather than restoring the original branch silhouette. Thus this algorithm

requires a certain minimum amount of shadow map resolution to be effective.

The way I understand it we'd need a pre-processing pass. Render the player's view into a temporary texture that stores the shadow occlusion results.

Then we use that to traverse the shadow edge in the shadowmap texture space.

My approach is going to be somewhat different to what they did because I want to leverage textureGather.

The biggest problem is how to store the shadow occlusion for all lights to use in the multi-light shader.

I'm only doing this in the multi-light shader path so as not to pollute the single-light path with 'construction garbage'

 

On the other hand I could combine that and what I have been planning for a while now - shadows for all lights in one pass.

 

The good ol' question is, do we want to wait until 2.07 branch.

I already have a simple nearest-neighbors AA filter in the multi-light shader while the above easily could take weeks of spare time coding.

@stgatilov any ideas on the 2.07 branch date?

 

Then, this is probably going to be mutually exclusive to SS and if so how do we present it to the audience?

[lowenz 521]

Robert Beckebans is looking into getting his GL conventions to use Direct State Access:

 

https://github.com/Fendroidus/Guide-to-Modern-OpenGL-Functions

 

I think we might've made some of these changes already but this might come in handy for further modernization efforts.

Some results with the lastest version available of RBDOOM3, 64 bit and self compiled with MSVC 2017 CE!

SMAA, no filmic filter, no autoexposure, reduced lens flare:

 

*750 Ti

 

*1050 Ti

Edited April 26, 2019 by lowenz

[lowenz 521]

Little discovery about half-lambert lighting implementation in RBDOOM3.....there're some issues with soft shadows

 

NO HLL:

HLL (default behaviour):

Edited April 28, 2019 by lowenz

[HMart 270]

Personally would love to use RBDOOM3 because of the nice features it has and also being a more modern idtech4 base but because the engine fullscreen gui's are based on Adobe flash and uses the Adobe deprecated/defunct actionscript 2.0, I just can't use it, specially when I know almost nothing about using Adobe flash and there's no tutorials on how to make flash guis to BFG.

 

Btw lowenz are you messing with the BFG code to see what could be transferred into TDM engine or are you thinking of making a game or mod with RBDOOM3?

[lowenz 521]

Simply testing the lighting/shadowing system and the performance hit

[peter_spy 1389]

Icecoldduke created a PBR interaction shader for Doom3, Quake4 and Prey version of idtech4: https://www.doomworld.com/forum/topic/108506-doom-3quake-4prey-pbr-interaction-shader/

 

It's using current Doom 3 and Prey textures, so it looks wrong, but with proper PBR materials this could look physically correct. And, IIUC, this can be used as a vfp program, so technically it's a supplementary feature, you can gradually switch to PBR shaders, and not rework all the textures and materials.

 

[New Horizon 478]

I had no idea what PBR was and had to look it up. lol  Physical based rendering?  Looks interesting.

[peter_spy 1389]

https://academy.substance3d.com/courses/the-pbr-guide-part-1

[peter_spy 1389]

The same user works on RTX-computed GI solution and baked shadowmaps for idtech4: https://www.doomworld.com/forum/topic/104202-megalight-wip-megatexture-lighting-tool/

[VanishedOne 527]

Since people on Doomworld have been using Icecoldduke's PBR shader with the games' existing non-PBR materials, I thought I'd try dropping it (the Doom 3 version) into TDM (2.05) as a replacement interaction_direct.vfp. The result is glitchy (the sparkle effect on the banner below is the least of it), and it's presumably calibrated for D3 levels of shininess, but it has striking effects on shiny surfaces (on the right is TDM's enhanced ARB interaction shader for comparison):

 

[peter_spy 1389]

Did you redo textures, so they're correct for PBR? If not, then don't expect major breakthroughs. Both TDM textures diffuse color/intensity range and speculars (if any) are incorrect for roughness/metallic workflow. Roughness map is an "inverted greyscale specular" in terms of scale, and it needs particular values to work correctly (all materials have constant values or value ranges, depending on their smoothness/porosity). AFAIK, with the current implementation you need to put the roughness map into the red channel, and metallic map into the green channel of the "specular" texture for the shader to work correctly.

Guidelines for correct textures are in the PBR guide pt. 2 (pt. 1 is linked above).