[Resolved in TDM 2.06] Soft shadows

[MirceaKitsune]

I know this might have been discussed before, so I'm sorry if I'm stating something obvious. I absolutely love the dynamic light system in idTech4... but I believe one problem which stands out at this stage is the lack of soft shadows. In real life, shadows become more blurred the farther they are from the object casting them. Gradual distance blur is probably a lot to ask for in a renderer... but shadows having a constant softness would likely look much better than the sharp ones used currently.

 

idTech5 has this fixed, but in Doom 3's day this was sadly the only possible lighting technique. Since the software as well as GPU's have evolved nowadays, I was wondering if the DarkMod engine could ever move past this limitation, and change the lighting system so it can allow for soft shadows to exist. This isn't to beg developers to implement it... I'm just curious if it's planned, and when we could expect it.

 

Hint: I play Xonotic a lot, and I know the Darkplaces engine (based on idTech1) implemented blurred shadows for its dynamic lighting system. Maybe some of its code could be of use here as well?

[SteveL]

Yes, we're actively experimenting with the options for this problem. We did try blurred shadows early in the 2.03 dev cycle thanks to a lot of work by Obsttorte with Sikkmod's implementation. We didn't get it working well enough to include it in the coming release, but we might return to it later.

 

Blurred shadows look better than hard shadows but not as good as projected shadows, where the umbra varies in size correctly with the distance from the caster and the radius of the light. Accurate shadows from area lights aren't needed however, which is lucky because no current technology can produce them in real-time. We want a good enough approximation to fool the eye, preferably better than blurred shadows but we'd take that over hard shadows if we can't get a projected solution working.

 

We use shadow volumes right now, and there are a few reasons why we should stick with those rather than try shadow mapping, the main one being that most of TDM's lights are omni lights not directional or spot lights, and a shadow map for an omni light costs 6 times as much as for the other kinds of light. The second reason is that shadow maps are getting comparatively more expensive with recent GPU hardware trends. Computing stuff is getting cheaper than storing and retrieving stuff from RAM.

[motorsep]

idTech5 has this fixed, but in Doom 3's day this was sadly the only possible lighting technique. Since the software as well as GPU's have evolved nowadays, I was wondering if the DarkMod engine could ever move past this limitation, and change the lighting system so it can allow for soft shadows to exist. This isn't to beg developers to implement it... I'm just curious if it's planned, and when we could expect it.

 

 

Hint: I play Xonotic a lot, and I know the Darkplaces engine (based on idTech1) implemented blurred shadows for its dynamic lighting system. Maybe some of its code could be of use here as well?

 

id Tech 5 is basically more advanced id Tech 3 - nothing is dynamic, everything is baked. So there is no real-time soft shadows there.

 

Darkplaces engine has quite a different architecture (and no cascaded shadow maps for parallel lights) and using its code would be problematic in id Tech 4.

[motorsep]

a shadow map for an omni light costs 6 times as much as for the other kinds of light.

 

It doesn't, if done correctly. Almost not a single modern game uses cubemap with 7 FBO bindings for omni lights. It uses atlased cubemap, which only takes 2 FBO bindings.

 

We used that old shadow mapping technique in Storm Engine 2 and it sucks. It's still better than shadow volumes in a sense that shadowing is decoupled from geometry's polycount, but it's much slower than unrolled cubemaps for omni lights

[MirceaKitsune]

 

id Tech 5 is basically more advanced id Tech 3 - nothing is dynamic, everything is baked. So there is no real-time soft shadows there.

 

Darkplaces engine has quite a different architecture (and no cascaded shadow maps for parallel lights) and using its code would be problematic in id Tech 4.

That's... unexpected. Why would idTech go back to baked lighting in the era of dynamic lighting, especially after one of their engines already implemented it? This probably means I misread something.

[motorsep]

You are reading me right - id Tech 5 (at least RAGE iteration, the original undiluted work of idSofware) is a step backward. However, Megatexures v2.0 are quite nice - you get almost no repetitive textures.

 

id Tech 5 seen is Evil Within is quite different engine, with a lot of middleware shoved into it. That one has dynamic lighting, but it's not unified light and shadow model seen in id Tech 4. There isn't anything revolutionary in id Tech 5.

 

Check out Tesseract engine:

 

 

 

Perhaps its real-time GI and shadows can be used in TDM.

[MirceaKitsune]

Yes, we're actively experimenting with the options for this problem. We did try blurred shadows early in the 2.03 dev cycle thanks to a lot of work by Obsttorte with Sikkmod's implementation. We didn't get it working well enough to include it in the coming release, but we might return to it later.

 

Blurred shadows look better than hard shadows but not as good as projected shadows, where the umbra varies in size correctly with the distance from the caster and the radius of the light. Accurate shadows from area lights aren't needed however, which is lucky because no current technology can produce them in real-time. We want a good enough approximation to fool the eye, preferably better than blurred shadows but we'd take that over hard shadows if we can't get a projected solution working.

 

We use shadow volumes right now, and there are a few reasons why we should stick with those rather than try shadow mapping, the main one being that most of TDM's lights are omni lights not directional or spot lights, and a shadow map for an omni light costs 6 times as much as for the other kinds of light. The second reason is that shadow maps are getting comparatively more expensive with recent GPU hardware trends. Computing stuff is getting cheaper than storing and retrieving stuff from RAM.

That's awesome news! Hope this will make it in 2.04 at least. I can compile the engine from SVN, so hopefully I'll be able to help with testing this when it's time.

 

Sadly I'm not in detail familiar with the differences between shadow maps and shadow volumes, despite doing some game development and 3D animation. In either case, I find the shape of the existing shadows accurate enough for me. Changing it would likely break existing missions too, since different areas would be lit and this would affect the player's lightgem. I also support sticking with the fastest technique, since TDM doesn't always run at 60 FPS even on a modern GPU (Radeon 6870 in my case). If the existing shadows can be softened, that sounds excellent to me.

 

It would be nice if shadow softness could still depend on some plausible factors otherwise. Since gradual softening based on distance is likely impossible, perhaps the radius or strength of the light will do; The further a light shines and the lower its intensity, the softer its shadow. Perhaps there could be some client settings to configure the start / end radius and the softness in between as well.

[lowenz]

However, Megatexures v2.0 are quite nice - you get almost no repetitive textures.

.....and if you want to modify one pixel of these non-repetive textures you need a monster machine

Edited January 2, 2015 by lowenz

[MirceaKitsune]

.....and if you want to modify one pixel of these non-repetive textures you need a monster machine

That's one problem... followed by inflexibility when it comes to editing a map. I think the classic texturing system is overall better: Have a bunch of textures, use the one you need where you need it.

 

Sure, it implies being a bit repetitive... but will anyone really notice? One way to get around this would be adding procedurally generated elements on top of the texture, or blending multiple textures using a random and smart pattern. So with each loop there's a little something different, and you see different cracks and details on every surface using the same texture. This would imply an intelligent shader and using a random seed, but it could be an alternative.

[motorsep]

You don't need a monster machine to author megatextures. The issue is with download size of the updates, since the entire megatexture needs to be updated.

 

The point of megatexture not only to have unique, non-repetitive look, but also to free designer/artist from technical aspects of it - simply paint away on the level in real-time, worry not about some quirky material setup and whatnot.

 

This technology is really good for streaming, when all the updates are done on the server and end-user doesn't have to worry about downloading 10Gb updates. But, we'll see what future holds, when id Tech 6 comes out

[lowenz]

xdelta (diff) patching isn't an option?

[motorsep]

xdelta (diff) patching isn't an option?

Not for a single image. I would work for .pk4 or .resources or .exe, but I don't think it works for images.

[SeriousToni]

Is it possible to implement some kind of image filter for hard borders like they appear at the end of a Shadow? You know - just to blur things that have a hard borderline via post processing effects? ..could make the whole view blurry, if I think about it now..

[demagogue]

There's a good system already available, it's just a matter of working out the few kinks it has.

[SteveL]

It doesn't, if done correctly. Almost not a single modern game uses cubemap with 7 FBO bindings for omni lights. It uses atlased cubemap, which only takes 2 FBO bindings.

 

We used that old shadow mapping technique in Storm Engine 2 and it sucks. It's still better than shadow volumes in a sense that shadowing is decoupled from geometry's polycount, but it's much slower than unrolled cubemaps for omni lights

I was thinking about the six renders you have to take to cover every direction seen from an omnilight's point of view. I did a bit of googling and see that some systems have managed to do it with fewer "curved" renders but they always seem to come back to 6 renders as being the most efficient solution. You only need one render to plot the entire scene that can be hit by a spot or directional light, seen from the light's point of view, but you'll always need more for an omni light, probably 6.

 

Is it possible to implement some kind of image filter for hard borders like they appear at the end of a Shadow? You know - just to blur things that have a hard borderline via post processing effects? ..could make the whole view blurry, if I think about it now..

Yeah that would make for a lot of blurry lines

 

We haven't talked much yet about how we could do blurred shadows better because we want to try adapting our existing volume shadows to project a penumbra first, but one way you could do it is to render the existing stencilled shadows to a texture instead of using them to make a stencil, blur the texture, and use that as an alpha mask instead of using the shadow stencils when drawing your light effects on surfaces. Similar to what Sikkmod did, except that his solution had to render the entire scene twice and blur between the two images, which softened *all* the details in the scene, caused artefacts, and halved FPS. That was all because the engine was closed source when he was working on it. We have more options now.

[MirceaKitsune]

Anyone know if it's possible to also make a shadow's blur depend on the light source casting it? My idea was to have the intensity of the blur depend on the radius of the light versus its intensity... so stronger and lower radius lights have little shadow blur, but distant and low intensity lights have much softer shadows.

[motorsep]

Anyone know if it's possible to also make a shadow's blur depend on the light source casting it? My idea was to have the intensity of the blur depend on the radius of the light versus its intensity... so stronger and lower radius lights have little shadow blur, but distant and low intensity lights have much softer shadows.

 

Check Unreal Engine 4 - it's only $20 and it does everything you want and more

[SeriousToni]

Someone do a convert of TDM into the Cry Engine Unreal Engine nao!!!!11

 

That's an old chestnut

[MirceaKitsune]

I use Linux and prefer sticking to open-source software exclusively. Epic Games aren't as awesome as ID Software, and never outsource their engines after a number of years

[motorsep]

Epic is extremely awesome now. Not only UE4 runs on Linux, the tools run too. And you get access to the full source code (although most likely you aren't gonna touch it).

 

My point is either use TDM / idTech 4 as is, or use something else that is designed to run with the features you want.

[Diego]

My brief experience with UDK and Cryengine made me hate both forever.

[HMart]

Epic is extremely awesome now. Not only UE4 runs on Linux, the tools run too. And you get access to the full source code (although most likely you aren't gonna touch it).

 

My point is either use TDM / idTech 4 as is, or use something else that is designed to run with the features you want.

why are you using idtech 4 (BFG) for your game if you think UE4 is so awesome? If you think BFG is not the engine for you then change, more and more i'm seeing you praising UE4 (grass is always greener on the other side effect?) but you still didn't made the jump.

[SeriousToni]

why are you using idtech 4 (BFG) for your game if you think UE4 is so awesome? If you think BFG is not the engine for you then change, more and more i'm seeing you praising UE4 (grass is always greener on the other side effect?) but you still didn't made the jump.

I guess that working in this field you can't simply focus on one engine. They all have their benefits but you can't have everything. So working with different engines depending on what you need is the thing to do I believe. I understand that debating about the Unreal Engine here on these forums isn't that applicable, since it is all about the idTech Engine here though.

[motorsep]

why are you using idtech 4 (BFG) for your game if you think UE4 is so awesome? If you think BFG is not the engine for you then change, more and more i'm seeing you praising UE4 (grass is always greener on the other side effect?) but you still didn't made the jump.

 

I sank 2 years of R&D and several thousand dollars into idTech 4 / BFG development. Plus I was already familiar with idTech engines, so I only needed to tweak my art pipeline to adopt idTech 4 / BFG. Plus we have a good chunk of gameplay code written. That's why I am using it. Once I am done and released, there is a good chance I move on away from open source engines (unless some miracle happens).

 

If I knew Unity / UDK a bit, I'd probably never looked at idTech 4 / BFG (and that would be most likely a better path) and I'd go with either Unity or UE4.

 

Thinking that old id Tech 4 can ever render like UE4 does is naive, at minimum. It seems that AMD purely hates idSoftware as neither of their engines, starting with idTech 4 work good on AMD. For example our engine runs at 60 fps on Nvidia, and 20 fps on equal AMD GPU (and otherwise slightly more powerful PC than the one that runs Nvidia GPU). BFG, RAGE, Wolf TNO, Evil Within - AMD doesn't run them well, even powerful models.

[lowenz]

I think it's not hate, it's.....Rage effect (CUDA texture decoder)

Edited January 4, 2015 by lowenz