TDM Engine Development Page

[Tels]

Don't believe me for a second, but there ya go, from Carmack himself: http://fabiensanglar...adowVolumes.txt and he stated that shadow volumes are slower, costly and complex, compare to shadow maps.

 

Note, that was a year 2000. We are getting into 2015 pretty soon, and quality issues with shadow mapping have been resolved on a certain level.

 

That's all nice and dandy, but who is going to rewrite the TDM engine? I don't think we have the resources in one way or another.

Edited October 30, 2014 by Tels

[OrbWeaver]

How do shadow maps work with omnidirectional lights? All the descriptions I have seen assume projected lights with a principal direction and limited "field of view". Do you just treat an omnidirectional light as six spotlights shining in each of the cubemap directions?

[revelator]

Hmm should be some info around the net but as far as i could understand it it uses the zbuffer (depthbuffer) by copying everything into a depthimage,

and then comparing every point copied,so that if you look out from a lightsource then everything is in the light and any object in the path of that light will be in shadow.

 

Its not as precise as shadow volumes but its a good deal faster

[SteveL]

Yes shadow maps need cubemaps for omni lights. And we use mostly omni lights.

Like I said above, I think shadow maps are impossible for us because we already have loads of content that can't be changed to allow the hacks and workarounds that they need, even if we had sufficient coding resource.

[Arcturus]

The only possible solution for shadow volumes is to have low poly mesh encased into render mesh, and have that to be shadow material, and render mesh materials have noShadows flag.

This is what we do.

[New Horizon]

Don't believe me for a second, but there ya go, from Carmack himself: http://fabiensanglar...adowVolumes.txt and he stated that shadow volumes are slower, costly and complex, compare to shadow maps.

 

The issue for us is that overhauling TDM to such a degree at this point would be suicide. Everything from the missions to the way the lightgem itself works would have to be fixed from the ground up. What we have works very well and continuing to improve what we have is the sane choice.

[nbohr1more]

Yeah, Penumbra Wedges seem to be a more reasonable option. Of course, for the sake of performance

you could make the wedge have a small fixed distance from the silhouette rather than do the whole distance

based fading thing... or have the two modes performance vs true penumbra.

 

I read an old Beyond3D discussion about Penumbra Wedges and it seems that the coders there were a little

scared to touch them because they thought they were a "patent minefield". Though I don't know why they

would be worried there when some Shadow Map techniques could also be patented.

[motorsep]

That's all nice and dandy, but who is going to rewrite the TDM engine? I don't think we have the resources in one way or another.

 

I am not proposing rewriting anything. Just noting that as is shadow volumes are performance hungry. Adding umbra on top of that will reduce performance even further. I was under impression TDM engine needs performance boost, not performance decrease.

Edited October 30, 2014 by motorsep

[motorsep]

Yes shadow maps need cubemaps for omni lights. And we use mostly omni lights.

Like I said above, I think shadow maps are impossible for us because we already have loads of content that can't be changed to allow the hacks and workarounds that they need, even if we had sufficient coding resource.

 

I am failing to understand how do you need to modify content for shadow maps ? Have you tried RBDoom 3 BFG ? No content needs modification. That's the first time I hear such thing. We added shadow maps to our engine and nothing had to be modified as far as content.

 

You could retarget material for shadow mesh via skin to "nodraw" and comment out noShadows key in the render mesh material. Or leave everything as is - shadow mesh would still cast shadow. Just with increased performance.

Edited October 30, 2014 by motorsep

[nbohr1more]

Treb's a good coder, and I'm sure he solved a great deal of the shadow acne issues with the conversion from stencil volumes to shadow maps.

He just doesn't seem to be interested in releasing the work he did on Vanilla Doom 3 to the public.

 

So there we have it.

 

Dead horse beaten beyond a bloody pulpy puddle

Edited October 30, 2014 by nbohr1more

[OrbWeaver]

I read an old Beyond3D discussion about Penumbra Wedges and it seems that the coders there were a little

scared to touch them because they thought they were a "patent minefield". Though I don't know why they

would be worried there when some Shadow Map techniques could also be patented.

 

I guess these coders must be pretty excited then that the patent on mip-maps and trilinear filtering will expire in 2016, at which point they are free to use this amazing cutting-edge "invention" in their project.

 

Pretty much everything under the sun is patented. If you're worried about patents you might as well give up writing software altogether.

[motorsep]

Treb's a good coder, and I'm sure he solved a great deal of the shadow acne issues with the conversion from stencil volumes to shadow maps.

He just doesn't seem to be interested in releasing the work he did on Vanilla Doom 3 to the public.

 

I don't think he did. He made that claim a long time ago. And yet he never released any screenshots, or videos. He did so for xReal, and for BFG, but never for idTech 4. That makes me wonder if such project ever existed. There is absolutely no logical reason to hide it.

[nbohr1more]

That paper was made by the guy that makes and markets the C4 Engine, a fantastic coder, he even invented alone a better replacement for the collada mesh files, called OpenGEX , about the shadows, on his new engine, version 4.0, he removed Stencil Shadows for shadow maps.

 

Stencil shadows were faster in the old days, but nowadays shadow maps aren't that much slower, plus shadow maps have so much benefits above stencil shadows that the small hit on performance is worth it. About the benefits, shadow maps don't need closed models, Stencil do, shadow maps can be made softer easily, stencil don't, shadow maps support alpha mapped surfaces easily, stencil don't, shadow maps can simulate near contact shadows (shadows that become softer and blurred by distance from the caster), stencil don't.

 

He removed Penumbra Volumes? His March 2014 GDC still speaks above shadow volume optimization:

 

http://www.terathon....c14_lengyel.pdf

 

(boy, you've gotta be a good mathematician to dive into that...)

 

Edit:

 

Some more goodies under the conference slides:

 

http://www.terathon.com/lengyel/

Edited October 30, 2014 by nbohr1more

[HMart]

He removed Penumbra Volumes? His March 2014 GDC still speaks above shadow volume optimization:

 

http://www.terathon....c14_lengyel.pdf

 

(boy, you've gotta be a good mathematician to dive into that...)

 

Yes he did if you read the Release Notes for the engine version 4.0 you will see that mentioned. Yes, he is a fantastic coder and also a very good mathematician, he even wrote a book about that subject.

 

http://www.amazon.com/Mathematics-Programming-Computer-Graphics-Edition/dp/1435458869

[motorsep]

The real question is why C4 engine isn't widely used :/

[161803398874989]

(boy, you've gotta be a good mathematician to dive into that...)

...hi. I'm only a first-year master's student, so not a great mathematician by any means, but I went through the slides and found them pretty understandable. The most relevant slides are the ones with title "Geometric Computation in 4D" and the "Explicit Formulas". The notations are scattered about through the previous slides.

 

Calculation of shadow planes is done by taking the wedge product of the light origin (point) and line along the edge of the object (refer the second-to-last slide). This will give you the plane containing both the edge and the light origin. By calculating the intersection of this shadow plane with the floor (using antiwedge product of the two planes), you then have found the edge of the shadow.

I guess the real issue would be learnign to work with bi- and antivectors.

 

For me it's more I know very little about rendering and game engines (I can code though), and hence I didn't understand anything about the diffuse lighting example. If you have any question about the mathematics I'd be happy to help though.

[HMart]

The real question is why C4 engine isn't widely used :/

 

We are going off topic here this is not the C4 engine forum, but if you have to know his engine is not widely used because people are looking for shine graphics and not a very well coded and structured engine, his demos were always very simplistic and not very appealing, full of coder art, Eric also takes his time implementing features he still lacks HDR lighting on his engine and SSAO was also implemented only recently, the tools were also not very artist friendly.

Edited October 30, 2014 by HMart

[nbohr1more]

Ack, I see now that native doom 3 doesn't output anything but vertex coordinates from the vertex program. I guess sikk's color values are just so he can hook into the result later. It still should be possible to output the penumbra gradient as alpha but we'd need a place to put it. BFG and Morpheus have a shadow fragment program that allows you to modulate the result set. Will have to see if we can easily hook the legacy draw to an ARB output. Either that or pass the alpha render to the light interaction vfp... A few hacks floating around in my mind but nothing that sounds good enough.

[nbohr1more]

Heh, this might not be as hard as I thought. The whole branch for GPU vertex processing is in draw_arb2. We can plug vfp invokes right there.

[SteveL]

An update on soft particles (yes, I'm still working on them). I found something interesting: the vanilla engine already has a crude particle softener: the "modelDepthHack".

 

That's a setting available on all models but it's used only for particles. It moves the near clip plane forward when a particle is being drawn which brings the particle forward a bit, so it allows some overdraw over solid objects that intersect. Without it, the intersections are much uglier still.

 

Here's a lamp tdm_glare_lamp_01 drawn as it would be by the vanilla engine. It doesn't intersect with the wall too badly, although if you look closely there's a line visible across the bricks, just to the right of the lamp. And the overdraw pretty much hides the lantern itself:

 

Now here's what it looks like from the same angle with the model depth hack disabled:

 

The modelDepthHack fails when the player gets more side-on to the wall. This is what caused the badly clipped lamps in PD2 mentioned above by nbohr1more:

 

The particle softener I'm working on is ok from all angles, and it lets the lamp show through because it adjusts the apparent thickness of the particle according how much of it you should be able to see in front of any solid stuff:

 

That approach works better for smoke than light glares, but I think it's still an improvement for the glares. But you can see in the pics above that it dims the particle when there's a wall immediately behind. That effect works well for giving the illusion of smoke or steam, because if there's a wall in the middle of a smoke cloud, there's only half as much smoke to see in front of the wall as there would be in the space over the wall. But it's not so good for lamp glares which should bounce off the object behind and stay at full brightness.

 

We already need different shader programs for smoke (alpha blends), and lamp glares (additive blends), so we can fix the dimming problem by using a small variation on the smoke method of calculating the "thickness" of the particle: change the depth at which the particle is at full strength from particle_depth + particle_radius to particle_depth + 0. Not sure I'll get that done and tested before going on hols for a week tomorrow, but I'll have a go

[Bikerdude]

Ah, nice one mister.

 

As a mapper, this one has bugged me for ages.

[Filizitas]

Is there any Note on when we do get a new Update on the Game? I mena its actually a game now and not a Mod

[Tels]

Hopefully the new soft particles will be the default and we can get rid of the old, ugly ones. I guess even if you are strapped for performance, you don't want the ugly transitions, and would rather dial another setting down instead.

 

As for the smoke vs. halo, would it be possible to make this a parameter in the material files instead of hard-coding it? Or is this unnec.?

 

Good work, btw!

[SteveL]

I've been playing round with the depth hack this morning to see whether we can use it instead of my ignoreDepth keyword to get our particles to draw over the solids in front. The advantage of doing it that way would be we still get to use efficient hardware clipping to cut away unseen parts of the particle, instead of relying on the shader program. I haven't seen any measureable performance difference, but efficiencies are always good. And it's set in the particle declaration, which would give mappers an easy way to tweak the effect.

 

It won't work, but I'll share what I found out anyway.

 

Here's an example of depthHack in action. The three glares on the stairs are all at the same depth, in the middle of the back stair, but the left one has "depthhack 0" in the particle declaration, the middle one has "depthHack 16", and the right one has "depthhack 32". The stairs are 16 deep. DepthHack moves the near clip plane temporarily while drawing the particle so that it gets clipped as if it were however many units in front of where it actually is:

 

The effect is the same as if the engine had moved the particle forward that many units. But that's where the problem comes in. If the player gets closer to the particle than its depth hack value, it vanishes abruptly. No smooth fade-out. And the effect would be really jarring for big particles like dustfogs. They'd vanish while 5-6 feet away. The best depth hack value is about half the particle size, i.e. the radius of the volume that the particle is imitating.

 

There's no way round that for soft particles. Moving the clip plane or the particle itself to get your overdraw means you'll never see a particle from within its "volume". So we'll stick with using ignoreDepth for drawing particles, but we could still use the depthHack parameter to control the particle depth or "radius". It gives mappers an easy way to tweak the effect, unlike the plan I was testing last night which was to get the engine to set the radius of the particle according to the size of the the square images being drawn. And where mappers have already tweaked a depthHack value to fix a particle showing through a wall etc, it'll always be safe to use that value in the soft particle effect too.

 

Hopefully the new soft particles will be the default and we can get rid of the old, ugly ones. I guess even if you are strapped for performance, you don't want the ugly transitions, and would rather dial another setting down instead.

 

As for the smoke vs. halo, would it be possible to make this a parameter in the material files instead of hard-coding it? Or is this unnec.?

 

Good work, btw!

 

Thanks

 

It's kind of in the material files already. The material file chooses the fragment program to use, and it's the fragment program that chooses how the particle fades out. But there's the usual conflict between flexibility and ease of use. The more flexibility we put in material files, the more changes there are to make (we have 97 existing materials used in particles!) and the fewer existing maps will benefit (if they have custom materials). If we hard-code it, you lose some flexibility but there will be less for mappers to do, and existing maps won't need re-releasing to take advantage. If we could find the *perfect* solution, it would make sense to apply it universally in the engine without any changes needed to material files at all. Every map / particle would benefit and it could then be toggled by a cvar/menu option. You can't do that if you've replaced all your materials, at least not without applying another global setting with conditional stages in every material, like we do with the ambient light setting and frob highlighting.

 

There's already special case code in the renderer back end for drawing particles (the model depth hack), so we have options. We can do all the setup in the material files, or we can let the engine handle everything, or some mixture of the two. Ideally we'd not have to change material files, which will mean the engine applying the effect perhaps in response to a parameter in the particle file (like "depthHack" above). And even if we put it all in the engine we can still give full flexibility to future mappers by allowing parameters. The main thing we have to think about is how to fix it up best for existing maps without anything needing changing.

 

I think I know everything there is to know about how the engine draws particles now I'll spend my free time while travelling this week thinking up the ideal feature proposal. Suggestions welcome!

[VanishedOne]

Is there any Note on when we do get a new Update on the Game? I mena its actually a game now and not a Mod

 

Only this, I think: http://forums.thedarkmod.com/statuses/id/2393/