TDM Engine Development Page

[SteveL]

I know that getting results out of the depth buffer is old news by now but I hate leaving slightly shoddy work on show, so here's my final update on it

The "optimal" zFar value I posted above was an approximation, the best-fit constant that I could find to make TDM's quirky projection matrix fit the standard formula that everyone uses. It turns out that simply taking the numbers from TDM's projection matrix and reversing the operation gives a simpler formula, with precise results. It takes only 2 assembly operations to do it unlike the standard formula. I've checked the theory on paper and made sure that I hadn't missed any steps by using a test map with patches 100 units apart and a special shader that painted horizontal bands on them positioned according to how many doom units they were away from the eye.

 

It's: depth_in_doom_units = 6.0 / ( 2.0 * zbuffer_value - 1.9990 )

 

Those magic numbers come from the magic numbers in the engines' projection matrix, which is constant. We'll never change them so we can use them freely in shader programs. And to save one calculation, we might as well divide everything on the right by 6, so: depth = 1 / ( 0.333... * zbuffer - 0.331666... )

 

I'll paste my assembly version. This is wordy but there are only 4 operations in this code: 2 to get the depth value for a given screen coordinate, and 2 to convert it back to doom units.

 

!!ARBfp1.0
# == Fragment Program ==
#
# Convert the stored zbuffer depth back to original depth in doom units
# by reversing the transformation. The constants used are derived from TDM's
# constant projection matrix:
#	originalDepth = 1 / ( 0.33333333 * zbuffer - 0.33316667 )
#
# Inputs from Material File:
#	texture[0] = _currentDepth
#
# Inputs from the engine:
#	program.env[4].xy = reciprocal of _currentDepth texture size
#
PARAM   depth_consts = { 0.33333333, -0.33316667, 0.0, 0.0 };
PARAM   image_size_rcp = program.env[4];
TEMP	tmp, zbuffer, originalDepth;

# Map the fragment to a texcoord on our depth image, and sample
MUL   tmp.rg, fragment.position, image_size_rcp;
TEX   zbuffer, tmp, texture[0], 2D;

# Recover original depth in doom units
MAD   tmp, zbuffer, depth_consts.x, depth_consts.y;
RCP   originalDepth, tmp.r;

# Now do something with original depth
END

 

[Arcturus]

It's always been in heathazewithmask (used by water_stream in the test map).

I think the iridescence appeared after Obstorttet's changes to heatHazeWithMask.vfp in revision 13969. Here's comparison:

 

Before:

 

 

After:

 

[Springheel]

Hmm, there's a clear difference there...blues and reds appearing where there are none in the original texture?

[SteveL]

Ack I see. Sorry about contradicting you, but there are no comments in the code so I assumed we'd inherited that effect from doom3.

[AluminumHaste]

Those look like chromatic dispersion/aberration or a facsimile of it.

 

Like this:

 

http://en.wikipedia.org/wiki/Chromatic_aberration#mediaviewer/File:Chromatic_aberration_%28comparison%29.jpg

[revelator]

Woa yeah that looks a bit like it, not sure if its just a small bugger in my code that showed up there or if its the shader ?!?.

gDebugger is a great tool but when theres loads of calls it can freeze so set it to break on deprecated calls and use frame stepping from there.

It can be a bit hard to learn all its capabilities, but it sure is worth it

[HMart]

I'm of the same opinion as AluminumHaste, that does look like the moder of the shader tried to simulate chromatic aberration or caustics.

[SteveL]

It's nothing to do with the depth capture. The color aberrations are apparently from a new shader prog added to the repo in August. I didn't notice it had been changed. No worries, I'll retry the fix with the 2.02 version of the water shader.

[SteveL]

Good news, this works out better all round.

 

The 2.02 release shader (that doesn't have the extra colours) uses far fewer samples than the one I was adapting above, so I've put in the extra sample to stop ALL foreground distortion. The shader now chooses between the distorted colour, the mirrored distorted colour (last night's fix), and the original color (the original fix). Now you can even hold up bright antlers in the light in front of the water without getting any visible distortion. It's looking better round the water edges too.

 

I've committed a new test map with 4 pools: 'Color' shader, 2.02 shader, fixed and not. It's now called "Depth capture related tests" on the game menu by the way, as I added another big room to demo particles in (although it'll have to stay empty until tomorrow as I've been working on this water fix tonight instead).

 

Here's the before and after, both pics based on the 2.02 shader:

 

[motorsep]

How come the other objects go out of whack ? ("bridge", walls, etc.)

[SteveL]

It looks less out of whack when it's moving rather than in a photo. But foreground objects get distorted because the water surface is a post-process effect. After all the solid stuff has been drawn, the water surface pulls the image about using a distorting normal map to simulate a rough water surface. This is a surface meant for a stream, not one you'd normally use on a small pool like this, but it's the right one to use for testing. The problem was that without depth capture, the post-process shader didn't know what was supposed to be in front of the water and what was behind, so foreground objects seen against water got pulled about too. With the depth fix, the shader knows if it's picked up a colour sample from an object in front of the water, and it re-samples from the opposite direction, mirroring the distortion so you don't get a sharp discontinuity. If it's unlucky enough to get another sample from in front of the water, it gives up and leaves that pixel undistorted. But that happens pretty rarely so it's a reasonable fix.

[Bikerdude]

if this hasen't been said enough times before then let me say again, this fix along with the others you have done are very much appreciated mister!

[Springheel]

Yes, this issue was probably on my top ten list of TDM annoyances.

[Obsttorte]

I think the iridescence appeared after Obstorttet's changes to heatHazeWithMask.vfp in revision 13969. Here's comparison:

 

Before:

 

 

After:

 

I never changed that. I only inspected the reason for the issues it was causing.

[SteveL]

I never changed that. I only inspected the reason for the issues it was causing.

Some confusion here evidently. Check the svn log ref that Arcturus gave above... you changed a few shaders, including introducing this effect.

[nbohr1more]

Probably came along with the Sikkpin shaders unintentionally. I suspect that Sikk never used any water effects in his mods so he altered this one

to look better during explosions, etc.

[revelator]

It's nothing to do with the depth capture. The color aberrations are apparently from a new shader prog added to the repo in August. I didn't notice it had been changed. No worries, I'll retry the fix with the 2.02 version of the water shader.

 

Ah thats a chip of my shoulder then i guess unless sikkpins shaders are straight forward that they might need some work to fit darkmod, i kinda like the effect when seen from a distance though.

Finally got over that darn flu but drowning in work on my MinGW64 based compiler suite porting everything to Msys2 fixing build scripts aghhh.

Atleast when its done it will be a nice addition for those who are used to work in a unix environment. Also need to update my special CodeBlocks IDE which i have put of for some time now because the changes needed are in the category freaking insanely hard and evil so im probably not not doing much work on revelation this week .

If anyone wants the latest source for it i can upload it to my sourceforge page, keep in mind that i removed all the editors from it untill we can get the code ported.

The depthbuffer code is also in it.

Edited October 27, 2014 by revelator

[SteveL]

Probably came along with the Sikkpin shaders unintentionally. I suspect that Sikk never used any water effects in his mods so he altered this one

to look better during explosions, etc.

That would explain it. No need for anyone to go tidying them right now, I'll replace them anyway with depth testing versions after leaving the test map up for a couple of days.

 

@revelator I really must learn more about your revelation work. I'm new to rendering projects and my only exposure to yours so far has been through this thread!

[STiFU]

Wow, amazing stuff!! =)

 

Is the depth of the water surface known, too? I am asking, because if both surface and ground depth are known, we could also render murky water I guess... The deeper the water, the less transparent it becomes. Might be problematic though, since we don't have any "correct" specular reflections on the water surface.

 

 

 

 

 

Hehe, when I googled for example images, I also stumbled over this one showing a bug in ass creed that I also came accross.

 

I also saw this. Blending waterborders might be another nice improvement.

[revelator]

http://sourceforge.net/projects/cbadvanced/files/gtkradiant/Revelation-src-27-10-2014.7z/download

 

latest source here, there are a rather large list of changes to the renderer so ask if not sure

 

notable. all editors are gone.

MH's ARB buffer range optimization polished up a bit.

MH's FPS counter (more accurate).

Convoluted DepthBoundsTesting into a wrapper function which is also used for light now, was only used for stencil shadowing originally.

Removed all unused features leftover from the old backends.

Removed the Fallback code for cards not able to use vertexshaders so do not try this on a voodoo2

Changed EPSILON values to double instead of float.

RB_SetStageTexGen new function to avoid the messy state changes the old code had.

Removed Glasswarp (not used).

Added Texture functions for accessing the depthbuffer.

Clamped depthranges.

Removed an ugly hack in the texture manager that was caused by the removed editor code.

Moved a huge part of the stencil shadow code into a struct to make use of pointers.

Stencil shadow code changed interaction to defered (part of a linux patch to use openmp in vanilla sadly i had no luck porting the openmp code).

Fixes from BFG to original vanilla code merged.

TGA changed to TGA2 code, fixes the need for rotating TGA images so that part was removed also.

JPEG upped to version 9a and code changed to fit.

Added nedmalloc to idlib (fast malloc version).

Changed a few functions in nedmalloc to C++ allocation.

updated ogg and vorbis to latest.

merged all fixes that dont change the way vanilla works from dhewm3.

merged more fixes from dante.

all glColor calls are now handled by C++ classes (wrappers for opengl's C interface).

 

a crapload more i cant remember right now.

 

source compiles with msvc2010 and i included ports to a few mods like grimm quest cdoom sikkmod and sikkmod-d3xp.

[Tels]

If anyone wants the latest source for it i can upload it to my sourceforge page, keep in mind that i removed all the editors from it untill we can get the code ported.

The depthbuffer code is also in it.

 

Speaking of these editors, is it really worth keeping them in TDM? Are we even using them? The less cruft we have, the easier maintainance is.

[Tels]

http://sourceforge.n...014.7z/download

 

latest source here, there are a rather large list of changes to the renderer so ask if not sure

 

This sounds like a nice list of improvements and cleanups. Would it somehow possible to merge your code and TDM? It's a shame that these things aren't shared but instead live in two different engine code branches.

[motorsep]

Changed EPSILON values to double instead of float.

 

Any particular reason for doing that ?

[SteveL]

Speaking of these editors, is it really worth keeping them in TDM? Are we even using them? The less cruft we have, the easier maintainance is.

 

Agreed. It would be good to remove the support for the obsolete graphics apis too. They'll get broken by almost anything new we do, unless we take the trouble to update the obsolete code too, which would mean all improvements taking 5 times as long and how would be test them anyway? Anyone got one of those voodoo2s to help with testing?

[motorsep]

So how do you set up AFs without AF editor ?