TDM Engine Development Page

[zergrush]

I've recently posted this into another thread, but I thought I might as well make a thread about it.

 

Regarding engine upgrading, I was wondering, it would perhaps be best to create a separate webpage dedicated to the TDM Engine, chiefly to document new features and updates that separate it from idtech4, but also to attract potential developers that can help improve it. Is there a github page for the code, or something like it?

Edited October 14, 2013 by zergrush

[Serpentine]

The engine and the mod work in lock step, this topic needs to be handled internally first so that we can get a better idea of the path forward in that regard.

 

I have some plans on what I want to do to the engine asap, but they are pretty major and require people be happy with them. If they even go ahead at all.

 

But yeah, first I think we'll talk it over as a team.

 

Also the engine doesn't really have a name, and its almost certainly not 'Dark Engine', since well, that's the Thief engine name.

[demagogue]

Yeah you need to change the name of the thread or put an explanation right up front in the OP that we're talking about TDM's engine and not Thief's Dark Engine, since it's very confusing.

 

It's true that our engine is quite different from idtech4 at this point. If you look at the sourcecode for both, I mean just the folder of files, it's blatantly obvious just by sight. So it deserves its own name. But I think in practice people would just call it the TDM Engine. Most of discussion about it is in the Developer's forum here though, since it's still in development from version to version.

 

What it looks like you're talking about in the OP--putting up a public page on the engine--is really for branch projects separate from the team, for fans that just want to play around with it. I mean, that's what I think its value would be, since we already have people working on the engine for the next version internally. Anyway, I think that would be a valuable thing, because we all have the open-source ethos around here of share-alike. Our engine is a good engine, and it'd be a great base for all sorts of indie games, much better than most other open source engines you could start with (to say nothing of smaller possible branch projects, like "multiplayer TDM" or "TDM+Mirror's Edge 2 Moves", etc, a fan could do just for fun.) But for it to have that role, it needs some primers to get fans out there to know how the code works and how to experiment with it.

 

Edit: That said, the code is maaasive. Such a project would be a fantastic undertaking. And I don't know how useful a basic explanation would be anyway. Nothing really beats just opening up the code & reading it oneself, and experimenting with it locally to see what happens, and then maybe asking one of our coders any questions. Something to be said for the good old fashioned way.

 

Edit2: Also like Serpentine is saying, you have to distinguish projects you want the team to do for TDM in future versions versus projects you want to do yourself locally, just for yourself, or other fans doing it for themselves. Projects for yourself you are free to do at your leisure. And if it turns out really well, you might show it to some people & make a good case for inclusion in the mod if it really is special. But for doing things for the mod, it's a much bigger deal, because then you have to get the team involved & there has to be a lot of debate and back & forth discussion and bug fixing and you have to know when to push and when to compromise... In my experience, when you're starting off learning the code, the best thing to do is just play around with it locally for yourself and make some changes that make you happy, and once you get used to how things work, and also how the mod works, you can think about what's worth bringing to the attention of the team.

 

Edit3: And incidentally, there are established things around already... There's a page specifically on things we might want to add to the engine now that it's open source. http://wiki.thedarkm...ch4_Open_Source Adding to that wiki page, making a thread on the forums with a request, & adding feature requests to the bug tracker (probably after vetting it in a post first) are the first places you'd probably start with requests for what we might do with the engine.

[zergrush]

I was convinced TDM's engine was named Dark Engine, because I had the impression I had read it once around here. I probably misread some post or whatever. My bad.

 

Anyway, regardless of the internal development by the team, I don't think having a separate project page and a main branch wouldn't hurt anything at all. People are already free to fork the engine as they want it, and a page wouldn't make it a lot easier for non-team members to submit fixes and suggest improvements. Given that TDM is clearly lacking on the performance side a little, this seems like a logical step to start smoothing out some rough edges.

 

*EDIT* changed thread name.

Edited October 14, 2013 by zergrush

[demagogue]

Yeah sure, get some public discussion going on the side. That could be fun for fans, as long as they manage expectations. But it'd be a good learning experience for people anyway.

 

Somebody has to do the gritty work putting up a page though, so if you're up to it then go for it!

[zergrush]

I'm not really a programmer or webdesigner, but what I would suggest to be done in a preliminary stage would be to create an official github repository.

[nbohr1more]

Not a bad idea, especially since most of the pool of contributors would be familiar with that workflow. That said, until TDM is merged with Dhewg most of the

existing Doom 3 coders may not have interest in touching it. I would love to see a few tech demos from folks like RaynorPat or LogicalError who've tinkered

with the original GPL release.

 

Waback Archive of Cubemap Light code:

 

https://web.archive.org/web/20130909073250/http://www.doom3world.org/phpbb2/viewtopic.php?t=24193

Edited June 24, 2014 by nbohr1more

[vorob]

In unreal engine when you put one brush to another, on its connection part there will smth like SSAO, selfshadow mechanism on the corner (http://www.hourences.com/tutorialimages/ue3lightmass/3.jpg), any chance to have the same in darkmod? So such wall-connection would look better:

 

https://dl.dropboxusercontent.com/u/5020311/TheDarkMod%202013-10-17%2021-03-24-54.png

[rich_is_bored]

Sure there's a chance albeit slim if you counting on existing contributors. But the code is available. If someone capable wants it enough they can do it.

[nbohr1more]

So, Sikkpin has offered to help.

 

At first blush all I could think of is his excellent shader work but I also know that he's not really good with GLSL so

if any future build of TDM goes GLSL his work would have to be converted (there are ARB to GLSL converters as I recall) then optimized.

 

For the near future would it be worth it to contact him for some shaders? I think everyone here would be pretty excited for his subsurface scattering

shaders (better human skin and hair). Plus, his SSAO work is pretty damned awesome... or at the very least some tangent space AO added to

the default ambient would look nice.

 

I have a feeling he would like to look at game code since he's been focusing more on that side of things for his own project. Anyone have any projects

you'd think he'd be suited for?

 

Maybe we should prompt him for assistance sooner rather than later before he changes his mind?

 

 

Link for reference:

 

http://forums.thedarkmod.com/topic/15017-current-render-and-fog-fixes-by-sikkpin-on-doom3-world/

Edited March 2, 2014 by nbohr1more

[New Horizon]

So, Sikkpin has offered to help.

 

At first blush all I could think of is his excellent shader work but I also know that he's not really good with GLSL so

if any future build of TDM goes GLSL his work would have to be converted (there are ARB to GLSL converters as I recall) then optimized.

 

For the near future would it be worth it to contact him for some shaders? I think everyone here would be pretty excited for his subsurface scattering

shaders (better human skin and hair). Plus, his SSAO work is pretty damned awesome... or at the very least some tangent space AO added to

the default ambient would look nice.

 

I have a feeling he would like to look at game code since he's been focusing more on that side of things for his own project. Anyone have any projects

you'd think he'd be suited for?

 

Maybe we should prompt him for assistance sooner rather than later before he changes his mind?

 

In my opinion, anything brought to the table in terms of these types of improvements would hopefully be optional and not require recreating a good chunk of our resources to support them. I seem to recall some of sikk's enhancements requiring a completely different type of normal map texture to even use them.

[nbohr1more]

That was an old AO method (he re-engineered a shader that Humus created), to my knowledge his latest stuff doesn't require new textures except for the POM or Parallax

shaders which need height maps.

 

Tangent space AO is a very light (performance wise) directional AO that could add back some of the nice shading that was lost when JC Denton's

enhanced ambient was removed.

 

Yes, that hard coded directional shading in JC Denton's work was "wrong" but it helped alleviate the flat look of some maps where no or few "normal" lights were used.

 

It's hard to say if that would be a good way forward anyway since so many textures already have now had AO baked into them (which would also be tangent space based).

(such work also complicates any further AO designs... so it may mean that "many" textures need to be reworked anyway for consistency).

 

Plus if there's something else on the roadmap for AO, then such an intermediate step would be a waste of energy (etc)?

 

Another possibility? "Real" Lights are reworked to consume less performance therefore can be used in place of Ambient Lights (in places where

more defined shading is needed) leaving the ambient for it's sole role as a baseline ambient brightness level.

 

Hard to say without knowing what the long term plans are.

Edited October 20, 2013 by nbohr1more

[nbohr1more]

Was wondering about whether they finally made the silhouette calculations run on the GPU in BFG edition...

 

Apparently not but the new skinning method has reduced the impact of the whole process and allows more

parallel processing of these operations. Not the way I'd hoped they would fix that but it's interesting nonetheless:

 

 

 

5

3

. Compute vs. Memory

Another interesting (but not too surprising) difference between the original DOOM 3 and DOOM 3 BFG is the

shift from memory usage to compute. Whereas in the original DOOM 3 an animated mesh for a game

character was only ever skinned once per frame, in DOOM 3 BFG an animated mesh

for a game character may be skinned many times per frame, simply because it is faster.

 

In the original DOOM 3 an animated mesh for a single instance of a game character

was skinned only once per frame and the uniquely skinned version of the mesh was stored out to memory.

The skinned copy in memory was then used to construct shadow volumes and it was also sent to the GPU to

actually render the mesh (or shadow volumes). This all made sense 8 years ago when the balance between compute

and memory bandwidth was somewhat different.

 

In DOOM 3 BFG an animated mesh for a single instance of a game character may be skinned many times per

frame. The simple observation is that the cost of skinning a mesh may be mostly hidden behind the cost of

reading/streaming the "un-skinned" source mesh from memory. In other words, reading/streaming the "un-skinned"

source mesh from memory and skinning it in-place does not cost much more than reading/streaming

an already skinned mesh from memory without performing any additional calculations.

The big win comes from never having to write the skinned mesh back to memory and never having to

send/copy it to the GPU. In addition, far less memory is being read because while the same "un-skinned"

source mesh may be used by many instances of a game character, a fully skinned mesh is unique to a single

instance of a game character. Reading less memory reduces memory bandwidth if data can remain in CPU

caches. It may also result in less cache thrashing, in particular when performing a lot of work in parallel where

all parallel threads (or tasks) use the same source data (as opposed to unique source data).

 

To construct a shadow volume the source mesh is read, skinned in-place, followed by immediately calculating

the facing triangles and silhouette edges. In DOOM 3 BFG a shadow volume is no more than a set of indices

that reference a static "un-skinned" source mesh in GPU memory that is skinned on the GPU right before

rasterization. As a result, an animating mesh for a single instance of a game character may be skinned many

times per frame in DOOM 3 BFG.

 

For instance, if an animating mesh interacts with 2 shadow casting lights

(very common) then the mesh may be skinned 7 times per frame in DOOM 3 BFG.

 

2x CPU skinned to construct 2 shadow volumes

1x GPU skinned to render the depth pass

2x GPU skinned to render 2 shadow volumes

2x GPU skinned to render 2 light surfaces

 

In other words, DOOM 3 BFG may, for a typical animated mesh, use 7 times the number of FLOPS compared

to the original DOOM 3. However, DOOM 3 BFG runs noticeably faster on today's hardware.

As an added bonus DOOM 3 BFG now maintains less state (no longer stores a skinned copy of a mesh per

instance of a game character). The less state is maintained the easier it is to understand and reason about

code and the less likely the code will have bugs. The stateless nature also allows a lot of code to run in parallel

without contention over resources. All shadow volumes can now be constructed in parallel without the need for

synchronization primitives like mutexes or critical sections.

 

http://fabiensanglar...hnical-Note.pdf

 

On the other hand:

 

 

Unlike the original DOOM 3, nothing is done lazily anymore (except for generating animation frames, no more

flags and tests like: "calculate this right now if it has not been calculated yet"). All the high performance CPU

work now follows the streaming programming model. For dynamic shadow volumes the engine now culls

triangles to the light volume, both to reduce the number of shadow volume triangles and to reduce the number

of triangles that are redrawn for each light pass. The engine now also performs a precise test to determine whether the view intersects or is

inside a shadow volume.

 

This precise test is important to significantly reduce the number of cases where shadow volumes have to be rendered with Z-fail [1] because Z-fail rendering is

significantly slower on a various graphics hardware (in particular at high resolutions). For this precise inside test a line-versus-expanded-

triangle intersection test is performed for every single shadow volume near cap triangle where the linegoes from the view origin to the light origin.

 

The shadow volume is also transformed into clip space and the polygons are clipped to the view in homogeneous space to calculate very tight depth

bounds. In particular in the cases where shadow volumes do have to be rendered with Z-fail, the depth bounds are used to get more Z-Cull/Hi-Z benefit

(at least on the hardware that supports the depth bounds test).

Edited October 30, 2013 by nbohr1more

[nbohr1more]

Looks like Raynorpat has ported the BFG GLSL backend to vanilla Doom 3:

 

https://github.com/raynorpat/morpheus

 

http://forums.inside3d.com/viewtopic.php?f=9&t=3491&start=765

 

Hopefully he'll get it fully functional for Intel GPU's

[HMart]

And it has OpenAL soft with EFX support already implemented

[Springheel]

Does that make it usable for us?

[Bikerdude]

Looks like Raynorpat has ported the BFG GLSL backend to vanilla Doom 3:Hopefully he'll get it fully functional for Intel GPU's

Is there a timeframe for that..? Also what news on Sikpin has he started working on any of the code..?

And it has OpenAL soft with EFX support already implemented

Does that make it usable for us?

+1, I dearly want OpenAL so we can start putting EAX / EFX into missions.

[nbohr1more]

It's a branch of the original vanilla source and it probably includes some dhewm3 fixes. It might even be easier to

port to than Dhewm3 but in any case the renderer fixes should be portable. He's got a better VBO Cache structure

and working GLSL from BFG edition, can't say what else is in there but if he's gone this far perhaps he'll keep going

and add GPU Skinning and better shadow code.

 

As for Sikkpin, we have the latest public code release from him. Rebb would be the guy to evaluate whether any of

it's worthwhile but it has some nice fixes... can't speak to how it performs. His code is even closer to vanilla than either

of these though.

[nbohr1more]

I created tracker http://bugs.thedarkmod.com/view.php?id=3684

 

to be a parent tracker for engine improvements.

 

Yes, it's more or less a wish list but having it on record at least gives a target to point to.

 

I tried to link child trackers to it but I don't have privileges to do this.

 

Existing trackers:

 

3234

3669

[nbohr1more]

Relevator just posted some code changes to optimize Shadow rendering and deferred lighting:

 

http://forums.inside3d.com/viewtopic.php?f=9&t=5467

[New Horizon]

Would these changes be compatible with the TDM engine? I don't have everything setup to build the engine anymore, otherwise I would attempt to paste those changes in and give them a run. Would be interesting to see what kind of performance bump it gave, if any.

[nbohr1more]

Looks like it. Relevators branch is based on vanilla.

[Obsttorte]

Will test this tomorrow.

[Obsttorte]

Hi everyone.

 

I was able to merge the changes into the source code. Beside a linker error, which I was able to fix doing a google search, everything compiles just fine and the game runs.

 

However, what I do not understand is what these changes should actually do. I've ran Tears of St. Lucia but I don't see any difference.

 

So what is this for?

[demagogue]

The term optimize sounds like the payoff is FPS boost. Check for that.