Jump to content


Photo

TDM Engine Development Page

idtech4 development engine

  • Please log in to reply
856 replies to this topic

#1 zergrush

zergrush

    Member

  • Member
  • PipPip
  • 184 posts

Posted 14 October 2013 - 04:24 AM

I've recently posted this into another thread, but I thought I might as well make a thread about it.

Regarding engine upgrading, I was wondering, it would perhaps be best to create a separate webpage dedicated to the TDM Engine, chiefly to document new features and updates that separate it from idtech4, but also to attract potential developers that can help improve it. Is there a github page for the code, or something like it?

Edited by zergrush, 14 October 2013 - 05:56 AM.

  • Bikerdude, someTaff and Moonbo like this

#2 Serpentine

Serpentine

    Uber member

  • Active Developer
  • PipPipPipPip
  • 2903 posts

Posted 14 October 2013 - 05:05 AM

The engine and the mod work in lock step, this topic needs to be handled internally first so that we can get a better idea of the path forward in that regard.

I have some plans on what I want to do to the engine asap, but they are pretty major and require people be happy with them. If they even go ahead at all.

But yeah, first I think we'll talk it over as a team.

Also the engine doesn't really have a name, and its almost certainly not 'Dark Engine', since well, that's the Thief engine name.
  • Petike the Taffer likes this

#3 demagogue

demagogue

    Mod hero

  • Active Developer
  • PipPipPipPipPip
  • 5382 posts

Posted 14 October 2013 - 05:20 AM

Yeah you need to change the name of the thread or put an explanation right up front in the OP that we're talking about TDM's engine and not Thief's Dark Engine, since it's very confusing.

It's true that our engine is quite different from idtech4 at this point. If you look at the sourcecode for both, I mean just the folder of files, it's blatantly obvious just by sight. So it deserves its own name. But I think in practice people would just call it the TDM Engine. Most of discussion about it is in the Developer's forum here though, since it's still in development from version to version.

What it looks like you're talking about in the OP--putting up a public page on the engine--is really for branch projects separate from the team, for fans that just want to play around with it. I mean, that's what I think its value would be, since we already have people working on the engine for the next version internally. Anyway, I think that would be a valuable thing, because we all have the open-source ethos around here of share-alike. Our engine is a good engine, and it'd be a great base for all sorts of indie games, much better than most other open source engines you could start with (to say nothing of smaller possible branch projects, like "multiplayer TDM" or "TDM+Mirror's Edge 2 Moves", etc, a fan could do just for fun.) But for it to have that role, it needs some primers to get fans out there to know how the code works and how to experiment with it.

Edit: That said, the code is maaasive. Such a project would be a fantastic undertaking. And I don't know how useful a basic explanation would be anyway. Nothing really beats just opening up the code & reading it oneself, and experimenting with it locally to see what happens, and then maybe asking one of our coders any questions. Something to be said for the good old fashioned way.

Edit2: Also like Serpentine is saying, you have to distinguish projects you want the team to do for TDM in future versions versus projects you want to do yourself locally, just for yourself, or other fans doing it for themselves. Projects for yourself you are free to do at your leisure. And if it turns out really well, you might show it to some people & make a good case for inclusion in the mod if it really is special. But for doing things for the mod, it's a much bigger deal, because then you have to get the team involved & there has to be a lot of debate and back & forth discussion and bug fixing and you have to know when to push and when to compromise... In my experience, when you're starting off learning the code, the best thing to do is just play around with it locally for yourself and make some changes that make you happy, and once you get used to how things work, and also how the mod works, you can think about what's worth bringing to the attention of the team.

Edit3: And incidentally, there are established things around already... There's a page specifically on things we might want to add to the engine now that it's open source. http://wiki.thedarkm...ch4_Open_Source Adding to that wiki page, making a thread on the forums with a request, & adding feature requests to the bug tracker (probably after vetting it in a post first) are the first places you'd probably start with requests for what we might do with the engine.
Posted Image

#4 zergrush

zergrush

    Member

  • Member
  • PipPip
  • 184 posts

Posted 14 October 2013 - 05:52 AM

I was convinced TDM's engine was named Dark Engine, because I had the impression I had read it once around here. I probably misread some post or whatever. My bad.

Anyway, regardless of the internal development by the team, I don't think having a separate project page and a main branch wouldn't hurt anything at all. People are already free to fork the engine as they want it, and a page wouldn't make it a lot easier for non-team members to submit fixes and suggest improvements. Given that TDM is clearly lacking on the performance side a little, this seems like a logical step to start smoothing out some rough edges.

*EDIT* changed thread name.

Edited by zergrush, 14 October 2013 - 05:54 AM.


#5 demagogue

demagogue

    Mod hero

  • Active Developer
  • PipPipPipPipPip
  • 5382 posts

Posted 14 October 2013 - 06:01 AM

Yeah sure, get some public discussion going on the side. That could be fun for fans, as long as they manage expectations. But it'd be a good learning experience for people anyway.

Somebody has to do the gritty work putting up a page though, so if you're up to it then go for it!
Posted Image

#6 zergrush

zergrush

    Member

  • Member
  • PipPip
  • 184 posts

Posted 14 October 2013 - 07:10 AM

I'm not really a programmer or webdesigner, but what I would suggest to be done in a preliminary stage would be to create an official github repository.

#7 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8872 posts

Posted 14 October 2013 - 07:57 AM

Not a bad idea, especially since most of the pool of contributors would be familiar with that workflow. That said, until TDM is merged with Dhewg most of the
existing Doom 3 coders may not have interest in touching it. I would love to see a few tech demos from folks like RaynorPat or LogicalError who've tinkered
with the original GPL release.

Waback Archive of Cubemap Light code:

https://web.archive....pic.php?t=24193

Edited by nbohr1more, 23 June 2014 - 07:20 PM.

Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#8 vorob

vorob

    Member

  • Member
  • PipPip
  • 88 posts

Posted 17 October 2013 - 12:37 PM

In unreal engine when you put one brush to another, on its connection part there will smth like SSAO, selfshadow mechanism on the corner (http://www.hourences...lightmass/3.jpg), any chance to have the same in darkmod? So such wall-connection would look better:

https://dl.dropboxus...21-03-24-54.png

#9 rich_is_bored

rich_is_bored

    Advanced Member

  • Member
  • PipPipPip
  • 878 posts

Posted 17 October 2013 - 01:54 PM

Sure there's a chance albeit slim if you counting on existing contributors. But the code is available. If someone capable wants it enough they can do it.

#10 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8872 posts

Posted 20 October 2013 - 11:01 AM

So, Sikkpin has offered to help.

At first blush all I could think of is his excellent shader work but I also know that he's not really good with GLSL so
if any future build of TDM goes GLSL his work would have to be converted (there are ARB to GLSL converters as I recall) then optimized.

For the near future would it be worth it to contact him for some shaders? I think everyone here would be pretty excited for his subsurface scattering
shaders (better human skin and hair). Plus, his SSAO work is pretty damned awesome... or at the very least some tangent space AO added to
the default ambient would look nice.

I have a feeling he would like to look at game code since he's been focusing more on that side of things for his own project. Anyone have any projects
you'd think he'd be suited for?

Maybe we should prompt him for assistance sooner rather than later before he changes his mind? :)


Link for reference:

http://forums.thedar...on-doom3-world/

Edited by nbohr1more, 01 March 2014 - 11:02 PM.

  • Bikerdude, AluminumHaste, STiFU and 1 other like this
Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#11 New Horizon

New Horizon

    Mod hero

  • Active Developer
  • PipPipPipPipPip
  • 13815 posts

Posted 20 October 2013 - 02:46 PM

So, Sikkpin has offered to help.

At first blush all I could think of is his excellent shader work but I also know that he's not really good with GLSL so
if any future build of TDM goes GLSL his work would have to be converted (there are ARB to GLSL converters as I recall) then optimized.

For the near future would it be worth it to contact him for some shaders? I think everyone here would be pretty excited for his subsurface scattering
shaders (better human skin and hair). Plus, his SSAO work is pretty damned awesome... or at the very least some tangent space AO added to
the default ambient would look nice.

I have a feeling he would like to look at game code since he's been focusing more on that side of things for his own project. Anyone have any projects
you'd think he'd be suited for?

Maybe we should prompt him for assistance sooner rather than later before he changes his mind? :)


In my opinion, anything brought to the table in terms of these types of improvements would hopefully be optional and not require recreating a good chunk of our resources to support them. I seem to recall some of sikk's enhancements requiring a completely different type of normal map texture to even use them.

#12 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8872 posts

Posted 20 October 2013 - 03:06 PM

That was an old AO method (he re-engineered a shader that Humus created), to my knowledge his latest stuff doesn't require new textures except for the POM or Parallax
shaders which need height maps.

Tangent space AO is a very light (performance wise) directional AO that could add back some of the nice shading that was lost when JC Denton's
enhanced ambient was removed.

Yes, that hard coded directional shading in JC Denton's work was "wrong" but it helped alleviate the flat look of some maps where no or few "normal" lights were used.

It's hard to say if that would be a good way forward anyway since so many textures already have now had AO baked into them (which would also be tangent space based).
(such work also complicates any further AO designs... so it may mean that "many" textures need to be reworked anyway for consistency).

Plus if there's something else on the roadmap for AO, then such an intermediate step would be a waste of energy (etc)?

Another possibility? "Real" Lights are reworked to consume less performance therefore can be used in place of Ambient Lights (in places where
more defined shading is needed) leaving the ambient for it's sole role as a baseline ambient brightness level.

Hard to say without knowing what the long term plans are.

Edited by nbohr1more, 20 October 2013 - 03:30 PM.

  • someTaff likes this
Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#13 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8872 posts

Posted 30 October 2013 - 11:23 AM

Was wondering about whether they finally made the silhouette calculations run on the GPU in BFG edition...

Apparently not but the new skinning method has reduced the impact of the whole process and allows more
parallel processing of these operations. Not the way I'd hoped they would fix that but it's interesting nonetheless:

5
3
. Compute vs. Memory
Another interesting (but not too surprising) difference between the original DOOM 3 and DOOM 3 BFG is the
shift from memory usage to compute. Whereas in the original DOOM 3 an animated mesh for a game
character was only ever skinned once per frame, in DOOM 3 BFG an animated mesh
for a game character may be skinned many times per frame, simply because it is faster.

In the original DOOM 3 an animated mesh for a single instance of a game character
was skinned only once per frame and the uniquely skinned version of the mesh was stored out to memory.
The skinned copy in memory was then used to construct shadow volumes and it was also sent to the GPU to
actually render the mesh (or shadow volumes). This all made sense 8 years ago when the balance between compute
and memory bandwidth was somewhat different.

In DOOM 3 BFG an animated mesh for a single instance of a game character may be skinned many times per
frame. The simple observation is that the cost of skinning a mesh may be mostly hidden behind the cost of
reading/streaming the "un-skinned" source mesh from memory. In other words, reading/streaming the "un-skinned"
source mesh from memory and skinning it in-place does not cost much more than reading/streaming
an already skinned mesh from memory without performing any additional calculations.
The big win comes from never having to write the skinned mesh back to memory and never having to
send/copy it to the GPU. In addition, far less memory is being read because while the same "un-skinned"
source mesh may be used by many instances of a game character, a fully skinned mesh is unique to a single
instance of a game character. Reading less memory reduces memory bandwidth if data can remain in CPU
caches. It may also result in less cache thrashing, in particular when performing a lot of work in parallel where
all parallel threads (or tasks) use the same source data (as opposed to unique source data).

To construct a shadow volume the source mesh is read, skinned in-place, followed by immediately calculating
the facing triangles and silhouette edges. In DOOM 3 BFG a shadow volume is no more than a set of indices
that reference a static "un-skinned" source mesh in GPU memory that is skinned on the GPU right before
rasterization. As a result, an animating mesh for a single instance of a game character may be skinned many
times per frame in DOOM 3 BFG.

For instance, if an animating mesh interacts with 2 shadow casting lights
(very common) then the mesh may be skinned 7 times per frame in DOOM 3 BFG.

2x CPU skinned to construct 2 shadow volumes
1x GPU skinned to render the depth pass
2x GPU skinned to render 2 shadow volumes
2x GPU skinned to render 2 light surfaces

In other words, DOOM 3 BFG may, for a typical animated mesh, use 7 times the number of FLOPS compared
to the original DOOM 3. However, DOOM 3 BFG runs noticeably faster on today's hardware.
As an added bonus DOOM 3 BFG now maintains less state (no longer stores a skinned copy of a mesh per
instance of a game character). The less state is maintained the easier it is to understand and reason about
code and the less likely the code will have bugs. The stateless nature also allows a lot of code to run in parallel
without contention over resources. All shadow volumes can now be constructed in parallel without the need for
synchronization primitives like mutexes or critical sections.



http://fabiensanglar...hnical-Note.pdf

On the other hand:



Unlike the original DOOM 3, nothing is done lazily anymore (except for generating animation frames, no more
flags and tests like: "calculate this right now if it has not been calculated yet"). All the high performance CPU
work now follows the streaming programming model. For dynamic shadow volumes the engine now culls
triangles to the light volume, both to reduce the number of shadow volume triangles and to reduce the number
of triangles that are redrawn for each light pass. The engine now also performs a precise test to determine whether the view intersects or is
inside a shadow volume.

This precise test is important to significantly reduce the number of cases where shadow volumes have to be rendered with Z-fail [1] because Z-fail rendering is
significantly slower on a various graphics hardware (in particular at high resolutions). For this precise inside test a line-versus-expanded-
triangle intersection test is performed for every single shadow volume near cap triangle where the linegoes from the view origin to the light origin.

The shadow volume is also transformed into clip space and the polygons are clipped to the view in homogeneous space to calculate very tight depth
bounds. In particular in the cases where shadow volumes do have to be rendered with Z-fail, the depth bounds are used to get more Z-Cull/Hi-Z benefit
(at least on the hardware that supports the depth bounds test).


Edited by nbohr1more, 30 October 2013 - 11:44 AM.

  • AluminumHaste likes this
Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#14 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8872 posts

Posted 17 February 2014 - 05:52 PM

Looks like Raynorpat has ported the BFG GLSL backend to vanilla Doom 3:

https://github.com/raynorpat/morpheus

http://forums.inside...=3491&start=765

Hopefully he'll get it fully functional for Intel GPU's
Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#15 HMart

HMart

    Advanced Member

  • Member
  • PipPipPip
  • 639 posts

Posted 17 February 2014 - 09:02 PM

And it has OpenAL soft with EFX support already implemented ;)

#16 Springheel

Springheel

    Creative Director (retired)

  • Admin
  • 37250 posts

Posted 17 February 2014 - 09:15 PM

Does that make it usable for us?
  • Deadlove likes this
TDM Missions:   A Score to Settle   *   A Reputation to Uphold   *   A New Job   *    A Matter of Hours
 
Video Series:   Springheel's Modules   *   Speedbuild Challenge   *   New Mappers Workshop  *   Building Traps

#17 Bikerdude

Bikerdude

    Mod hero

  • Member
  • PipPipPipPipPip
  • 19978 posts

Posted 17 February 2014 - 10:24 PM

Looks like Raynorpat has ported the BFG GLSL backend to vanilla Doom 3:Hopefully he'll get it fully functional for Intel GPU's

Is there a timeframe for that..? Also what news on Sikpin has he started working on any of the code..?

And it has OpenAL soft with EFX support already implemented

Does that make it usable for us?

+1, I dearly want OpenAL so we can start putting EAX / EFX into missions.

#18 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8872 posts

Posted 17 February 2014 - 10:28 PM

It's a branch of the original vanilla source and it probably includes some dhewm3 fixes. It might even be easier to
port to than Dhewm3 but in any case the renderer fixes should be portable. He's got a better VBO Cache structure
and working GLSL from BFG edition, can't say what else is in there but if he's gone this far perhaps he'll keep going
and add GPU Skinning and better shadow code.

As for Sikkpin, we have the latest public code release from him. Rebb would be the guy to evaluate whether any of
it's worthwhile but it has some nice fixes... can't speak to how it performs. His code is even closer to vanilla than either
of these though.
Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#19 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8872 posts

Posted 06 March 2014 - 12:45 PM

I created tracker http://bugs.thedarkm...iew.php?id=3684

to be a parent tracker for engine improvements.

Yes, it's more or less a wish list but having it on record at least gives a target to point to.

I tried to link child trackers to it but I don't have privileges to do this.

Existing trackers:

3234
3669
Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#20 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8872 posts

Posted 14 June 2014 - 02:40 PM

Relevator just posted some code changes to optimize Shadow rendering and deferred lighting:

http://forums.inside....php?f=9&t=5467
  • Bikerdude, Lux and SteveL like this
Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#21 New Horizon

New Horizon

    Mod hero

  • Active Developer
  • PipPipPipPipPip
  • 13815 posts

Posted 14 June 2014 - 02:54 PM

Would these changes be compatible with the TDM engine? I don't have everything setup to build the engine anymore, otherwise I would attempt to paste those changes in and give them a run. Would be interesting to see what kind of performance bump it gave, if any.

#22 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8872 posts

Posted 14 June 2014 - 04:00 PM

Looks like it. Relevators branch is based on vanilla.
Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#23 Obsttorte

Obsttorte

    Scripting guru, Mapper

  • Active Developer
  • PipPipPipPipPip
  • 5609 posts

Posted 14 June 2014 - 04:07 PM

*
POPULAR

Will test this tomorrow.
  • Springheel, nbohr1more, RPGista and 3 others like this
FM's: Builder Roads, Old Habits, Old Habits Rebuild
WIP's: Several. Although after playing Thief 4 I really wanna make a city mission.
Mapping and Scripting: Apples and Peaches
Sculptris Models and Tutorials: Obsttortes Models
My wiki articles: Obstipedia
Let's Map TDM YouTube playlist: ObstlerTube
Texture Blending in DR: DR ASE Blend Exporter

End of shameless self promotion.

#24 Obsttorte

Obsttorte

    Scripting guru, Mapper

  • Active Developer
  • PipPipPipPipPip
  • 5609 posts

Posted 15 June 2014 - 04:02 AM

Hi everyone.

I was able to merge the changes into the source code. Beside a linker error, which I was able to fix doing a google search, everything compiles just fine and the game runs.

However, what I do not understand is what these changes should actually do. I've ran Tears of St. Lucia but I don't see any difference.

So what is this for?
FM's: Builder Roads, Old Habits, Old Habits Rebuild
WIP's: Several. Although after playing Thief 4 I really wanna make a city mission.
Mapping and Scripting: Apples and Peaches
Sculptris Models and Tutorials: Obsttortes Models
My wiki articles: Obstipedia
Let's Map TDM YouTube playlist: ObstlerTube
Texture Blending in DR: DR ASE Blend Exporter

End of shameless self promotion.

#25 demagogue

demagogue

    Mod hero

  • Active Developer
  • PipPipPipPipPip
  • 5382 posts

Posted 15 June 2014 - 04:17 AM

The term optimize sounds like the payoff is FPS boost. Check for that.
Posted Image





Also tagged with one or more of these keywords: idtech4, development, engine

1 user(s) are reading this topic

1 members, 0 guests, 0 anonymous users


    New Horizon