TDM Engine Development Page

[revelator]

http://sourceforge.net/projects/cbadvanced/files/gtkradiant/darkmod-svn-build.7z/download

 

first build with GLEW and new VBO.

I have linked it to the shared version of glew to ease debugging but when im satisfied theres no problems ill change it to use the static version.

[revelator]

Btw i have removed the old opengl logging functions from source they newer worked anyway and it also makes porting to GLEW more easy.

 

I also started getting rid of the buggers pointed out by fabian sanglard as well as a few i discovered later.

They dont do anything for speed but they might avoid a few crashes that could normally crop up.

 

Ill also start yanking in fixes from other engines like the now defunct iodoom3 and dhewm3 that dont affect how the engine works.

[revelator]

Ok tested and works like a charm i have committed the changes to your github account.

Changed to static GLEW.

Fixed a few buggers pointed out by fabian sanglard.

New VBO code.

Engine is now GLEW based.

[nbohr1more]

I've used the like button too many times today, lol

 

Thanks Rev!

[revelator]

Your welcome m8 hehe.

 

Had to ammend the first commit as i forgot to clean up the build directory everything should be in order now though.

[nbohr1more]

Now the wait to get home and test will be unbearable. Too bad I can't play TDM at work

[HMart]

Fantastic work revelator and team. Nice to see the TDM engine being upgraded.

 

BTW can i start dreaming of something like this on the future?

 

Edited November 12, 2014 by HMart

[revelator]

Hehe not unless you want to keep your job

 

comitted a few more tweaks, and a request to update the boost libray used as the version TDM uses has a problem that was fixed in later versions.

I allready made a new version for revelation though i only use boost in one place atm, i could add that one to the next commit.

 

If you want i could move the changes i made in revelation to use depthbounds testing with light as well as shadows ? it might all add up to better performance

[revelator]

Should be doable with shaders depends on how much cruft the devs will accepth though.

[HMart]

Should be doable with shaders depends on how much cruft the devs will accepth though.

 

Yes i don't expect all of those post-processing effects to be useful to TDM, or even desirable but they do give mission designers extra options, i especially like the color correction one.

[revelator]

could still make the interfaces for them even if unused but not sure if TDM would be the right port to do it in, probably better to make a repo somewhere and copy of some engine with most of the work allready done and go from there. Your welcome to copy of mine allthough i dont have most of those postprocess shaders down yet its on the planner.

[revelator]

Small question for Stevel, what do you think about this change ?

 

// anything that references _currentRender or _currentDepth will automatically get sort = SS_POST_PROCESS
// and coverage = MC_TRANSLUCENT
for (i = 0; i < numStages; i++) {
 shaderStage_t *pStage = &pd->parseStages[i];
 if (pStage->texture.image == globalImages->currentRenderImage ||
  pStage->texture.image == globalImages->currentDepthImage) {
  if (sort != SS_PORTAL_SKY) {
   sort = SS_POST_PROCESS;
   coverage = MC_TRANSLUCENT;
  }
  break;
 }
 if (pStage->newStage) {
  for (int j = 0; j < pStage->newStage->numFragmentProgramImages; j++) {
   if (pStage->newStage->fragmentProgramImages[j] == globalImages->currentRenderImage ||
 pStage->newStage->fragmentProgramImages[j] == globalImages->currentDepthImage) {
 if (sort != SS_PORTAL_SKY) {
  sort = SS_POST_PROCESS;
  coverage = MC_TRANSLUCENT;
 }
 i = numStages;
 break;
   }
  }
 }
}

 

this would mark _currentDepth as a postprocessing command in the engine.

 

im toying a bit with sikkpins ssao shader from the feedback engine to see if its useable with the depthmap changes we cooked up.

[SteveL]

this would mark _currentDepth as a postprocessing command in the engine.

That would mess it up for surfaces that have to define their own draw order I think. Mappers have tweaked the sort value in materials to make transparent things like particles (which normally sort in the 5-10 range) draw in front of glass or water for example, and different particle materials also use different sort values so they can be used together in one particle. What would forcing the sort order help with? Right now any material at any sorting stage can use _currentDepth, and if you need a material to draw as a post-process you can just say so in the material decl.

[SteveL]

Ok tested and works like a charm i have committed the changes to your github account. Changed to static GLEW. Fixed a few buggers pointed out by fabian sanglard. New VBO code. Engine is now GLEW based.

 

Swift work!

 

How does the new vbo code work by the way? nb mentioned combining all VBOs into one but does it include animated meshes? Just wondering how updates to the global vbo work. I'd love to read up on it so I can keep up with this conversation if someone has a link that isn't just a code patch.

[revelator]

Took a little digging but heres the description from MH himself.

 

What the glMapBufferRange stuff does is allow you to take advantage of a VBO streaming pattern that D3D has enjoyed since at least version 7 - in D3D terms it's known as the discard/no-overwrite pattern.

 

A VBO is a GPU resource, and normally, if you try to update a GPU resource that is currently in use for drawing with (entirely possible because of the asynchronous nature of CPU/GPU operation), everything must stall and wait for drawing to complete before the update can happen. The stock Doom 3 code actually double-buffers it's streaming VBOs to try avoid this (in a slightly obfuscated way) but glMapBufferRange is a more robust way.

 

So, I mentioned discard/no-overwrite above. Here's what they do.

 

The buffer is filled in a linear manner. You've got 2mb (or whatever) of space, vertexes are added beginning at position 0, as new vertexes are added they get appended until the buffer fills, then magic happens.

 

This standard update is no-overwrite; your code makes a promise to GL that it's not going to overwrite any region of the buffer that may be currently in use for drawing, and in return GL will let you update the buffer without blocking. In order to be able to keep this promise your code must maintain a counter indicating how much space in the buffer it has previously used, and add new verts to the buffer at this counter position.

 

When the buffer becomes full you "discard". This doesn't throw away anything previously added, instead GL will keep the previous block of buffer memory around for as long as is needed to satisfy any pending draw calls, but will give you a new, fresh block for any further updates. That's the "magic" I mentioned above, and it's what lets you use a streaming VBO without any blocking.

 

This pattern will also let you get rid of Doom 3's double buffering, thus saving you some GPU memory (I haven't yet done this in my code). Because there's no more blocking it will run faster in cases where there is a lot of dynamic buffer usage, but because Doom 3 locks at 60fps it may not be as directly measurable as if the engine was unlocked. Hence the "it feels more responsive but I can't quite put my finger on it" result.

 

There's another chunk of code in the standard Alloc call which deals with updates of non-streaming VBOs and which is implemented in quite an evil manner by the stock Doom 3 code. When updating such a VBO you can get a faster update if the glBufferData params are the same as was previously used for that VBO (the driver can just reuse the previous block of buffer memory instead of needing to fully reallocate). Doom 3 doesn't do that, so it doesn't get these faster updates, but by searching the free static headers list for a VBO that matches and using that instead of just taking the first one from it, it can. Obviously it sucks that you need to search the list in this way, and a better implementation would just store the VBO with the object that uses it, and reuse the same VBO each time. Since this mainly happens with model animations an ever better implementation would use transform feedback to animate the model instead of animating it on the CPU and needing to re-upload verts each frame, but I haven't even looked at that yet.

 

So all in all the stock VBO implementation is an unholy mess that needs serious work to get it functioning right, much the same way as Quake 1 lightmap updates were a mess. That code just represents the start of a process, but I personally don't think it's worth continuing with. I say - wait for the BFG edition, wait and see if that's going to get a source release (Carmack seems keen), and use that as a base for further work instead - chances are that all of this stuff will be fixed in that.

 

That would mess it up for surfaces that have to define their own draw order I think. Mappers have tweaked the sort value in materials to make transparent things like particles (which normally sort in the 5-10 range) draw in front of glass or water for example, and different particle materials also use different sort values so they can be used together in one particle. What would forcing the sort order help with? Right now any material at any sorting stage can use _currentDepth, and if you need a material to draw as a post-process you can just say so in the material decl.

 

Ok im going to remove that part again just wanted to know if there would be any pitfalls and seems there are so not going in.

 

P.s finally nailed a rather obscure bugger with GLEW + vanilla, the typeinfo program would error out of the parser claiming that _MSC_VER was not defined which is utter and total bullshit on an msvc compiler, well guess what you need to enclose 3rd party headers in #ifdef WIN32 for windows headers #ifdef __linux for linux headers and so on and then it works ofc if i had left any linux support in my own engine i would have tons of those defines but atm it only supports windows so this one was rather odd.

[SteveL]

Thanks. That all makes sense. So it's not about merging vbos, it's about a more intelligent way to stream the vbo data to the gpu without halting the process if you happen to want to overwrite a memory block that's in use. You mark that destination memory block for the dustbin once it's finished being read and write the new version of it elsewhere. That's a common technique in databases too (my day job). If I understood it right, it would remove one possible draw bottleneck, where the drawing process prevents new data being sent to the gpu.

 

It'll be interesting to see in what circumstances the technique can speed up rendering. Lots AI on screen would be my first test. Earlier opinions/comments in this thread have been about bus bandwidth being the problem because we send skinned verts for AI to the GPU every frame when they're on screen, rather than drawing being the bottleneck. We also (apparently) have that double-buffering technique to do the same thing less efficiently, so it's a bit unpredictable what effect it'll have until we try it. (or did you go on to remove the double-buffering too?). Thanks for trying it out.

 

Is the build you posted last night able to load a TDM map? or am I jumping the gun?

[nbohr1more]

Yeah, the build works great for me. No real performance gains as far as I can tell but my GPU is probably a bigger bottleneck. Someone with a real video card should test it. Indoor areas seem smoother but that could be placebo.

[Bikerdude]

What do I need to do a test on the large WIP Im working on atm..

[nbohr1more]

What do I need to do a test on the large WIP Im working on atm..

 

Download this:

 

http://sourceforge.net/projects/cbadvanced/files/gtkradiant/darkmod-svn-build.7z/download

 

then extract the TheDarkMod.exe, gamex86.dll, glew.dll, and tdm_game01.pk4 and replace the files in your

SVN folder with the ones from the download. (I renamed the files with a .old extension... eg. TheDarkMod.exe.old for our executable

so I can return the files back if needed. )

[SteveL]

You can simply delete the files and do an SVN Update to revert them to the originals. It'll restore the files from your cache, it won't need to download them again. At least, that's what happens with Tortoise svn.

[nbohr1more]

@Revelator:

I'm not seeing the changes on my Github? Perhaps it take the site awhile to update? You set your updates to be public correct?

[Bikerdude]

then extract the TheDarkMod.exe, gamex86.dll, glew.dll, and tdm_game01.pk4 and replace the files in your SVN folder with the ones from the download. (I renamed the files with a .old extension... eg. TheDarkMod.exe.old for our executable so I can return the files back if needed. )

You can simply delete the files and do an SVN Update to revert them to the originals. It'll restore the files from your cache, it won't need to download them again. At least, that's what happens with Tortoise svn.

Ta.

[Tels]

Hehe not unless you want to keep your job

 

comitted a few more tweaks, and a request to update the boost libray used as the version TDM uses has a problem that was fixed in later versions.

 

I allready made a new version for revelation though i only use boost in one place atm, i could add that one to the next commit.

 

I hoped we could get rid of boost. It solves only a few problems that I'm sure we can solve without, but brings the massive downside binding the compiled version to the compatiblity version of the OS. E.g. you can't run a compiled-on-a-new Ubuntu TDM on old Ubuntu, and you can't run the compiled-on-old-ubuntu version on newers, so you have to compile TDM on a semi-old ubuntu to make it available to more users....

[nbohr1more]

I hoped we could get rid of boost. It solves only a few problems that I'm sure we can solve without, but brings the massive downside binding the compiled version to the compatiblity version of the OS. E.g. you can't run a compiled-on-a-new Ubuntu TDM on old Ubuntu, and you can't run the compiled-on-old-ubuntu version on newers, so you have to compile TDM on a semi-old ubuntu to make it available to more users....

 

+1

 

Edit: If OpenMP can't be made to work under windows, Boost may be one of the few choices left for pre-built threading... Of course, BFG style

jobs would also be an option.

Edited November 13, 2014 by nbohr1more

[motorsep]

OpenMP limits you to physical cores (unless it's no longer the case). Job queues limit you to threads. So on my 3-core CPU, with OpenMP I'd get only 3 tasks running in parallel. With BFG job queues I can get up to 8 on Win 7.