Doom 3 Dante Engine Runs Faster With GLX Over EGL

[Ladro]

This is the news:

 

http://www.phoronix....item&px=MTE4ODM

 

http://youtu.be/PYGeXko_xf0

 

(if some moderators think that it's OT here, you can move this topic on the offtopic section, no problem )

Edited September 20, 2012 by Ladro

[New Horizon]

Interesting though the delivery of the content was less than riveting. lol

[lost_soul]

How hard would it be to use something like this? Is TDM tied to the stock (although now modified) Doom 3 engine, or will we be able to run TDM on other (faster?) Doom 3 engines that come along? Think of something like Team Fortress for Quake, which will happily run on pretty much any Quake fork. I imagine this would be more difficult for TDM because of the light-gem system. Hopefully it isn't hard to run TDM on custom Doom 3 engines that show up for e.g. Android.

[New Horizon]

How hard would it be to use something like this? Is TDM tied to the stock (although now modified) Doom 3 engine, or will we be able to run TDM on other (faster?) Doom 3 engines that come along? Think of something like Team Fortress for Quake, which will happily run on pretty much any Quake fork. I imagine this would be more difficult for TDM because of the light-gem system. Hopefully it isn't hard to run TDM on custom Doom 3 engines that show up for e.g. Android.

 

Since the release of the source code TDM no longer runs 'on' Doom 3...it runs on a Doom 3 variant...which is now the TDM engine. So you can't simply install TDM next to a modified D3 engine and expect it to work.

 

TDM would either have to be completely ported to one of these engines, or the desired features would have to be imported into the TDM engine. From what I understood of this talk though, his engine which now runs on ESL is "slower" than when it previously ran on GLX. At least that's my understanding.

 

1.08 has already seen some great optimizations in terms of ingame performance.

[Melan]

1.08 has already seen some great optimizations in terms of ingame performance.

That's good to hear! How much of an improvement are we talking? (Very roughly speaking, of course)

[nbohr1more]

I just want to know the results for Tracker 3234.

 

Purportedly this improves Doom 3's Vertex processing tremendously in terms of lowering CPU use.

 

Mh is not motivated to keep enhancing this because "Doom 3 is locked at 60fps and already runs full speed".

 

He needs to see that other Id Tech 4 projects can benefit from optimizations then he might complete this work.

 

Any takers?

[Tels]

That's good to hear! How much of an improvement are we talking? (Very roughly speaking, of course)

 

Not much, really. A few percent here or there.

 

Bigger improvements like the compressed normals (reduce memory usage and loading times by 25%..30%), animations on the GPU (no longer CPU bound) are all doable, but we don't have anyone working on them right now.

[Tels]

I'm not sure how to test this, it would need a controlled before/after test but FPS fluctuates on my system way to wildly. Also, he says tha the code needs even more cleanup, plus the animations should not be done on the CPU and then uploaded,but done on the GPU. These would probably give much bigger boosts.

[nbohr1more]

Try running a timedemo. The FPS is unbound in a timedemo.

 

This is a renderer fix so AI is not a concern here.

[New Horizon]

Not much, really. A few percent here or there.

 

Depends on the system. For mine, I see an improvement of at least 5 to 10 fps. I'll have to try return to the city on the latest build.

[RPGista]

In native resolution I get around 10-15 fps in my laptop, so that would give me at least a 50% boost, not bad at all.

[New Horizon]

Haha. Well, results may vary. I think lower end systems like mine may get the most out of it, but higher end systems won't see much difference since they're already handling things well.

[Radiant]

I've got a questions about all this light and performance stuff:

In Carmack's recent keynote he mentioned that Doom 3 handled every light dynamic. In Rage they came down to the point that an addition of lightmaps, to the dynamic lights are much more better for outside areas. You can see this (half precalculated, half dynamic generated shadows) in the shady side of a mountain. The shadow of a car is slightly darker than the shadow of the mountain. (They don't merge.)

 

Is there a chance to put in lightmaps back into Doom3 with the now known source code? I bet this would improve performance a lot.

(The lightmap could be used on any non-extinguishable light.)

 

 

====

About the video:

Go, get this man into TDM team!

Edited September 20, 2012 by Radiant

[HMart]

I've got a questions about all this light and performance stuff:

In Carmack's recent keynote he mentioned that Doom 3 handled every light dynamic. In Rage they came down to the point that an addition of lightmaps, to the dynamic lights are much more better for outside areas. You can see this (half precalculated, half dynamic generated shadows) in the shady side of a mountain. The shadow of a car is slightly darker than the shadow of the mountain. (They don't merge.)

 

Is there a chance to put in lightmaps back into Doom3 with the now known source code? I bet this would improve performance a lot.

(The lightmap could be used on any non-extinguishable light.)

 

 

====

About the video:

Go, get this man into TDM team!

 

 

You can already use lightmaps on Doom3 by using a shader like this one:

 

//////////////////////////////////////////////////////////
//
// To test Lightmaps
//
/////////////////////////////////////////////////////////

textures/bod/lightmaps/vulcano_lm
{
polygonOffset 1
sort decal
noSelfShadow
noshadows

qer_editorimage textures/bod/lightmaps/vulcano_lm_ed.jpg
// Multiply lightingmap by any color behind this surface on the screen.
{
	blend modulate
	map textures/bod/lightmaps/vulcano_lm.tga
}
}

 

There's some problems with this, you still need a dynamic light casting on all the geometry for the lightmap to show (just use a big ambient light for example) and all the dynamic objects will not take the lightmap into account, they will only be illuminated by dynamic lights, of course coders can just make it so the character casts a ray on the floor and samples the color of the polygon it hits, if it's in shadow just lower the brightness of the character texture, of course this is far from realistic also you lose normal maps.

Edited September 20, 2012 by HMart

[Radiant]

Thank you very much, I will definitely try this!

[deadite4]

I just want to know the results for Tracker 3234.

 

Purportedly this improves Doom 3's Vertex processing tremendously in terms of lowering CPU use.

 

Mh is not motivated to keep enhancing this because "Doom 3 is locked at 60fps and already runs full speed".

 

He needs to see that other Id Tech 4 projects can benefit from optimizations then he might complete this work.

 

Any takers?

 

What kind of optimizations. We are always looking for improvements

[nbohr1more]

What the glMapBufferRange stuff does is allow you to take advantage of a VBO streaming pattern that D3D has enjoyed since at least version 7 - in D3D terms it's known as the discard/no-overwrite pattern.

 

A VBO is a GPU resource, and normally, if you try to update a GPU resource that is currently in use for drawing with (entirely possible because of the asynchronous nature of CPU/GPU operation), everything must stall and wait for drawing to complete before the update can happen. The stock Doom 3 code actually double-buffers it's streaming VBOs to try avoid this (in a slightly obfuscated way) but glMapBufferRange is a more robust way.

 

So, I mentioned discard/no-overwrite above. Here's what they do.

 

The buffer is filled in a linear manner. You've got 2mb (or whatever) of space, vertexes are added beginning at position 0, as new vertexes are added they get appended until the buffer fills, then magic happens.

 

This standard update is no-overwrite; your code makes a promise to GL that it's not going to overwrite any region of the buffer that may be currently in use for drawing, and in return GL will let you update the buffer without blocking. In order to be able to keep this promise your code must maintain a counter indicating how much space in the buffer it has previously used, and add new verts to the buffer at this counter position.

 

When the buffer becomes full you "discard". This doesn't throw away anything previously added, instead GL will keep the previous block of buffer memory around for as long as is needed to satisfy any pending draw calls, but will give you a new, fresh block for any further updates. That's the "magic" I mentioned above, and it's what lets you use a streaming VBO without any blocking.

 

This pattern will also let you get rid of Doom 3's double buffering, thus saving you some GPU memory (I haven't yet done this in my code). Because there's no more blocking it will run faster in cases where there is a lot of dynamic buffer usage, but because Doom 3 locks at 60fps it may not be as directly measurable as if the engine was unlocked. Hence the "it feels more responsive but I can't quite put my finger on it" result.

 

There's another chunk of code in the standard Alloc call which deals with updates of non-streaming VBOs and which is implemented in quite an evil manner by the stock Doom 3 code. When updating such a VBO you can get a faster update if the glBufferData params are the same as was previously used for that VBO (the driver can just reuse the previous block of buffer memory instead of needing to fully reallocate). Doom 3 doesn't do that, so it doesn't get these faster updates, but by searching the free static headers list for a VBO that matches and using that instead of just taking the first one from it, it can. Obviously it sucks that you need to search the list in this way, and a better implementation would just store the VBO with the object that uses it, and reuse the same VBO each time. Since this mainly happens with model animations an ever better implementation would use transform feedback to animate the model instead of animating it on the CPU and needing to re-upload verts each frame, but I haven't even looked at that yet.

 

So all in all the stock VBO implementation is an unholy mess that needs serious work to get it functioning right, much the same way as Quake 1 lightmap updates were a mess. That code just represents the start of a process.

 

https://dl.dropbox.c...7706561/VBO.zip

 

https://dl.dropbox.com/u/17706561/VBO.zip

 

Just replace VertexCache.cpp and VertexCache.h with the files in this package.

 

 

You might also need to add glMapBufferRange to the GL extensions list (I think Mh just queries

GL caps at runtime).

 

 

Reckless posted this to the internal Id Tech 4 Source code forum over at Doom3world.

 

I'd love to see it profiled for CPU usage against the native version...

 

 

Edit:

 

Got a reply from Mh:

 

 

So, using the Doom 3-style update method we get 522fps; with glMapBufferRange it's 572, consistent across multiple timedemos.

 

Here he was just comparing particle systems in quake with the same vertex code structure as Doom 3 so he may be hitting other bottlenecks too (at such high FPS).

Edited September 21, 2012 by nbohr1more

[HMart]

Thank you very much, I will definitely try this!

 

I forgot to say that shader alone will not work has intended, to see how to really use this technique see this link.

 

http://forum.tgmonline.it/showthread.php?5137-Lightmaps-in-Doom-3-Engine-A-Hybrid-Approach-To-Real-Time-Lighting

[nbohr1more]

The problem with lightmaps in Doom 3 is that they need their own vertices. The engine cannot natively share UV space across materials. That said, if the lightmap "model" itself is not "lit" by realtime lights, then the vertex count is static compared to the additive mechanism for real-time lights. It would be a net win as long as parts of the model were culled somehow... On small maps the "whole world light model" is small enough... and if it's just for ambient gradation you could arguable go pretty low poly and low resolution for the UV map. Of course, a native "true lightmap" method that can share the existing UV's or (better) a volume lightmap or irradiance volume would impart it's light on dynamic objects too (thus avoiding that painted-on look... when dynamic objects have a different light model than the scenery).

[lost_soul]

I imagine light maps would be good for a city. You could have lots of lights on high buildings and e.g. windows and it wouldn't be a problem in areas where the player can not get to them.

[demagogue]

Those kinds of lights I'd think a mapper would already put "noshadow" on them anyway, so they're not taking up too many resources as it is.

 

I've come to appreciate id4's unified lighting system. You drop in a light; it hits geometry; you get shadows ... all dynamically. There's no handwaving involved anywhere. Of course I could understand places looking very good or getting a performance boost with soft static shadows or light maps baked in, but in the balance of options, I'm ok with these being things you have to spend time on to get rigged in id4 & looking good, whereas for most lighting it's just drop & go and the engine dynamically takes care of the rest whatever happens in the game.

[nbohr1more]

I just checked the gl_arb_map_buffer_range extension for hardware compatibility and it is supported

as far back as Radeon 9xxx and FX 5xxx series hardware.

 

Also, Serpentine doesn't think the patch from Mh will break any critical changes he made as he didn't do

must to the VertexCache code.

 

I wish I could afford MSVC 2012 so I could compile this after the release. I guess I could install Ubuntu and learn Scons...

 

 

BTW... does this help at all:

 

 

 

Hey,

 

If the mods want "how to" stuff posted in a different format or manner just let me know, I posted up the gui one and this one back to back since I just put these in my code.

 

A FULL recompile of the entire codebase on a quad core CPU takes over 15 to 20 minutes. After fixing the precompiled header settings(walk through below) the time it takes to recompile the codebase drops down to 3.26 minutes. Recompiling the game project(which happens when you modify any of the headers in the game project) went from almost 5mins to 16.47 seconds.

 

THE PROBLEM: If you make any changes to any shared header files it takes FOREVER to recompile. Any global engine stuff requires EVERYTHING to be recompiled; changing anything in game_local also requires all the game code to be recompiled, this also has to reparse all the engine stuff in precompiled.h which takes forever, since the precompiled header settings aren't setup properly. Also a lot of people have stated intellisence is REALLY slow in VS2010 when editing the doom 3 codebase, this is a direct result of not using precompiled headers.

 

THE SOLUTION: Fixing the precompiled headers so they operate correctly just means all the global shared engine stuff is compiled/parsed ONCE per project, which will cut down compile times and speed up intellisence.

 

The easiest way to fix this problem is doing numerous "replace in files". In doing so just in case you type something wrong screws up your whole project, you should backup your entire codebase or have a way to revert to its current state before you do this.

 

Open up the "Replace in files" dialog in Visual Studio and have it replace the following with #include "precompiled.h". Also ensure "look in" is set to entire solution.

• 
  • #include "../precompiled.h"
  • #include "../idlib/precompiled.h"
  • #include "../../idlib/precompiled.h"
  • #include "../../../idlib/precompiled.h"
  

In the typeinfo project we need to add a simple hack so typeinfo knows were to find the precompiled.h.

 

 

 

Code:

if(!strstr( fileName.c_str(), "precompiled.h" )) {

fileName = fileSystem->RelativePathToOSPath(va("../%s/idLib/precompiled.h", SOURCE_CODE_BASE_FOLDER));

}

• 
  • TypeInfoGen.cpp -- function CreateTypeInfo.
   
  Just above "if ( !src.LoadFile( fileName, true ) ) {", add the following:
  

In idlib,TypeInfo, game, mayaimport, and DoomDLL projects do the following:

• 
  • In the project settings(right click the project -> Properties -> C/C++ -> precompiled headers), change "Precompiled Header" to "Use", and change "Precompiled Header File" to "precompiled.h", and hit ok.
   
  • In all the projects listed above(except idLib) add "idlib/precompiled.cpp" to your project.
   
  • Right click on precompiled.cpp, navigate to the "precompiled.cpp" properties section and switch "Precompiled Header" to "Create".
  

In the game project, TypeInfo.cpp can't use precompiled headers

 

• You need to set switch the "Precompiled Header" attribute on TypeInfo.cpp to "Not using precompiled headers", and in that file at the top change #include "precompiled.h" to #include "../../idlib/precompiled.h".

 

In the DoomDLL project, .c can't use precompiled headers that are compiled using c++

• Using the same technique as stated above, all the .c files in the Rendering/JPeg-6, Sound/oggsrc and Sound/vorbissrc, switch the compile flag to "Not using precompiled headers"

 

Done.

 

 

(From jmarshall23 over at Doom3world)

[Tels]

BTW... does this help at all:

 

(From jmarshall23 over at Doom3world)

 

It sounds something we like to have, but I fear it only helps the Windows guis. But would be absolutely lovelable when it works for Linux, because here everything is recompiled everytime all the time every. Fucking. Single. Time. And I can't stop this, so it compiles for 16..18 minutes even when you just change a single line.

 

Makes working with the code base in Linux impossible.