Testers and reviewers wanted: BFG-style vertex cache

[HMart]

No problem i'm glad to help. Btw i tried to take a screenshot of the bug but it seems the screenshot feature doesn't capture exactly what we see.

[lowenz]

Now it seems fine.

Why are the savegames NOT compatible?

 

[AluminumHaste]

Savegames are never compatible between revisions, but you can disable the safety with a cvar, but you will usually get weird things that can happen.

[cabalistic]

@lowenz: Great! Btw, am I mistaken, or is the performance now a bit better with VC?

 

@HMart: You were right on the money, it has something to do with fog lights. Setting r_skipFogLights to 1 "resolves" the problem. I'm looking into why this isn't working.

By the way, do you have the latest graphics drivers installed?

[stgatilov]

As part of my work on a VR port, I have been spending some time looking at the Doom3 BFG edition code for opportunities to improve performance. A core concept in BFG is a vastly different vertex cache, which I've been working on porting to TDM for a while. I believe it is a worthwhile change and will make porting additional improvements from BFG much easier in the future.

The VertexCache system of original Doom 3 is a mess.

It was noted at least here and here.

You can review the proposed changes here: https://github.com/fholger/thedarkmod/pull/3/files

The change is based on the BFG source code, but I refactored it because the original implementation had a lot of code duplication and functionality that we don't need.

Do you normally work on GitHub?

Probably I need to checkout it and look at some places...

 

From brief glance, it looks exactly as the VertexCache used in BFG code.

Which is great!

I hope that it would significantly reduce driver overhead on CPU side

• The new approach always allocates fixed size buffers on the GPU, which has to be sufficiently large to accomodate all maps and scenes. This means that, on average, GPU memory consumption is higher. At the same time, it also places a hard upper limit on the amount of static vertex data that can be used in maps, whereas the current implementation is more flexible. Changing the upper limit requires a code change.

 

Isn't the same true for Doom 3 BFG ?

It takes about 192MB of video RAM. I hope everyone has at least 256 MB today.

Can you provide any numbers, i.e. how much bytes are used currently on a large mission?

• I ported a function idInteraction::CreateStaticInteractions, which is supposed to offload static interactions into the static buffers to not have to (re-)generate them every frame when they are needed. It works in principle, however, when loading a savegame, these interactions are not working for some inexplicable reason. Therefore, this function is not currently called; performance-wise it didn't really make a difference, but I'm still investigating how I can get it to work with savegames.
• There is also some ported code that is supposed to create static shadow caches for models. This is commented out right now, because it just doesn't work.

I thought interaction is just a pair of entity + light pointers + few other data.

Are they so costly to generate each frame?

 

Shadow mesh generation can make a difference in theory.

But as far as I know, dmap already generates shadow meshes and stores them in .proc file (e.g. noted in this review).

What is the supposed additional benefit of static shadow cache generation over the precomputed shadow geometry we have now?

 

 

[lowenz]

@lowenz: Great! Btw, am I mistaken, or is the performance now a bit better with VC?

Yes but I can't use the same savepoint, so the comparison process is a bit erratic.

Edited December 29, 2017 by lowenz

[nbohr1more]

Lightgem works with the interleave options again.

No crashing with any normal gameplay I've tested.

 

I have noted a performance boost in the pathological scene from Rightful Property. (47FPS vs 40FPS).

 

Bug report: Even with the lg args at safe values, the lights in front of the Bank in Rightful Property

are producing shadow triangle artifacts.

[stgatilov]

Yes but I can't use the same savepoint, so the comparison process is a bit erratic.

Most likely you can use tdm_force_savegame_load to load savegame from different revision.

 

If loading a save from different version crashes after this, then two versions have different savegame format.

But I do not recall any savegame format changes recently, so maybe it will work.

[cabalistic]

Alright, I think I found the fog light bug. @nbohr1more, perhaps this also solves your shadow triangles? If not, it might be a very similar problem, so just let me know and I'll look again

New test build here: https://github.com/fholger/thedarkmod/releases/download/cache_v3/TheDarkMod_VertexCache_v3.zip

@HMart: I switched the buffer handling again, let me know if you have any issues with it and how the performance is compared to beta build.

 

@stgatilov: I don't fully understand interactions, either, but they do generate vertex and index data. Possibly the culled versions? In any case, disabling the static interactions did not make any noticable performance difference, so you might be right. But in theory, anything that doesn't have to constantly be regenerated would lessen the load on the CPU->GPU memory pipeline. As for the shadows, I really don't know. I basically just tried to port what BFG did, and it didn't quite work, so I left that part out. Perhaps it is indeed unneeded

[lowenz]

@lowenz: Great! Btw, am I mistaken, or is the performance now a bit better with VC?

It's better, tested the lastest iteration (3) and a solid boost is confirmed.

Edited December 29, 2017 by lowenz

[nbohr1more]

It looked like the shadow problem was gone but when I just checked my cvar settings

without changing them, I got this console spam:

 

RB_T_FillDepthBuffer: !tri->ambientCache
RB_T_FillDepthBuffer: !tri->ambientCache
RB_T_FillDepthBuffer: !tri->ambientCache
RB_T_RenderShaderPasses: !tri->ambientCache
RB_T_RenderShaderPasses: !tri->ambientCache
RB_T_RenderShaderPasses: !tri->ambientCache
RB_T_RenderShaderPasses: !tri->ambientCache

then the problem returned.

 

I'm guessing this might be due some sync breakage when in console and invoking render cvars?

(I recall all the diagnostic cvars had warnings about SMP in them.)

[HMart]

@lowenz: Great! Btw, am I mistaken, or is the performance now a bit better with VC?

 

@HMart: You were right on the money, it has something to do with fog lights. Setting r_skipFogLights to 1 "resolves" the problem. I'm looking into why this isn't working.

By the way, do you have the latest graphics drivers installed?

 

Yes i do.

 

 

Version 3

 

Menu works fine, fog seems to work fine, performance is much better than the vanilla 2.06 Beta, where i add 30 fps's before now i got above 40 fps's with this latest version, nice work.

Edited December 29, 2017 by HMart

[cabalistic]

@nbohr1more: Hm, might be a similar problem to the fog issue. I'll look through the code again

@HMart: Thanks for testing. Great that it's now working and giving the performance boost that I hoped for

Edited December 29, 2017 by cabalistic

[cabalistic]

@nbohr1more: Found a few more instances where caches were potentially not renewed consistently. It seems to have fix one instance of flickering triangles I could previously observe in Rightful Property. Please test again: https://github.com/fholger/thedarkmod/releases/download/cache_v4/TheDarkMod_VertexCache_v4.zip

 

As for those messages: they are probably fine and should recover in the next frame. So unless they are spammed consistently, I wouldn't pay them too much attention. I believe this is some border case when geometry is being loaded late or something like that.

[nbohr1more]

Seems to be cured now.

 

I encountered the messages but there were two lines rather than the spam I was seeing before.

 

No shadow tris corruption seen.

[cabalistic]

Excellent. Thanks for testing

[lowenz]

Any updates?

[cabalistic]

Nothing significant. I cleaned up a few things in code, but apart from that the latest executable is still state-of-the-art

 

Far as I can tell, it runs pretty stable now, so from my side it could be merged to SVN after 2.06 is released. Only thing I still need to check is underwater vision. I have a feeling it might be broken.

[duzenko]

Nothing significant. I cleaned up a few things in code, but apart from that the latest executable is still state-of-the-art

 

Far as I can tell, it runs pretty stable now, so from my side it could be merged to SVN after 2.06 is released. Only thing I still need to check is underwater vision. I have a feeling it might be broken.

Tremendous achievement! Can't wait for the merge

[cabalistic]

Now that 2.06 is basically done and in feature freeze, I would like to get this merged to svn trunk Any objections?

[nbohr1more]

Since SVN is already divergent enough that you need to backport any fixes to 2.06 release, I don't see any logical reason for any objection.

 

1) As stated above Trunk is already ahead of 2.06 to the point we must back-port fixes

2) None of these changes are Gameplay related

3) The last BFG executable was perfectly stable

4) We are in release candidate phase

5) SVN already has broken Lightgem in FBO mode

 

@grayman... ?

[grayman]

You can commit working changes to SVN, but I won't merge them to 2.06 at this late stage.

 

It's not recommended to commit non-working changes to SVN because a lot of people work/run from SVN, and it's not fair to them to screw it up.

[cabalistic]

Of course, that's why this was postponed after 2.06, anyway, and why I had testers give this a spin.

 

I'll merge it to trunk later, then.

[nbohr1more]

You can commit working changes to SVN

 

 

 

Approved. The 2.06 release branch is a different place. Eg. Go ahead (but fix the Lightgem please.)

[cabalistic]

The lightgem on this branch *does* work, even better than before, because it's integrated into the frontend/backend split. It's the fbo/MSAA part that's not working with it And yeah, I'll fix that.