Jump to content


Photo

Testers and reviewers wanted: BFG-style vertex cache


  • Please log in to reply
43 replies to this topic

#1 cabalistic

cabalistic

    Member

  • Member
  • PipPip
  • 139 posts

Posted 26 December 2017 - 03:02 PM

*
POPULAR

As part of my work on a VR port, I have been spending some time looking at the Doom3 BFG edition code for opportunities to improve performance. A core concept in BFG is a vastly different vertex cache, which I've been working on porting to TDM for a while. I believe it is a worthwhile change and will make porting additional improvements from BFG much easier in the future.

I finally think my port is ready and would like to merge it, but given the scope of the change, I'm looking for:

  • testers to rule out any unforeseen problems with the change and
  • reviewers from the team to actually approve or disapprove the change.

Help with testing

 

For the testers, I've prepared a 64bit Windows test build which can be downloaded here: https://github.com/f...texCache_v3.zip

It requires 2.06 beta (or trunk) assets to run and should be compared to the current beta build. I'm specifically looking for:

  • problems of any kind (particularly rendering artefacts or issues with custom maps) that are not present on the current beta build
  • performance changes. From my own testing on a couple of different machines and a few select scenes, I found that performance stays roughly the same with a minor framerate improvement here or there. However, duzenko reported a loss of performance, so I'm looking for a wider assessment of whether this change hurts or helps.

Help with reviewing the changes

 

You can review the proposed changes here: https://github.com/f...od/pull/3/files

The change is based on the BFG source code, but I refactored it because the original implementation had a lot of code duplication and functionality that we don't need.

 

My motivations for changing the vertex cache is as follows:

  • It allows us to get rid of the secondary shared GL context for the frontend. Less syncing overhead that way.
  • It allows for further parallelization of the frontend. I've already experimented with this a bit, and I can indeed cut off more frontend drawing time, although the effects are currently limited in most cases because the backend then becomes the bottleneck.
  • My findings suggest a minor performance improvement from the new implementation
  • Although not a strict requirement, this change should make it easier to port further improvements from BFG, in particular GPU skinning, which would reduce the amount of vertex cache needed and thus further reduces unnecessary memory syncing between CPU and GPU.

There is, however, a potential drawback:

  • The new approach always allocates fixed size buffers on the GPU, which has to be sufficiently large to accomodate all maps and scenes. This means that, on average, GPU memory consumption is higher. At the same time, it also places a hard upper limit on the amount of static vertex data that can be used in maps, whereas the current implementation is more flexible. Changing the upper limit requires a code change.

Finally, two open points in the review:

  • I ported a function idInteraction::CreateStaticInteractions, which is supposed to offload static interactions into the static buffers to not have to (re-)generate them every frame when they are needed. It works in principle, however, when loading a savegame, these interactions are not working for some inexplicable reason. Therefore, this function is not currently called; performance-wise it didn't really make a difference, but I'm still investigating how I can get it to work with savegames.
  • There is also some ported code that is supposed to create static shadow caches for models. This is commented out right now, because it just doesn't work.

In any case, I'm looking for team member opinions on whether we want this change or not.

 

Thanks to everyone for helping out!


Edited by cabalistic, 29 December 2017 - 01:07 PM.

  • AluminumHaste, Judith, stgatilov and 3 others like this

#2 grayman

grayman

    Master Builder, Coder

  • Active Developer
  • PipPipPipPipPip
  • 12156 posts

Posted 26 December 2017 - 03:12 PM

If this is indeed a large block of changes, I would appreciate it if you didn't merge to SVN until after 2.06 ships. Any non-2.06 change to SVN has the potential of hampering subsequent 2.06 changes. A large block even more so.

Thanks.


  • AluminumHaste and Anderson like this

#3 lowenz

lowenz

    Advanced Member

  • Member
  • PipPipPip
  • 1696 posts

Posted 26 December 2017 - 03:18 PM

What are the differences of the included ExtLibsx64.dll ?


Task is not so much to see what no one has yet seen but to think what nobody has yet thought about that which everybody see. - E.S.


#4 cabalistic

cabalistic

    Member

  • Member
  • PipPip
  • 139 posts

Posted 26 December 2017 - 03:20 PM

@grayman: Fair enough. This is certainly aimed post-2.06.

@lowenz: Probably none. I included it just to be safe.


  • Anderson likes this

#5 lowenz

lowenz

    Advanced Member

  • Member
  • PipPipPip
  • 1696 posts

Posted 26 December 2017 - 03:44 PM

Some quirks with the lightgem (using the same config file of 2.06beta, maybe I've changed something about it) as you can see in the screenshots below.

Performance is slightly worse compared to 2.06b x64 exe (5%) but CPU utilisation is lower! (memory higher :P but I think it's precisely  the point of the vertex cache tech)

 

The_Dark_Modx64_2017_12_26_21_28_46_220.
 
The_Dark_Modx64_2017_12_26_21_31_47_949.

Edited by lowenz, 26 December 2017 - 03:55 PM.

  • Anderson likes this

Task is not so much to see what no one has yet seen but to think what nobody has yet thought about that which everybody see. - E.S.


#6 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8214 posts

Posted 26 December 2017 - 03:48 PM

IMHO the biggest reason for such a change is to align the code with BFG more closely so
that other BFG code features can more easily ported (GPU skinning...).

Bravo on the progress. I'll test when I get a chance. :)
  • Anderson likes this
Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#7 cabalistic

cabalistic

    Member

  • Member
  • PipPip
  • 139 posts

Posted 26 December 2017 - 03:54 PM

Thanks for the report lowenz! I'll take a closer look into the lightgem. The lightgem is a little awkward because it renders between each frame and thus "violates" the frontend/backend split. So that might have some unintended consequences here.

Shame that the performance dropped, even if just a little. Just for the record, can you tell me your PC specs, and also roughly what graphics settings you are using?


  • Anderson likes this

#8 lowenz

lowenz

    Advanced Member

  • Member
  • PipPipPip
  • 1696 posts

Posted 26 December 2017 - 03:58 PM

Thanks for the report lowenz! I'll take a closer look into the lightgem. The lightgem is a little awkward because it renders between each frame and thus "violates" the frontend/backend split. So that might have some unintended consequences here.

Shame that the performance dropped, even if just a little. Just for the record, can you tell me your PC specs, and also roughly what graphics settings you are using?

Maxed settings @1920x1080

 

i3 6300

16 GB DDR4

GeForce 1050 Ti (a little bit overclocked)

Win 10 x64 FCU


Task is not so much to see what no one has yet seen but to think what nobody has yet thought about that which everybody see. - E.S.


#9 HMart

HMart

    Advanced Member

  • Member
  • PipPipPip
  • 590 posts

Posted 26 December 2017 - 06:49 PM

Ok tested this, first i most say that avast flagged the .exe has a virus and put it on quarantine, but i sure it was a false positive so i restored it, tested normal 2.06 Beta on Volta 2, then installed your modifications, after starting the game the menu text was flashing like crazy, appearing and disappearing, started Volta 2 again, strangely the menu behaved normal with the mission loaded, saw no problems that original 2.06 didn't already add, like the water bug, performance was virtually identical for both versions. More or less minimum 30 and max 60 fps's depending on direction. 

 

Max settings 1920x1080

 

i7 860 2.8GZ

8GB RAM DDR3

 

AMD R9 270x 2GB, latest drivers

 

Win10 64bits


Edited by HMart, 26 December 2017 - 06:49 PM.

  • Anderson likes this

#10 cabalistic

cabalistic

    Member

  • Member
  • PipPip
  • 139 posts

Posted 26 December 2017 - 07:51 PM

Thanks HMart. Is the menu flickering reproducible, i.e. does it happen on every start and go away after mission load?


  • Anderson and CarltonTroisi like this

#11 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8214 posts

Posted 26 December 2017 - 08:32 PM

None of the lightgem interleave args work.

 

tdm_lg_split 1 = no lightgem and graphic artifacts

 

tdm_lg_interleave > 1 = no lightgem and graphic artifacts


  • Anderson likes this
Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#12 cabalistic

cabalistic

    Member

  • Member
  • PipPip
  • 139 posts

Posted 27 December 2017 - 03:14 AM

@nbohr1more: What kind of graphic artifacts? Are they restricted to the lightgem? Do they go away if you disable lightgem rendering with tdm_lg_weak 1?

 

After thinking about it, the lightgems are probably broken. I'll try to refactor them to fit in the backend/frontend split.


  • Anderson likes this

#13 AluminumHaste

AluminumHaste

    Darkmod Contributor

  • Development Role
  • PipPipPipPipPip
  • 5964 posts

Posted 27 December 2017 - 09:06 AM

Will this test version work on my work computer?
 

Spoiler

I always assumed I'd taste like boot leather.

 

#14 cabalistic

cabalistic

    Member

  • Member
  • PipPip
  • 139 posts

Posted 27 December 2017 - 09:25 AM

See no reason why it shouldn't. If the 2.06 beta works...


  • Anderson likes this

#15 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8214 posts

Posted 27 December 2017 - 12:31 PM

@nbohr1more: What kind of graphic artifacts? Are they restricted to the lightgem? Do they go away if you disable lightgem rendering with tdm_lg_weak 1?
 
After thinking about it, the lightgems are probably broken. I'll try to refactor them to fit in the backend/frontend split.


I'll test lg_weak. The artifact is random shadow triangles. (Resembles the older lightgem bug around this functionality.)
Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#16 HMart

HMart

    Advanced Member

  • Member
  • PipPipPip
  • 590 posts

Posted 27 December 2017 - 06:58 PM

Thanks HMart. Is the menu flickering reproducible, i.e. does it happen on every start and go away after mission load?

 

Yes happens every time I start the game and goes away at (Volta 2) mission load (didn't tested others) if i quit the mission the flickering happens again. 


  • Anderson likes this

#17 duzenko

duzenko

    Advanced Member

  • Active Developer
  • PipPipPip
  • 951 posts

Posted 28 December 2017 - 08:42 AM

 

Yes happens every time I start the game and goes away at (Volta 2) mission load (didn't tested others) if i quit the mission the flickering happens again. 

Same here, on Intel



#18 cabalistic

cabalistic

    Member

  • Member
  • PipPip
  • 139 posts

Posted 28 December 2017 - 10:23 AM

That's super strange. Flickering is usually a sign that the buffers are written to while they are still rendering. But this really shouldn't be happening, and I've already made them more restrictive than Doom3 BFG is... I have a couple more ideas I can try, I will probably need you to test two specialized builds when I'm ready.


  • HMart and Anderson like this

#19 duzenko

duzenko

    Advanced Member

  • Active Developer
  • PipPipPip
  • 951 posts

Posted 28 December 2017 - 03:43 PM

That's super strange. Flickering is usually a sign that the buffers are written to while they are still rendering. But this really shouldn't be happening, and I've already made them more restrictive than Doom3 BFG is... I have a couple more ideas I can try, I will probably need you to test two specialized builds when I'm ready.

Try it on your integrated Intel, I think you should have it?



#20 cabalistic

cabalistic

    Member

  • Member
  • PipPip
  • 139 posts

Posted 28 December 2017 - 03:54 PM

No, my Intel is fine in this build :(

 

Unless this is coupled to some specific settings. Perhaps you could attach your cfg?



#21 duzenko

duzenko

    Advanced Member

  • Active Developer
  • PipPipPip
  • 951 posts

Posted 28 December 2017 - 04:14 PM

No, my Intel is fine in this build :(

 

Unless this is coupled to some specific settings. Perhaps you could attach your cfg?

Can't repeat now... :wacko:



#22 HMart

HMart

    Advanced Member

  • Member
  • PipPipPip
  • 590 posts

Posted 28 December 2017 - 04:26 PM

No, my Intel is fine in this build :(
 
Unless this is coupled to some specific settings. Perhaps you could attach your cfg?

 
My .cfg
 
Spoiler


#23 cabalistic

cabalistic

    Member

  • Member
  • PipPip
  • 139 posts

Posted 28 December 2017 - 04:47 PM

@HMart: Thanks. Unfortunately, I still can't reproduce it.

 

In any case, I've prepared a new release which should fix the lightgem. At least it looks good for me. I also slightly changed the way I update the buffers; perhaps it helps with the flickering. Please give it a try and let me know.

 

https://github.com/f...texCache_v2.zip


  • nbohr1more likes this

#24 HMart

HMart

    Advanced Member

  • Member
  • PipPipPip
  • 590 posts

Posted 28 December 2017 - 06:04 PM

That version solved the menu flickering on my end but brought performance down, never managed to go above 40 fps's and caused graphical glitch's in Volta 2, at the beginning I walked to the stairs and then looked back at the initial entrance and the sky was not behaving normally, some sort of z-fighting, seamed to be a problem with the fog or something.  



#25 cabalistic

cabalistic

    Member

  • Member
  • PipPip
  • 139 posts

Posted 28 December 2017 - 06:16 PM

Ah, I can reproduce the sky flickering, at least. Hm, I'll look into it. Thanks for spotting it...


  • HMart likes this




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users