Render bug, large black box occluding screen

[nbohr1more]

@stgatilov I am guessing you won't like this option, but how about passing a GLSL variable that declares

the video card vendor ( u_vendor ? ) so we can do things like this in the shader :

if ( u_vendor == AMD && ( isnan(totalColor.r) || isnan(totalColor.g) || isnan(totalColor.b) )

?

Might come in handy for other driver bugs?

[stgatilov]

18 minutes ago, nbohr1more said:

@stgatilov I am guessing you won't like this option, but how about passing a GLSL variable that declares the video card vendor ( u_vendor ? ) so we can do things like this in the shader :

I don't think this case is worth introducing such a hack.
I suggested adding such flag for bindless textures, but I recall @cabalistic was strongly against, and preferred disabling bindless textures altogether instead. I bet he will be against this one too.

Also, I don't understand why you want so much to check for NaNs.
Right now this only happens for one user. If there are many of them and we can't find better solution, then yes: this mitigation would be better than doing nothing.
 

[stgatilov]

Things are so much simpler when you have access to the machine

Luckily, I managed to boot the watcher automation script (although it was rather buggy to be honest: like treating all reads as modifications) to have instant feedback for all shader changes. Well... as instant as it can be when computer and keyboard are on the opposite points of the planet

The results are far beyond what I expected:

1. Interaction shader receives NaNs in params[attr_DrawId].colorModulate. The problem can be mitigated by checking its components with isnan and assigning some constant color if any is NaN. To be strict, I can't swear NaNs don't come from other sources too (forgot to check), but they surely come from this source at least.
2. The problem does not happen on the old backend, although @AluminumHaste will do a bit more testing on it.

I'd like to have one more quick session in order to rule out the unlikely possibility that we supply NaNs from C++ code.
And if we don't, then the problem happens somewhere in the "new backend" + "AMD driver" combo.

Also, I wonder if the problem happens on 2.09.
Because 2.09 also has new backend enabled by default. If the problem does not happen there on the same machine, then it means we broke something in the new backend after 2.09. Given that I don't recall @cabalistic working on it, and others did not modify it either, I wonder how we managed to do that

[AluminumHaste]

Yeah it does not happen in 2.10Beta6 on old backend at all.
 

2.09 New backend works fine, no glitches.

[HMart]

7 hours ago, nbohr1more said:

@stgatilov I am guessing you won't like this option, but how about passing a GLSL variable that declares

the video card vendor ( u_vendor ? ) so we can do things like this in the shader :

if ( u_vendor == AMD && ( isnan(totalColor.r) || isnan(totalColor.g) || isnan(totalColor.b) )

?

Might come in handy for other driver bugs?

Please don't disable stuff based only on GPU brand, if you can really avoid it, you are just punishing all AMD users, even those not having this particular problem, like me (unless I failed to see it somehow...). 

 

Edited February 8, 2022 by HMart

[nbohr1more]

11 minutes ago, HMart said:

Please don't disable stuff based only on GPU brand, if you can really avoid it, you are just punishing all AMD users, even those not having this particular problem, like me (unless I failed to see it somehow...). 

 

No, that wasn't the intention. I was suggesting that we do additional data sanitation based on vendor so if AMD has a driver bug it won't result in rendering errors due to invalid numeric data.

The only downside will be a small performance penalty.

And based on stgatilov's findings that might just be what we will do if nothing is wrong on the TDM C++ or GLSL side.

[OrbWeaver]

Adding an additional check in the shader for the GPU vendor is going to introduce a branch and a performance penalty of its own, which won't necessarily be any better than just doing the isnan check on all GPUs.

I don't think there should be any vendor checks within the shader itself. If there are critical driver bugs that cannot be worked around in a standard-compliant way, it would be better to have separate shaders for each vendor and choose these once, at application startup. That way we could have shaders optimised for each GPU type without any additional branches.

[stgatilov]

To be honest, I don't think isnan check will help.

Currently an affected player sees a big flickering spot, enlarged by bloom's blurring. If we check for isnan and replace the value with zero, then he will most likely see the whole object flickering with black. This is a color uniform which affects an entire surface, so if we replace it, then the whole surface of an object will look differently.

@AluminumHaste, I'm sorry for wasting your time, but maybe you can bisect by dev builds and see where the issue started happening?
UPDATE: Although if this is some backend/driver issue, then it can be some weakly related change, which does not cause the issue but only triggered it.

[nbohr1more]

43 minutes ago, stgatilov said:

To be honest, I don't think isnan check will help.

Currently an affected player sees a big flickering spot, enlarged by bloom's blurring. If we check for isnan and replace the value with zero, then he will most likely see the whole object flickering with black. This is a color uniform which affects an entire surface, so if we replace it, then the whole surface of an object will look differently.

I'm not sure I follow here. Aren't both the bloom and interaction common shaders additive? If you replace a NaN with zero you will simply prevent any added illumination no?

If the value is supposed to max out the bloom or interaction, then you would replace it with a 1 or if there is no way to reasonably know what value it should have you could just assume some minimum bloom amount to add like 0.05 ?

[stgatilov]

5 minutes ago, nbohr1more said:

I'm not sure I follow here. Aren't both the bloom and interaction common shaders additive? If you replace a NaN with zero you will simply prevent any added illumination no?

The problematic uniform is the color of the object.

Set it to NaN, and the whole object fills with NaN, bloom blurs it over larger area.
Set it to zero, and the whole object turns black, bloom does not do anything with it.

If the issue happened to the same objects every frame, then users would not notice they have e.g. black health potions. But the issue happens on random frames, so user will see red potion flickering black.

 

[nbohr1more]

35 minutes ago, stgatilov said:

The problematic uniform is the color of the object.

Set it to NaN, and the whole object fills with NaN, bloom blurs it over larger area.
Set it to zero, and the whole object turns black, bloom does not do anything with it.

If the issue happened to the same objects every frame, then users would not notice they have e.g. black health potions. But the issue happens on random frames, so user will see red potion flickering black.

 

As I recall, one workaround for the error in this thread was to disable bloom.

So even if the object color data is incorrect at this phase of the shader pipeline, previous phases must have the correct data or else disabling bloom would not work as a workaround. Thus ignoring the bad color data should prevent changes to the underlying texture no? ( should be equivalent to disabling bloom )?

If disabling bloom still acts as workaround that might be a place to focus on for bisecting as well.

@AluminumHaste ?

[AluminumHaste]

Sorry not working today so not at home. Will hopefully be on tonight so I'll try.

[nbohr1more]

3 hours ago, AluminumHaste said:

Sorry not working today so not at home. Will hopefully be on tonight so I'll try.

Sure. When you are able to test, please disable bloom and see if the artifact goes away. If there are other artifacts without Bloom then killing the error on the shader side might be fruitless.

[AluminumHaste]

We already determined that turning bloom off gets rid of the issue.

[AluminumHaste]

On 5/2/2021 at 3:03 PM, OrbWeaver said:

I don't know if it's the same issue or an unrelated one, but I just got this weird artifact after entering the basement sewer area of Blackgrove Manor. It's red rather than black, but does appear a similar shape. It started on the left and sort of spread across the screen towards the right.

UPDATE: Further experimentation revealed that Bloom is the cause of this particular artifact. If I disable bloom, the red boxes disappear. I then noticed some much smaller patches of the same red colour which were not affected by the bloom settings; these disappeared when I switched color depth to 16 bits rather than 64 bits.

OrbWeaver reported this last year that it was bloom. Not being a dick here sorry, just making sure I'm not mis-remembering.

[AluminumHaste]

Ugh, now I don't know what to think. I started by checking out a new directory with SVN at Jan 2021. Loaded up black mage and it worked, so went to Feb 2021 etc.

Everything was going fine until I got to May. The Black Mage crashes on level start. Went through middle of month, then a little further etc. Then skipped to June 1st 2021, and that also crashes on map load.

Deleted GLprogs and darkmod.cfg, re-checked out that revision on June 1st and still crashes. Now I don't know if there's something up with that directory now? Or if there's something F'ed about those builds back in May/June?

Out of time for now, just more stuff to get in the way of testing.

 

SVN build Jan 1st 2021 is GOOD
SVN build Feb 1st 2021 is GOOD
SVN build Mar 1st 2021 is GOOD
SVN build Apr 2nd 2021 is GOOD
SVN build May 2nd 2021 is BAD - CRASHES The Black Mage
SVN build June 1st 2021 id BAD - Crashes loading the black mage

 

[OrbWeaver]

My interpretation of what I saw, combined with the discussion in this thread, is that Bloom is amplifying a particular problem, spreading invalid colours across the screen in large boxes, but Bloom itself is not the cause of the original invalid values. The invalid colours disappeared when I switched color depth from 64 to 16.

Assuming that "64 bit" means "double float" and "16 bit" means "integer", this would be consistent with the appearance of NaNs in some part of the rendering backend. Integer arithmetic does not have NaNs, which would explain why the problem went away.

[stgatilov]

14 hours ago, AluminumHaste said:

We already determined that turning bloom off gets rid of the issue.

Which issue?
Are you sure you paid attention to color of the flickering objects?
Or you only checked whether you see these large spots sticking over the whole screen?

13 hours ago, AluminumHaste said:

Everything was going fine until I got to May. The Black Mage crashes on level start. Went through middle of month, then a little further etc. Then skipped to June 1st 2021, and that also crashes on map load.

Did you reset your config after every checkout?
Did you rebuild engine from source code?
Bisecting with tdm_installer is not prone to such things, that's why I usually start by suggest it.

Also, our trunk is not 100% reliable, sometimes it gets broken temporarily.
 

[AluminumHaste]

I'll start over

[AluminumHaste]

@stgatilov

Ok, wanted to double check what I'm looking for and you're right.

In 2.10 Beta 6, I get flickering textures with ONLY 64bit color. With Bloom on, I also get the colored boxes around the flickering textures.

 

[nbohr1more]

Sounds like some part of our pipeline is causing the driver to implicitly convert between float and integer. I don't recall the use of swizzle or any bitwise operations in our GLSL? The fact that it flickers in a periodic fashion sounds like it only happens on an interleave cycle like the lightgem process.

[AluminumHaste]

You did mention try longer interleave on lightgem, I'll go check.

So yes, if I set tdm_lg_interleave to 0, it doesn't flicker, it just stays there. Setting the interleave to higher and higher values seems to cause the flicker at the same rate, I can't discern a difference. Even setting com_maxFPS to 1 and changing interleave rate from 1 to 10, I still get 1 flash per frame.

[nbohr1more]

Or turning off the lightgem tdm_lg_interleave 0

 

[AluminumHaste]

Nope, that doesn't fix it, still flickers but in different spots?

This shit don't make sense N

 

[nbohr1more]

That is pretty strange XD

Hmm...

I wonder if the thread scheduler for r_useParallelAddModels is where this cycle is happening?

Let's see what it looks like if you set:

tdm_lg_interleave 0

com_smp 0

r_useParallelAddModels 0