AntiAliasing won't work

[MirceaKitsune]

I get the feeling antialiasing does not wish to work for me, even when I use the 4x or 16x option in the main menu. I remember getting the impression that it worked sometimes in the past, but yesterday I looked at the edges of some objects and could clearly see the jagged edges of the geometry.

I wonder if this is specific to the Linux version: I have an AMD card and use the free amdgpu driver... there was talk about Mesa having issues handling AA properly depending on how the engine calls it. What do you suggest I check?

[nbohr1more]

Does it work if you disable Sharpening?

[MirceaKitsune]

2 hours ago, nbohr1more said:

Does it work if you disable Sharpening?

Doesn't seem to. This is with sharpen disabled and 4x AA, settings applied at startup.

[nbohr1more]

I don't see aliasing there? Can you attach a PNG with close proximity to an aliased object without using the spyglass?

[stgatilov]

Perhaps start with looking at GL_MAX_SAMPLES  value in glxinfo?

UPDATE: Do you have such problem in other OpenGL applications?

[MirceaKitsune]

9 hours ago, stgatilov said:

Perhaps start with looking at GL_MAX_SAMPLES  value in glxinfo?

UPDATE: Do you have such problem in other OpenGL applications?

glxinfo doesn't appear to say anything about GL_MAX_SAMPLES, I attached the full output. The issue doesn't occur in Xonotic when I enable 4x AA, over there I can clearly see the filter smoothing the edges when on and the same effect as in TDM when off... this should thus be TDM specific. I attached a cropped jpg exported at 100% quality, should be the same as a png... need to ask the forum admins if they can raise the attachment limit to 1 MB per post, 500 KB is rather limiting.

glxinfo.txt

[stgatilov]

Maybe glxinfo -l ?

UPDATE: Please post full output of glxinfo -l, it should have GL_MAX_SAMPLES and other constants.

[OrbWeaver]

I've never seen in-game selection of MSAA samples work on Linux. I guess it's a driver issue.

The super-sampling AA (r_fboResolution > 1) works fine, although it's heavy on performance.

[MirceaKitsune]

6 hours ago, stgatilov said:

Maybe glxinfo -l ?

UPDATE: Please post full output of glxinfo -l, it should have GL_MAX_SAMPLES and other constants.

glxinfo -l.txt

The value seems to be 8. This would explain why in the past setting it to 16x disabled the feature while using 4x seemed to work. Problem is that now even 4x is no longer working, none of the settings will allow AA to work.

4 hours ago, OrbWeaver said:

I've never seen in-game selection of MSAA samples work on Linux. I guess it's a driver issue.

The super-sampling AA (r_fboResolution > 1) works fine, although it's heavy on performance.

I can imagine... that would be like rendering at an increased resolution right? Perhaps that's why AA worked in the past for me? I think I might have had that enabled at some point... I had to disable it because it caused the view to shrink and show corruption when going underwater (some sort of shader issue).

Like I said Xonotic and others have AA working for me. But I heard the free Linux video drivers are iffy when it comes to AntiAliasing, so perhaps the engine properly needs to tell Mesa how to set it? There are a few Linux specific bugs left, such as the TDM window stealing the mouse pointer when alt-tab switching to another window in front. Luckily no more bad ones at least, I had to struggle to get sound working back when there was no 64bit engine.

[MirceaKitsune]

Just tried r_fboResolution. Sadly it's not an option: It works but the performance loss is huge! Even at 1.5 which would be the right amount to get some decent AA, the FPS loss is gigantic. Unless another form of AA is implemented in the engine, my only options are to either get the normal one working or leave it like this which is ugly. There must be other Linux users with the free drivers though, is there no knowledge on how this could be solved?

[nbohr1more]

I was unable to get AA working until I changed my Window manager to "Marco + Compton" under Preferences > Desktop Settings > Windows.

Linux composite managers have varying degree of reliability with respecting AA.

Are you able to use "Marco + Compton" or a non-composite desktop option?

[MirceaKitsune]

8 minutes ago, nbohr1more said:

I was unable to get AA working until I changed my Window manager to "Marco + Compton" under Preferences > Desktop Settings > Windows.

Linux composite managers have varying degree of reliability with respecting AA.

Are you able to use "Marco + Compton" or a non-composite desktop option?

Never experimented with this I'm afraid, don't think there is one. Also my desktop is KDE... I run TDM on openSUSE Tumbleweed x64 (X11 not Wayland).

Edited December 18, 2020 by MirceaKitsune

[stgatilov]

There are two ways of using multisampling in OpenGL.

The "older" one is to ask OS for multisampled default framebuffer. It must be done during OpenGL initialization, and can surely depend on OS and window manager, and uses OS-specific functions and settings.

The "newer" one is to ask OpenGL for multisampled off-screen framebuffer. This framebuffer is newer visible directly. It is completely unrelated to OS: a pure OpenGL call. And it has nothing to do with windows. That's what TDM uses since 2.07 at least.

It is strange if multisampling in TDM depends on windows manager. It should be more of a driver issue.

[MirceaKitsune]

9 hours ago, stgatilov said:

There are two ways of using multisampling in OpenGL.

The "older" one is to ask OS for multisampled default framebuffer. It must be done during OpenGL initialization, and can surely depend on OS and window manager, and uses OS-specific functions and settings.

The "newer" one is to ask OpenGL for multisampled off-screen framebuffer. This framebuffer is newer visible directly. It is completely unrelated to OS: a pure OpenGL call. And it has nothing to do with windows. That's what TDM uses since 2.07 at least.

It is strange if multisampling in TDM depends on windows manager. It should be more of a driver issue.

If it's the later I definitely shouldn't have any issues then. Perhaps something is missing... I suspected an OS / driver related limitation, but maybe this is a bug in the engine? Especially since again, I remember 4x AA used to work months ago... I couldn't stick my hand in fire over it but I remember looking at geometry edges at the time and seeing the smoothness clearly. With TDM 2.09 being in Beta it might be nice if we could clarify this so it can be fixed in the final release if so.

[cabalistic]

I think it might actually be universally broken. MSAA values are clamped by glConfig.maxSamples, however, in the current code it does not appear to be initialised anywhere. This may have been lost in some refactoring...

[cabalistic]

Well, it was in fact not universally broken, because on Windows maxSamples was set at an obscure place in platform-specific code. A fix is on trunk, should make its way into the next beta.

[nbohr1more]

Woo! I just reduced my internal resolution to 65% and applied 4x MSAA and there are almost no crawling artifacts now!

Edit:

Hmm... Trouble in paradise. Restarting TDM causes any fbo resolution change to shrink the percentage of rendered screen area

if MSAA is enabled. I will try other window managers to see if this problem persists.

[nbohr1more]

Yes, the problem happens no matter the Window manager or DPI settings.

Strangely, you can temporarily get it working with a fresh start of a small mission like Closemouthed Shadows but after about a minute of

roaming the mission the viewport will occupy the bottom left corner of the screen at a percentage matching the resolution scale percent...

[cabalistic]

That's because you set g_enablePortalSky to 1, correct? With 2, I cannot reproduce the issue. Unfortunately, 1 overwrites the currentRender image and messes up the resolution, which it really shouldn't.

[cabalistic]

Actually, this has nothing to do with anti-aliasing. g_enablePortalSky 1 is just fundamentally broken with r_fboResolution changes. And I'm not sure what the proper fix is, as I don't really know how that particular method works. It should probably use a different texture than currentRender.

[nbohr1more]

Thanks for the clarifications!

As I understand it, only NHAT mission 1 needs this setting so perhaps I can get one of the mission contributors to fix the portals in that mission instead.

[nbohr1more]

On 12/23/2020 at 6:12 AM, cabalistic said:

Actually, this has nothing to do with anti-aliasing. g_enablePortalSky 1 is just fundamentally broken with r_fboResolution changes. And I'm not sure what the proper fix is, as I don't really know how that particular method works. It should probably use a different texture than currentRender.

Might need to dive into this deeper. Reflective surfaces are now rendering parts of the mission splash screens

[cabalistic]

You did restart TDM after changing, right? There are some lingering effects otherwise.

[nbohr1more]

26 minutes ago, cabalistic said:

You did restart TDM after changing, right? There are some lingering effects otherwise.

Yes. The Animated Grass demo is the quickest way to see it but you will also see it in puddles in missions like Penny Dreadful 3.

Disabling AA or setting renderscale to 1 fixes it.

[nbohr1more]

Hmm...

Maybe we need to multiply these by r_fboResolution:

// setup global fixup projection vars
		if ( 1 ) {
			int vidWidth, vidHeight;
			idVec2 shiftScale;

			renderSystem->GetGLSettings( vidWidth, vidHeight );

			float pot;
			int	 w = vidWidth;
			pot = w;// MakePowerOfTwo( w );
			shiftScale.x = (float)w / pot;

			int	 h = vidHeight;
			pot = h;// MakePowerOfTwo( h );
			shiftScale.y = (float)h / pot;

			hackedView.shaderParms[6] = shiftScale.x; // grayman #3108 - neuro used [4], we use [6]
			hackedView.shaderParms[7] = shiftScale.y; // grayman #3108 - neuro used [5], we use [7]
		}

Edit: nope