Vertex blending not working with bump maps

[grodenglaive]

The diffuse maps works as expected, but the shader just ignores vertexColor (and inverseVertexColor) for the bump maps and they still get applied across the whole model.

For example: in this material, the grass bump appears everywhere so my dirt path has a grass bump pattern visible in it, which does not look good.

textures/darkmod/map_specific/grass4_dirt_01_blend
{
	qer_editorimage textures/darkmod/nature/grass/grass4_ed
	surftype15
	description "grass"

	{
		blend diffusemap
		map textures/darkmod/nature/grass/grass4
		vertexColor
	}
	{
		blend bumpmap
		map textures/darkmod/nature/grass/grass4_local
		vertexColor
	}
	{
		blend diffusemap
		map textures/darkmod/nature/dirt/dry_earth_muddy
		inverseVertexColor
	}
	{
		blend bumpmap
		map textures/darkmod/nature/dirt/dry_earth_muddy_local
		inverseVertexColor
	}
}

I've tried other textures with the same result. Note that this shader was made using the built in material editor in DR 3.8.

[grodenglaive]

On 7/28/2023 at 4:12 PM, grodenglaive said:

 

here's an example with grass and cobblestones with vertex blending. Seems like a bug to me.

[OrbWeaver]

I can confirm this, but it's a game issue not a DarkRadiant issue. Actually it's both a game issue and a DarkRadiant issue — DR renders the same thing (presumably because it's using shaders ported from the game code).

Vertex blending simply doesn't work with bump maps. One of the bump maps gets applied across the whole model, ignoring the vertex blend. This used to work fine in vanilla D3 but I assume it got broken by one of the many shader changes in TDM.

Example material:

textures/test/red_blue_vcol
{
   {
      blend diffusemap
      map textures/test/flatred
      vertexColor
   }
   {
      blend bumpmap
      map textures/test/spheres_local
      vertexColor
   }
   {
      blend diffusemap
      map textures/test/flatblue
      inverseVertexColor
   }
   {
      blend bumpmap
      map textures/test/square_pyramids_local
      inverseVertexColor
   }
}

Result:

Notice how only the "square pyramids" normal map is appearing. The "spheres" normal map is not visible at all, although the diffuse maps (which are solid red and blue) are being blended correctly. It seems that the last normal map completely replaces everything else.

[grodenglaive]

It's unfortunate. Hopefully someone can fix that. Blending terrain materials to export from Blender is a snap and looks really smooth. The alternative of using the edge (alpha) patches in DarkRadiant is tedious and doesn't look as good.

Edited May 26 by grodenglaive

[nbohr1more]

@OrbWeaver @grodenglaive @stgatilov

https://bugs.thedarkmod.com/view.php?id=5718

Biker was able to get it working by removing the vertexColor args from the bump stages. Perhaps we implicitly pair all diffuse stages with the previous bump stage now?

[stgatilov]

24 minutes ago, nbohr1more said:

@OrbWeaver @grodenglaive @stgatilov

https://bugs.thedarkmod.com/view.php?id=5718

Biker was able to get it working by removing the vertexColor args from the bump stages. Perhaps we implicitly pair all diffuse stages with the previous bump stage now?

I see there is test map in the ticket, I'll try to have a look.

Unlike "ambient stages" which are processed as they are written, interaction stages don't work as stages. The backend C++ code heavily reworks them. I guess some keywords are not supported at all, and some are occasionally broken...

[nbohr1more]

Now that I think of it, Biker's syntax was fated to work anyway.

When you specify a bump stage it needs a diffuse input because we don't offer any true flat-shaded lit modes. If the new diffuse stage drops out to black due to vertex color, then the new bump stage does so as well.

Not sure why adding vertexColor args is somehow causing the first bump stage to persist throughout the whole material render though.

[OrbWeaver]

1 hour ago, nbohr1more said:

@OrbWeaver @grodenglaive @stgatilov

Biker was able to get it working by removing the vertexColor args from the bump stages. Perhaps we implicitly pair all diffuse stages with the previous bump stage now?

I tried that. Unfortunately, removing [inverse]vertexColor from one or both of the bump stages makes no difference.

1 hour ago, stgatilov said:

Unlike "ambient stages" which are processed as they are written, interaction stages don't work as stages. The backend C++ code heavily reworks them.

Right, we do the same in DR — read them one by one, then assemble them into "triplets" of diffuse/bump/specular maps so the shader can process all three at once. My guess is that the vertex colours are not being passed through to the part of the shader which deals with the normal mapping, so the last normal map is being treated as the one and only normal map.

Quote

I see there is test map in the ticket, I'll try to have a look.

If there are no objections, I can commit my red/blue example into the maps/test/dr directory. The 256x256 normal maps are only 87K and being RGTC/BC5 they also exercise another part of the shader which can easily get broken (at least in DR).

[OrbWeaver]

I identified the issue in DarkRadiant, which may or may not be similar to the main engine renderer. However, the cause is a couple of different aspects of the code whose function I don't fully understand, so I'm cautious about making a fix.

First, although we parse shader layers in order, we then sort them so that bump maps appear before diffusemaps. I don't know the reason for this sorting, but it is obviously intentional:

    // Sort interaction stages: bumps go first, then diffuses, speculars last
    std::sort(
        interactionLayers.begin(), interactionLayers.end(),
        [](const IShaderLayer::Ptr& a, const IShaderLayer::Ptr& b) {
            // Use the enum value to sort stages
            return static_cast<int>(a->getType()) < static_cast<int>(b->getType());
        }
    );

For my red/blue test example, this results in a sequence of material stages like this:

OpenGLShader: sorted list:
+ textures/test/spheres_local
+ textures/test/square_pyramids_local
+ textures/test/flatred
+ textures/test/flatblue

Notice at this point, we have completely lost all information about which bump maps go with which diffusemaps.

At render time (which is done by the light class, since like the engine we now render light-by-light so that shadows etc will work), we iterate over the sorted list of stages and try to assemble them into D/B/S triplets.

switch (interactionStage.stage->getType())
{
case IShaderLayer::BUMP:
    if (draw.hasBump())
    {
        draw.submit(objects); // submit pending draws when changing bump maps
    }
    draw.setBump(&interactionStage);
    break;
case IShaderLayer::DIFFUSE:
    if (draw.hasDiffuse())
    {
        draw.submit(objects); // submit pending draws when changing diffuse maps
    }
    draw.setDiffuse(&interactionStage);
    break;
case IShaderLayer::SPECULAR:
    if (draw.hasSpecular())
    {
        draw.submit(objects); // submit pending draws when changing specular maps
    }
    draw.setSpecular(&interactionStage);
    break;
default:
    throw std::logic_error("Non-interaction stage encountered in interaction pass");
}

This logic seems largely correct: look for a stage of each type to build up a triplet, but submit an incomplete triplet if we see the same stage type again. The problem here is that the draw.submit() method does not clear out the material stages, so we see them again on the next iteration, and this includes the default stages which are used for an incomplete triplet (black for diffuse/specular, "flat" for bumpmap).

So the triplets we build and submit/render go as follows:

1. See the first bumpmap spheres_local and add it to the empty triplet.
2. See the next bumpmap square_pyramids_local, which triggers a render because we already have a bumpmap in the triplet.
3. RENDER the first bumpmap spheres_local with black diffuse and specular (pointless, it's just a black render).
4. Add the second bumpmap square_pyramids_local to our triplet.
5. See the first diffusemap flatred and try to add it to the triplet, which triggers a render because we already have a diffusemap: the default black diffusemap from step 3!
6. RENDER the second bumpmap square_pyramids_local with black diffuse and specular (another pointless draw call).
7. Add the first diffusemap flatred to our triplet.
8. See the second diffusemap flatblue, which triggers a render because we already have the flatred diffusemap from step 7.
9. RENDER the flatred diffusemap with the existing square_pyramids_local bumpmap (which is the wrong bumpmap for this diffusemap according to the material, because we lost this info with the sorting).
10. Add the second diffusemap flatblue to our triplet.
11. No more layers, so submit the final triplet: flatblue with the square_pyramids_local bumpmap.

Aside from being 4 draw calls when we should have only two, this also gives rise to the observed problem: only square_pyamids_local is ever actually seen as a bumpmap. The spheres_local bumpmap is rendered, but we never see it because it was rendered with a black diffusemap.

So in order to fix this and get the correct result, I need to change two things:

1. Remove the sorting, and preserve the order of material stages defined in the material file.
2. Make sure each submit() call in the rendering code clears out the triplet and leaves it empty for the next loop iteration.

But I don't know if either of these are going to be correct in all situations. The outstanding questions are:

• What is the motivation for sorting material stages by stage type, bumpmaps before diffusemaps? Can this goal be achieved without discarding information about which bumpmaps are associated with which diffusemaps (and specularmaps)?
• Is it correct to clear out a D/B/S triplet after submitting it for rendering, or are there situations where we actually want the images from the previous render? For example, are there material situations where we might want to render diffusemap A with bumpmap B, then another render with bumpmap C which re-uses diffusemap A without it being listed again in the material file?

[stgatilov]

You might as well ask: what is the motivation to specify interaction properties as 3 seemingly independent stages?
I think the answer is that when Doom 3 ran on shader-less platforms (hello from 2001), each stages was rendered separately and blended into total result. Perhaps it was important bumpmap stage was rendered first.
Today the stages are always combined into diffuse+specular+bumpmap packets.

The sorting logic in game is much more convoluted:

Spoiler

/*
===============
idMaterial::SortInteractionStages

The renderer expects bump, then diffuse, then specular
There can be multiple bump maps, followed by additional
diffuse and specular stages, which allows cross-faded bump mapping.

Ambient stages can be interspersed anywhere, but they are
ignored during interactions, and all the interaction
stages are ignored during ambient drawing.
===============
*/
void idMaterial::SortInteractionStages() {
	int		j;

	for ( int i = 0 ; i < numStages ; i = j ) {
		// find the next bump map
		for ( j = i + 1 ; j < numStages ; j++ ) {
			if ( pd->parseStages[j].lighting == SL_BUMP ) {
				// if the very first stage wasn't a bumpmap,
				// this bumpmap is part of the first group
				if ( pd->parseStages[i].lighting != SL_BUMP ) {
					continue;
				}
				break;
			}
		}

		// bubble sort everything bump / diffuse / specular
		for ( int l = 1 ; l < j-i ; l++ ) {
			for ( int k = i ; k < j-l ; k++ ) {
				if ( pd->parseStages[k].lighting > pd->parseStages[k+1].lighting ) {
					shaderStage_t	temp;

					temp = pd->parseStages[k];
					pd->parseStages[k] = pd->parseStages[k+1];
					pd->parseStages[k+1] = temp;
				}
			}
		}
	}
}

 

It splits all stages into groups, such that each group (except maybe the first one) starts with bump.
Then every group is sorted in order: ambient, bump, diffuse, specular.
It is not important where ambient stages are.

So according to what I see, the engine does not support setting bump/diffuse/specular stages in arbitrary order and never intended to support it. It expects bumpmap to be first in every "interaction packet".

The simplest solution here is to invent a condition which looks like wrong order and post warning in this case.
For instance, if we have a bump at the end without matching diffuse, it is most likely an error. If we have several diffuse/specular maps in single group, it is also an error.

UPDATE: Ok, there is a special condition when first stage is not bumpmap.
And I think it does not work properly... only provides an illusion that bumpmap not at first place is supported.

[OrbWeaver]

1 hour ago, stgatilov said:

You might as well ask: what is the motivation to specify interaction properties as 3 seemingly independent stages?

 

 

 

 

 

I think the answer is that when Doom 3 ran on shader-less platforms (hello from 2001), each stages was rendered separately and blended into total result. Perhaps it was important bumpmap stage was rendered first.
Today the stages are always combined into diffuse+specular+bumpmap packets.

The sorting logic in game is much more convoluted:

  Hide contents

/*
===============
idMaterial::SortInteractionStages

The renderer expects bump, then diffuse, then specular
There can be multiple bump maps, followed by additional
diffuse and specular stages, which allows cross-faded bump mapping.

Ambient stages can be interspersed anywhere, but they are
ignored during interactions, and all the interaction
stages are ignored during ambient drawing.
===============
*/
void idMaterial::SortInteractionStages() {
	int		j;

	for ( int i = 0 ; i < numStages ; i = j ) {
		// find the next bump map
		for ( j = i + 1 ; j < numStages ; j++ ) {
			if ( pd->parseStages[j].lighting == SL_BUMP ) {
				// if the very first stage wasn't a bumpmap,
				// this bumpmap is part of the first group
				if ( pd->parseStages[i].lighting != SL_BUMP ) {
					continue;
				}
				break;
			}
		}

		// bubble sort everything bump / diffuse / specular
		for ( int l = 1 ; l < j-i ; l++ ) {
			for ( int k = i ; k < j-l ; k++ ) {
				if ( pd->parseStages[k].lighting > pd->parseStages[k+1].lighting ) {
					shaderStage_t	temp;

					temp = pd->parseStages[k];
					pd->parseStages[k] = pd->parseStages[k+1];
					pd->parseStages[k+1] = temp;
				}
			}
		}
	}
}

 

It splits all stages into groups, such that each group (except maybe the first one) starts with bump.
Then every group is sorted in order: ambient, bump, diffuse, specular.
It is not important where ambient stages are.

So according to what I see, the engine does not support setting bump/diffuse/specular stages in arbitrary order and never intended to support it. It expects bumpmap to be first in every "interaction packet".

The simplest solution here is to invent a condition which looks like wrong order and post warning in this case.
For instance, if we have a bump at the end without matching diffuse, it is most likely an error. If we have several diffuse/specular maps in single group, it is also an error.

UPDATE: Ok, there is a special condition when first stage is not bumpmap.
And I think it does not work properly... only provides an illusion that bumpmap not at first place is supported.

 

[OrbWeaver]

1 hour ago, stgatilov said:

You might as well ask: what is the motivation to specify interaction properties as 3 seemingly independent stages?

I have wondered that myself, but I always assumed the purpose was to allow individual stages to be given different transforms, e.g. you can have the diffusemap twice as large as the bumpmap, or give it different RGB scaling or whatever. If the D/B/S maps were all part of a single stage declaration this wouldn't be possible without different syntax (e.g. nested sub-stages).

1 hour ago, stgatilov said:

Today the stages are always combined into diffuse+specular+bumpmap packets.

So according to what I see, the engine does not support setting bump/diffuse/specular stages in arbitrary order and never intended to support it. It expects bumpmap to be first in every "interaction packet".

That's perfectly fine, as long as the "packets" themselves are kept whole.

If the material stages are {D1, B1, D2, B2} and the engine sorts this into {B1, D1} {B2, D2} then there is no problem, but what DarkRadiant (and maybe the engine) is currently doing is sorting the original list into {B1, B2, D1, D2}, then reconstructing the triplets at render time as {B1, <black>} {B2, <black>} {B2, D1} {B2, D2} which is obviously wrong and gives rise to the observed behaviour of only the second bumpmap appearing on a vertex-blended model.

Maybe our sort code is fundamentally sound, but it needs to happen within triplets, not on the whole list of parsed material stages? But then within a triplet, each material has a specific purpose and is assigned to a dedicated shader uniform, so I'm not sure if the order has any effect.

[stgatilov]

46 minutes ago, OrbWeaver said:

If the material stages are {D1, B1, D2, B2}

This is wrong.
Bump stage must be the first in each packet, otherwise everything breaks.

To be clear: single-interaction materials can have stages out of order I think. Multi-interation must be a sequence of packets, each packet must start with bump stage.

[OrbWeaver]

19 hours ago, stgatilov said:

This is wrong.
Bump stage must be the first in each packet, otherwise everything breaks.

I was referring in this case to the order of stages in the MTR declaration. I don't think there is any requirement for the bumpmap to appear first in the MTR, is there? I certainly haven't seen any mention of this requirement in the texture guidelines, and most of our materials have the diffusemap first.

19 hours ago, stgatilov said:

Multi-interation must be a sequence of packets, each packet must start with bump stage.

I'm still confused as to which part of the rendering code cares about the ordering. Is it simply that the engine passes around arrays or vectors (rather than structs with named fields), and it needs to assume that images[0] is always the bump map?

[stgatilov]

5 hours ago, OrbWeaver said:

I was referring in this case to the order of stages in the MTR declaration. I don't think there is any requirement for the bumpmap to appear first in the MTR, is there? I certainly haven't seen any mention of this requirement in the texture guidelines

I have not seen any mention about order of stages in the docs.

But Doom 3 is such a thing where requirements are imposed by implementation.
The implementation of material stages sorting most likely has not changed since Doom 3.
It clearly expects bumpmaps to go first in each packet.

Moreover, I would argue that "fixing the issue" would only increase confusion.
Because D1 B2 D2 B3 looks like a valid combination that would work in unexpected way, no?
It is much better to restrict the rules and prints warnings, then to allow any order of stages and silently work in some way... which can be not the way mapper intended it.

Quote

, and most of our materials have the diffusemap first.

Because most of the materials use single interaction pass.
If there is one bumpmap, then everything is assumed to be single interaction pass and is sorted accordingly.

I think the question is: is bumpmap stage required if diffuse or specular is set?
If it is not required and is implicitly replaced with _flat, then the rules should definitely not be relaxed. In fact they should be restricted further, but we cannot do it.

[OrbWeaver]

Well I guess you learn something new every day. The blending works correctly in the engine if you put the bumpmaps before the diffusemaps in the MTR. It is also the order that KatsBits tutorial example is using, although he doesn't mention that it's a requirement.

I guess we can consider this a documentation issue rather than an engine coding issue (it still needs fixing in DR though).

[stgatilov]

I wanted to add a warning, but quickly realized that it is hardly possible.
It is even impossible to statically decide how stages are split into interaction blocks!
The full writeup is here: https://bugs.thedarkmod.com/view.php?id=5718#c16735

I suggest adding "interaction separator" keyword to materials syntax.
If at least one interaction separator is present, then:

1. Implicit bump stage is not added (backend will use _flat anyway).
2. Stages are split into interaction blocks by interaction separators only.
3. If a block contains several active stages of the same type, backend selects the first one and ignores the rest.
4. If stage of some type is missing, backend uses _flat, _black, _black respectively instead.
5. The stages within interaction block are stable-sorted by type, but this is merely technical detail since only respective order of same-type stages matters.

With interaction separator, mappers don't need to remember the complicated and inconsistent rules regarding how they must write multi-interation materials, since they denote the stage blocks themselves.

 

[OrbWeaver]

I'm certainly in favour of new syntax if it makes things clearer or more flexible. We might consider something like POV-Ray's nested block approach, e.g.

Blend between two materials:

interaction {
	diffusemap textures/first_d
	bumpmap textures/first_local
	vertexColor
}
interaction {
	diffusemap textures/second_d
	bumpmap textures/second_local
	inverseVertexColor
}

Shared bumpmap, two diffusemaps:

interaction {
	bumpmap textures/bump_local
	diffusemap {
		map textures/first_d
		vertexColor
	}
	diffusemap {
		map textures/second_d
		inverseVertexColor
	}
}

 

[OrbWeaver]

The renderer is now fixed in DarkRadiant to give the same results as the engine, provided the stages in the MTR are correctly sorted.

This finally answered the question of why the sorting matters: we need to choose a particular material stage to be the "delimiter" between interaction passes, otherwise there is no way to correctly handle both blending between two completely different textures sets (B1, D1, B2, D2) and sharing a bumpmap between two blended diffusemaps (B1, D1, D2). There's no particular reason why the delimiter has to be the bump map, but I assume the D3 devs chose this approach because sharing a bump map between two diffuse maps is more useful than sharing a diffuse map between two bump maps.