Experimental support of parallax mapping in 2.13

[lowenz]

Experimental implementation of parallax mapping

Isn't it reatroactive (in old FMs), right?

[MirceaKitsune]

17 minutes ago, lowenz said:

Experimental implementation of parallax mapping

Isn't it reatroactive (in old FMs), right?

Shaders and engine changes are almost always retroactive, which is a very good thing in my opinion: Such settings can be disabled in the main menu if anyone wishes, when enabled they should work in any FM new and old.

I was just thinking about parallax mapping in TDM but didn't expect it to actually happen: This is a wonderful surprise! Still wishing Depth of Field could be added to the list, could provide a more realistic and cinematic field, who knows

Unfortunately my main internet connection is down due to a power outage in the hallway of the apartment block, I'm using a limited mobile connection and can't download a lot of stuff. Once it's fixed it will be a pleasure to update and try out the parallax, hopefully it works well on the Linux / amdgpu driver.

Edit: We have OCCLUSION based parallax mapping with self shadowing?! I thought it's just the simple distortion at first. My god, this will be glorious

Edited October 14 by MirceaKitsune

[stgatilov]

16 hours ago, lowenz said:

Experimental implementation of parallax mapping

Isn't it reatroactive (in old FMs), right?

Of course it is not retroactive.

Parallax mapping requires heightmap and height scale, which does not exist for old materials.
Also, any kind of parallax mapping introduces visual artefacts and might require tuning for specific use case, so we can't just force it onto everything.

[datiswous]

I'm looking forward to a new techdemo.

I think it would actually be cool to have such a video with every new TDM release, showcasing (some of) the new features. Off course it can only be done if somebody does it.

[AluminumHaste]

4 hours ago, datiswous said:

I'm looking forward to a new techdemo.

I think it would actually be cool to have such a video with every new TDM release, showcasing (some of) the new features. Off course it can only be done if somebody does it.

I did a few of them and they had a lot of traction, motivation is hard to come by

[MirceaKitsune]

5 hours ago, stgatilov said:

Of course it is not retroactive.

Parallax mapping requires heightmap and height scale, which does not exist for old materials.
Also, any kind of parallax mapping introduces visual artefacts and might require tuning for specific use case, so we can't just force it onto everything.

Oh, I understand now. That is a bit sad: I was hoping it would work everywhere like ambient occlusion.

As for new maps I thought the shader found a way to extract the information from the normal map, it makes sense that a heightmap would still be needed. I recently worked with generating height maps for a texture package in Godot, and found that just extracting the blue channel of the normal map into a grayscale image worked for me. There are some tools to automatically produce better results, but if the decision isn't to patch all existing materials everyone will have to use those independently.

[MirceaKitsune]

I keep wondering if nothing can be done to let everyone benefit from the new parallax shader, for FM's both new and old. Apart from those of us who want to not being able to enjoy them on existing missions, the biggest issue is this great feature will almost never be used at large: No author will copy hundreds / thousands of materials from brush textures to skins applied on entities just to enable the effect. It feels like wasted potential for something that's already there and could look so amazing.

Regarding the lack of height maps for existing textures, I suggest testing something if anyone knows how and has the time, I'm not familiar with the new system and trying to get a quick FM done while busy with several things at the moment: Extract the blue channel from the normal map and use that as the heightmap, maybe share some images of how that looks. From my experience with other texture sets in Godot, that provides accurate enough parallax and I'd to know if it looks good in TDM too.

If it does we have a solution for applying it retroactively as a choice: The setting can have two modes... one to enable parallax only for materials that explicitly specify a heightmap, the other to generate an estimated heightmap from the normal map otherwise. As a bonus we don't increase the size of TDM by adding hundreds of new textures either!

Edit: Here's an image from the test project I was working on in Godot. The Philipk texture set doesn't have heightmaps, I had to generate those and invert the specular texture to turn it into roughness: I automated an ImageMagick command to do it for all images and generate PBR versions. The parallax was extracted from the normal texture, it worked fine and looks exactly as I wanted it to.

Edited October 19 by MirceaKitsune

[Wellingtoncrab]

@MirceaKitsune I don't want to devote too much space in this release notes thread to lengthy POM discussions (apologies in advance), as I am sure it will have it's own thread eventually as it matures in development.

Just so you are managing your expectations even if it were desirable to just enable it for existing missions in the game (it's not), POM is a fragile illusion and in my experience so far it's not as straightforward as just extracting rough height information from a normal map and then flipping the "POM ON" switch. This will provide awful results and break the artistic intent behind missions for many reasons including:

1. Parallax can really only create a relief - you need to work on your displacement values and material settings quite a bit to produce a result that is convincing as well as acceptable in terms of noise and is still useable as a typical tiling texture in the game that can achieve results like convincingly wrapping an edge or corner, allowing ai/objects to convincingly walk/rest on it, or allowing any other surfaces to intersect with it. 
2. Displacement also really makes things which are small issues with normal tiling materials look quite problematic. Things like texture alignment for example - how many TDM missions do you think were authored considering displacement when placing and aligning textures ? (Answer: 0%) The depth of relief is also influenced by things like the texture scale - what proportion of TDM missions were authored considering displacement when scaling textures? (Answer: 0%) This is something you would not just want to "flip on" for the entire game.
3. When you have actual displacement it becomes quite apparent if/when your height information is not physically correct. Many if not most of the materials in the game have normals which were derived from the diffusemap color information using tools like “crazybump”. Convincing enough on a flat surface, but that is not going to be a given when you introduce actual displacement. Even if you addressed this with a new custom heightmap, you would then also need new normals for the parallax to be lit correctly in game.

The POM looks really *really* awesome, but it is something which should be used rather intentionally and somewhat sparingly. If this feature makes it into 2.13 (keep in mind it is experimental) - it will be coming with many *new* assets which support the feature and I am sure the community will start providing even more. This may even include some classic textures getting parallax treatment, but as far as the core game is concerned these will be new materials. There is no intention of replacing existing material definitions in the game and the choice of how to deploy parallax assets should be the decision of mission authors.

The idea that this does not “let everyone benefit”, or that new assets “will never be used at large” by authors has no bearing in reality in my opinion.

I really don’t understand that mindset personally (I like how our classic missions look already) but if this is truly a source of disappointment for you, I suggest you pursue a mod or texture pack. I would just advise you consider the points above!

[MirceaKitsune]

Thanks for clarifying, and I agree with most of your points. I'm aware the effect doesn't look too realistic as it's still a 2D illusion and doesn't pop out of the actual geometry, though I do think it often looks better to have it than not. Especially with the self shadowing which can make all those highlights more realistic, that's the part I was excited about rather than the deform. I think the difference is I'm thinking of it more along the lines of shadow maps or ambient occlusion, where once they were added they worked everywhere... those are post-processing shaders not material shaders so it's indeed a different matter.

The reason I say is it won't be used at large is that at no point will a FM enable it on every single texture: That would require turning it on for every brick, wood, metal, AI skin, etc used on the map meaning dozens of copied material / skin / entity definitions. Using it on everything like that is what I initially hoped would happen, but if there's really no way I understand, the devs who actually know the code and shaders have their reasons for the limitation. And yes, feel free to move this to another thread if that's more appropriate, though I think your point clarifies it.

[AluminumHaste]

8 hours ago, MirceaKitsune said:

Is parallax even working in this image? It looks.....weird? Like something is off.

[MirceaKitsune]

2 hours ago, AluminumHaste said:

Is parallax even working in this image? It looks.....weird? Like something is off.

That's Godot's parallax shader, it tends to look like that especially when occlusion depth is turned off. The direction and highlights are correct though: That's mainly what I wanted to test when I generated the PBR versions of the texture pack.

[Arcturus]

I still have some heightmaps lying around from the olden days of normalmap making in GIMP.

Less exaggerated:

 

[stgatilov]

Since parallax mapping has sort-of gone public, maybe we should create a public thread about it?
I can extract this discussion into a new thread as a start.

[Wellingtoncrab]

5 hours ago, stgatilov said:

I can extract this discussion into a new thread as a start.

Yeah I would do that at this point.

[stgatilov]

Mathematically it is possible to extract heightmap from normalmap by solving partial differential equation.

If heightmap is f(x, y), then normal vector is proportional to N = (-df/dx, -dy/dx, 1).
Hence we can find df/dx = -Nx / Nz, df/dy = -Ny / Nz.
So normalmap gives us partial derivatives of heightmap in pretty straightforward way, but their precision is around 0.4% at best.

If normalmap was generated properly from a real heightmap, then this problem is called "reconstructing potential function for conservative vector fields". But since our values are rather imprecise, we need some stable solution, not just some trivial prefix sums.

I'm not sure how stable the possible solution could be, and how good will be the quality of the output.
Even if they would be OK, this approach can only be usable as offline tool to be run explicitly.

[Arcturus]

For my textures I was typically making black and white maps rather painstakingly by hand, at least for the low frequency details.

https://drive.google.com/file/d/18KhQA54g_hwZ_GM3x5tAdAg7OpVafo57/view?usp=sharing

But there's a lot of crappy normalmaps in Darkmod too.

[Arcturus]

Here's a depth map estimated by "depth anything v2" extension in Stable Diffusion, using nothing but the diffusemap as input:

Gives decent results, except for the weird slope. After tweaking using "wavelet decompose" in GIMP:

It's still not perfectly tiling, so one would have to remove seams to use it.

[Wellingtoncrab]

These days a height map is relatively straightforward to derive from a normal map using pretty much any node based material software (there is usually a dedicated node for this).

Provided the result is even physically correct, in my limited experience this will not necessarily be suitable to use as a displacement map in the game right away. 

There will be a lot of high frequency detail which give a noisy result (this kind of detail in a texture remains better handled by the normal map) and the black/white values will need to be adjusted to get both a good displacement effect in the game AND retain any practical versatility in the texture itself (for all the reasons I outlined above). The greater a proportion of the displacement map you get to approaching a value of 1 (white) while still keep the effect you want the better result you will get when you try to go and use the texture in a map.  

This is obviously very early, and I am just a amateur so consider this advice as opposed to firm guidance. If I am dealing with an existing heightmap I will usually:

1. Level adjust to remove detail and bring the values closer to 1
2. Apply blur and reduce resolution to further remove detail
3. Prep a corner wrapped material test and begin to adjust the min/max values in the material def and/or steps 1 & 2 in until you get *acceptable* tiling across the transition - this will *not* be perfect - you just want to minimize and to minimize distortion or being able to see *past* the edge via the parallax.

41 minutes ago, Arcturus said:

For my textures I was typically making black and white maps rather painstakingly by hand, at least for the low frequency details.

https://drive.google.com/file/d/18KhQA54g_hwZ_GM3x5tAdAg7OpVafo57/view?usp=sharing

But there's a lot of crappy normalmaps in Darkmod too.

This looks like a pretty great example of what you want in a displacement map in terms of the frequency of detail!

[Arcturus]

I downloaded another plugin for Stable Diffusion for depth and normal maps. It has additional options and renders 16 bit images. They still have large tonal variation,  so I created 3 different versions by simply rotating the diffusemap. I overlayed them in GIMP, which leveled the image somewhat. Then I applied high pass filter with high contrast. I had to smear the seams, but that wasn't a big problem.

https://drive.google.com/file/d/1VsvnCuK_U9JYLY4mY-4HTScYdyMi9maV/view?usp=sharing

 

[MirceaKitsune]

Thanks for giving this its own thread, as well as that demo video which looks incredible and even better than I imagined!

[Arcturus]

I did not expect this to be possible but you can blend two materials with different parallax effects using vertex colors. Since you can control how far or how close the image is supposed to be rendered using min / max keywords, it's possible to make a trippy effect that resembles autostereogram illusions.

 

[Arcturus]

How about (pseudo) volumetric clouds.

Continued in this thread.

[Arcturus]

Here's textures/darkmod/plaster/panel_decorative_white texture, current normalmap only:

Current normalmap plus heightmap generated in Stable Diffusion (with a bit of wavelet decompose + highpass filter in GIMP):

And this one is using only the heightmap. Normalmap is generated from it using the blend bumpmap command in the material.

textures/darkmod/plaster/panel_decorative_white
{
    surftype15
    description "tile"

    qer_editorimage textures/darkmod/plaster/panel_decorative_white_ed
    diffusemap      textures/darkmod/plaster/panel_decorative_white
    {
    blend bumpmap
    map heightmap  (textures/darkmod/plaster/panel_decorative_white_height , 10)
    }
    {
        blend parallaxmap
        map textures/darkmod/plaster/panel_decorative_white_height
        min    -0.03
        max    0.03
    }
}

I used the Depth Anything v2 Large model using this addon I mentioned before and I think it did a great job.

https://drive.google.com/file/d/1cf1V861gnc0aPs8rn2BGcQuO0PYhIuLG/view?usp=drive_link

[datiswous]

On 10/21/2024 at 6:10 PM, Arcturus said:

it's possible to make a trippy effect that resembles autostereogram illusions.

I was looking at the video before reading the text and was immediately thinking about that.

[Arcturus]

2 hours ago, Arcturus said:

Normalmap is generated from it using the blend bumpmap command in the material.

Tried the same method on the cobblestone wall. Instead of the current handmade normalmap, I used the Stable Diffusion-generated heightmap.