Jump to content
The Dark Mod Forums
Sign in to follow this  
Ralle321

Vertex Colors break shading

Recommended Posts

I'm getting broken shading on models using vertex colors. This happens irregularly and might need a few tdm restarts to show up.

Without vertex colors the shading is always fine.

So i made a testmap showing a model with vertex colors(green) and a copy without vertex colors(grey):

Broken:

https://s12.directupload.net/images/200615/5zowqjao.png

https://s12.directupload.net/images/200615/nlr7crb3.png

Fine:

https://s12.directupload.net/images/200615/mk2rbalo.png

https://s12.directupload.net/images/200615/6tnswcyq.png

Always fine without vertex colors:

https://s12.directupload.net/images/200615/z5rbpj86.png

https://s12.directupload.net/images/200615/pushrgbk.png

Testmap for download: https://mega.nz/file/1IYXCaJQ#WttwX7bBmlH-GDVKzQtuubNUxm3kbZRAvlqr8NS2Tv4

Also interesting, when removing the vertexcolor line from the mtr file, the problem can still be observed.

Can you guys confirm this issue?

Share this post


Link to post
Share on other sites

Moin Ralle!

As far as I know, Vertexcolors are exclusively used to blend textures. Conversely, only grayscale colors should be used.

Details here and here.

Share this post


Link to post
Share on other sites

I'm using DR 2.8.0x64 with TDM 2.07x64 and haven't checked out 2.08 beta yet.

Will try that out and report back.

  • Like 1

Share this post


Link to post
Share on other sites

No luck with 2.08 beta 7: https://s12.directupload.net/images/200615/y7oq9xmv.png

2 hours ago, STiFU said:

As far as I know, Vertexcolors are exclusively used to blend textures. Conversely, only grayscale colors should be used.

Details here and here.

Grayscale values don't make the problem go away for me: https://s12.directupload.net/images/200615/llynbogo.png

Are vertex colors only meant to be used for blending materials in the dark mod? Vertex colored models totally work for me, when the game(or whatever is responsible) doesn't decide to screw their shading.

Share this post


Link to post
Share on other sites

@cabalistic since you've recently redone the whole backend, can you shed some light onto this topic? Is vertex color used for anything other than blending materials and are color rather than grayscale value supported?

Share this post


Link to post
Share on other sites

It's definitely not a DR problem — DR does not write or in any way modify materials or model vertex colors. The screenshots are from in game, so it must be an aspect of the way the game engine renders vertex colors. It's possible that DR might render vertex colors incorrectly (although I'm not aware of any specific issues) but this would not affect the appearance in game.

13 hours ago, Ralle321 said:

Are vertex colors only meant to be used for blending materials in the dark mod? Vertex colored models totally work for me, when the game(or whatever is responsible) doesn't decide to screw their shading.

Blending between two textures is the most common use-case for vertex colors, but it is definitely not the only possible one. Using vertex colors to modify the visible color of a mesh is a perfectly legitimate technique and this should work correctly.

I'm guessing the problem is caused by something in the more recent versions of the graphical shaders used by the engine, but one of our graphical developers would need to investigate further.

Share this post


Link to post
Share on other sites

I walked your test map through a graphical debugger - this has absolutely nothing to do with the vertex colors, themselves. I can comment out their contributions to the shading, and it still looks wrong. The problem stems from the vertex normals/tangents/bitangents not giving a smooth interpolation over the model with vertex colors, resulting in the underlying tris being visible in the specular term calculation.

So, to me it looks like the model with vertex colors has gotten its vertex normals/tangents/bitangents messed up. Whether that happened during export or during load by the game, I'm not sure.

Share this post


Link to post
Share on other sites

On both 2.07 and 2.08 the green model looks okay on my end, although it has wrong topology, so it's great that it doesn't show any tension from so many edges ending in one vertex, although it does have some problems where the sphere meets the flat surface, and this should be expected:

Clipboard01.jpg

Still, if that's something to do with export options, maybe you could give us a screengrab of the export dialog? I'm using .ase for models as well, and this is what I use:

Clipboard02.jpg.2d9ba8a9cf6df428efa359efcdc93108.jpg

Edited by peter_spy

Share this post


Link to post
Share on other sites

Sure thing, here are my export settings (i'm using Blender 2.83 with the ase exporter from Orbweaver):

hqupsa2y.jpg

You can have the blend file as well: https://mega.nz/file/ZFRjASTY#wxZ62WdHDWATLWek6EjOJw830HSdEr3OUia331s9GOw

4 hours ago, cabalistic said:

So, to me it looks like the model with vertex colors has gotten its vertex normals/tangents/bitangents messed up. Whether that happened during export or during load by the game, I'm not sure.

I doubt this has something to do with the export, since the ingame render from the vertex colored model can become good and bad with the same file. Plus, both models had the same export settings and the one without vertex colors always renders fine.

Share this post


Link to post
Share on other sites

No, the 'keep sharp' checkbox on the weighted normal modifier makes it ignore edges marked as sharp.

It's displaying fine in Blender. The problem may only appear in the dark mod.

Share this post


Link to post
Share on other sites

The model topology in the flat parts imo is really not optimal, so perhaps is smoothing groups causing problems? 

If it was me I would cut (separate in blender I think) the model manually at the "hard edges", to see if that solves the problem, this if the exporter doesn't decide to "glue" the model again at export time, afaik exporters should do this cut automatically at export, when using smoothing groups or "hard edges" but perhaps is failing in this case?  If that solves the problem then I would rework the topology.

Btw what model is that for? Or is a secret? 

Share this post


Link to post
Share on other sites
7 hours ago, Ralle321 said:

No, the 'keep sharp' checkbox on the weighted normal modifier makes it ignore edges marked as sharp.

It's displaying fine in Blender. The problem may only appear in the dark mod.

That's a weird name for an option then. Anyway, something looking fine in modeling app is no indication that everything is fine with it. Game engines don't support certain features and have their own requirements, although I think TDM isn't super picky about this, at least for .ase. Some modeling app features won't work though, like edit normals modifiers.

Edited by peter_spy

Share this post


Link to post
Share on other sites

Does it make a difference which modifiers you use in Blender? If you skip Weighted Normals and just use regular smoothing or the Autosmooth functionality, do you get different results after exporting? I wonder if those weighted normals somehow aren't making it correctly into the ASE file.

  • Like 1

Share this post


Link to post
Share on other sites

@HMart: It's just a demonstration model for my issue

 

So, here is a more simple example using just a cylinder beveled twice:

Cylinder types from left to right:

1) smooth with hard edges

2) smooth

3) smooth + weighted normals

4) smooth + weighted normals + vertex colors

5) smooth + vertex colors

6) smooth with hard edges + vertex colors

8ut3he3y.png

The vertex colored cylinders look corrupted.

After making the first Screenshot, i quit tdm, start it again, make the second Screenshot and get this:

szht9j3e.png

As you can see, two vertex colored cylinders display correctly this time.

So, i could quit tdm and start it again over and over, and each time i will get a random result for the vertex colored cylinders, while the ones not containing vertex colors always display correctly. Simply having vertex colors in the models is required to get broken shading, commenting them out in the .mtr file doesn't fix their faulty behaviour, only removing them from the model does.

Comparing the ase files from cylinder 3 & 4 with notepad++ shows as expected no difference besides the left out vertex color information on cylinder 3.

 

Testmap updated with cylinders: https://mega.nz/file/hBgl2byQ#MdVn2JVIhGRPuuPtXjSDOLZdY9Ix2ohQAA_0zHYIOh0

Share this post


Link to post
Share on other sites
38 minutes ago, Ralle321 said:

So, i could quit tdm and start it again over and over, and each time i will get a random result for the vertex colored cylinders, while the ones not containing vertex colors always display correctly. Simply having vertex colors in the models is required to get broken shading, commenting them out in the .mtr file doesn't fix their faulty behaviour, only removing them from the model does.

Based on this evidence, I would guess that the problem is something to do with uninitialised values in the code which loads model files. Perhaps some variables are being re-used without being properly cleared, resulting in leftover vertex color data influencing normal vectors.

Share this post


Link to post
Share on other sites

I've used vertex colors in some models before (not on TDM version of idtech 4) and never saw this behavior, idSoftware also used them for the Mars terrain peaces and afaik they look fine, but perhaps I just didn't looked carefully or this bug only happens on .ase models? All my models are in .lwo format.

If this ends being a original idtech 4 bug then not even idsoftware artists were aware of it, but Doom 3 was rater dark so that could hide some bugs. 

Edited by HMart

Share this post


Link to post
Share on other sites
1 hour ago, peter_spy said:

On a different note, I didn't know the engine actually supports weighted normals. IIRC it's a deprecated technique with normal baking, but might be useful in other workflows.

I don't think the engine is supporting it — the weighted normals are calculated by a modifier in Blender which is applied before export, resulting in (presumably) modified explicit normal vectors baked into the ASE file.

Share this post


Link to post
Share on other sites

I just found a max script for weighted normals, will try to use it in the evening. I won't use this technique in the long run, but I'm curious :) I was trying to export a model before, with Edit Normals modifier, but it was ignored during export.

Edited by peter_spy

Share this post


Link to post
Share on other sites
59 minutes ago, nbohr1more said:

@Ralle321 is this happening for both stencil and shadowmap modes?

Yes.

Btw, i forgot to mention DR doesn't show vertex colors, neither in unlit mode, lighting mode or in the model viewer.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...