Jump to content
The Dark Mod Forums

I made a script adding cubemap reflections for all materials with specular maps


MirceaKitsune
 Share

Recommended Posts

I recently saw discussion about PBR materials being added to Doom3BFG with folks also talking about it on Discord. One of the things I always wanted from PBR is proper reflections, especially in a way that works on all FM's new and old without requiring changes (eg: new light entities). Lack of proper reflections is one of the biggest limitations we still have, leaving us with mere boring specular reflections lacking any detail.

While currently we can't have things like metallicity or per-pixel roughness, not even the ability to use the skybox or player camera feed as a reflection map, we do have a generic cubemap used only on windows and a few special textures. So the thought itched me: What if we could make every material with a specular map also blend a cubemap reflection? I've done Linux batch scripts for complex tasks before, so after lots of searching (and dealing with DOS era line terminations getting in the way) I managed to create a bash script that will do just that!

This 50 line script will detect all materials with a specular map, inject a customized cubemap reflection, then repack everything. It scans every material in TDM thus it changes all map textures models and entities alike, everything gets modified to benefit from this. Modifications are NOT made to the official pk4 files, instead a single pk4 named Z-tdm_materials_cubemap.pk4  is generated to override the old defs, you can revert at any time simply by removing this one file. The cubemaps are subtly blended in without using extra vertex / fragment shaders which should have minimal performance impact, they also respect the bump map of the material and are deformed by it... each cubemap is masked by the specular map which gives it a close feel to PBR, materials without a specularity texture are considered rough and remain unchanged. Simply download the script and use it in your TDM folder, you'll need either Linux or a bash environment on Windows (untested):

material_cubemaps.sh

#!/bin/bash 
# Add cubemap reflections to all TheDarkMod materials containing specular maps
# Use sub(/\r$/,"") to fix DOS line termination, https://stackoverflow.com/questions/45772525/why-does-my-tool-output-overwrite-itself-and-how-do-i-fix-it

# Unpack all materials
mkdir "temp"
for f in *.pk4; do
	unzip -o $f "materials/*" -d "temp"
done
mv "temp/materials" "temp/mtr"
mkdir "temp/materials"
cd "temp/mtr"

# Inject cubemap code into all materials with specular maps, the reflection is masked by specular intensity
# First ensure the file contains at least one definition that need to be modified to avoid needless repacking
for f in *.mtr; do
	awk '{
		sub(/\r$/,"")
		if($1 == "specularmap" && $2 != "_black" && $2 != "")
			exit !f
	}' "$f"
	if [ $? == 1 ]; then
		awk '{
			sub(/\r$/,"")
			print $0
			if($1 == "specularmap" && $2 != "_black" && $2 != "") {
				print ""
				print "	// Cubemap reflection for specularity"
				print "	{"
				print "		maskcolor"
				print "		map makealpha(" $2 ")"
				print "	}"
				print "	{"
				print "		blend		gl_dst_alpha, gl_one"
				print "		maskalpha"
				print "		cubeMap		env/gen3"
				print "		texgen		reflect"
				print "	}"

			}
		}' "$f" > "../materials/$f"
	fi
done

# Pack modified materials
cd ".."
zip -r "../Z-tdm_materials_cubemap.pk4" "materials"
rm -r "../temp"

At the moment I haven't done a full comparison and only tried it out on a map I'm working on: It's possible I might do my next TDM stream with this on which will allow others to see it better. From what I'm noticing it's pretty much perfect: Very subtle and doesn't disrupt visually, it does improve realism in a lot of cases as you move around and see the shine... given the texture is almost always masked and distorted by bump you don't feel the reflection is ugly and fake but it feels natural. Currently this is here as a mod for players that wish to use it... not gonna lie part of me is tempted to suggest it be considered for 2.11, it's definitely an improvement to having no reflections at all until a better method is found. Please try it out and share your own thoughts and images, below are a few screenshots I took during my first test to confirm it works as expected.

vB38BC8.jpg

bbXnH9R.jpg

RRmRBQh.jpg

kQvqaVZ.jpg

msmqiWj.jpg

iw25SNq.jpg

Md8DhcU.jpg

g8ZqSnw.jpg

  • Like 3
Link to comment
Share on other sites

Here's a bunch of comparisons from other FM's which I did recently. Hard to find spots where the results are obvious in an image, it's very subtle but very good where you do see it.

Spoiler

mvntajC.jpg

TMbRhUj.jpg

Spoiler

yXnSoOL.jpg

GV8WVds.jpg

Spoiler

8XhRGUF.jpg

vbLIZRr.jpg

Spoiler

hNHO0Jd.jpg

TdZSYhU.jpg

Some bonus ones better showing the effect on guards and ice and some models:

adAr71i.jpg

nC9GkbQ.jpg

9BnOTvF.jpg

Dzofqum.jpg

Also here's the pk4 for TDM 2.11 Beta 5 in case anyone has issues with the script, just drop it in the base darkmod directory. It should be rebuilt after every update of course.

Z-tdm_materials_cubemap.pk4

 

Link to comment
Share on other sites

Please don't take this negative - maybe I'm just too dumb and blind - I don't see any difference anywhere except the bath tub. This makes me wonder if this even noticeable while playing the game. What exactly were you aiming for? Beware I'm not an expert 😁

"Einen giftigen Trank aus Kräutern und Wurzeln für die närrischen Städter wollen wir brauen." - Text aus einem verlassenen Heidenlager

Link to comment
Share on other sites

6 hours ago, SeriousToni said:

Please don't take this negative - maybe I'm just too dumb and blind - I don't see any difference anywhere except the bath tub. This makes me wonder if this even noticeable while playing the game. What exactly were you aiming for? Beware I'm not an expert 😁

It's alright, and that is normal: You won't see a huge difference on everything, just a few surfaces typically smooth metal and stone. The goal of the experiment was to see how close I could get to realistic reflections that would match what you get from a PBR engine... obviously this has nothing to do with PBR itself, just a comparison to its feel.

Like in real life only some surfaces will produce reflections: You won't see sharp reflection on your walls for instance, but will see a light one on your door knob. The trick in my approach is only those materials with a specular map are given a cubemap reflection, the reflection is masked by the specular map and works as a supplement to it; This ensures every material is only as shiny as intended by the texture pack, which makes the effect barely noticeable on surfaces with low specularity. Making everything shiny could have looked prettier in the moment but would be fake and unrealistic.

Also worth noting: There are three default /env/gen# cubemaps. gen1 is very low intensity (almost unnoticeable), gen2 is the strongest (makes everything bright), gen3 is somewhere in the middle (looks best so I used it). You can change the script to gen2 but apart from making the universal cubemap too obvious it might make everything too bright by default.

4 hours ago, OrbWeaver said:

You used bash for this?

Why not a proper cross-platform language with decent text processing support that others might be able to understand and maintain, like Python?

I prefer Python myself, Bash is weird and uses strange syntax. It's what Linux has by default so I got used to thinking it's best for stuff like editing files, especially as I also need to zip / unzip archives. Something like this is easy to do though, if there was real need it could be easily redone in Python or Lua or NodeJS.

2 hours ago, nbohr1more said:

Neat trick.

Too bad such an approach can't easily generate parallax corrected cubemaps:

That would be ideal and what I wanted, either something like this with light probes or actual SSR (screen space reflections). Unfortunately it's not currently possible with existing capabilities: I spent the first day trying to trick the material into using the _currentRender or _scratch pass to fake reflections, obviously it won't work. My approach is admittedly meant to be a crutch till we have that, for those of us who prefer seeing some kind of shine rather than just the boring specular reflections... both approaches are ugly and unrealistic but at the moment it's a pick between which is the least ugly.

  • Like 2
Link to comment
Share on other sites

Hey wow, this is so gnarly!  I spent the last little chunk of my life learning a bunch of commandline stuff like awk, sed, grep and a lot of versions and other utilities, things like ImageMagick.  I thought I was the only one actually taking that seriously for making utilities like this.  The last year of my life has involved obsessively hoarding and learning all the CLI stuff I can find, to automate as much as possible, combined with a little Python for Blender and MAXscript for 3DS.

 

I'm currently working on something that will convert lights in MAP files to DAE Collada files, for import into Blender and 3DSMAX.  After that, a solution to read from MTRs and setup basic (or potentially not so basic) materials to automatically take care of monkeywork between Radiant and that software, by using MaterialIDs coordinating with headers and propagating its only little contextual base of assets for itself.  It's convoluted, but should get the job done.  It wouldn't be so far off what you've done here. but I hope to expand it along the way and make it easy for people to finally render their maps in Eevee, etc. and beyond.

  • Like 1
Link to comment
Share on other sites

1 hour ago, LDAsh said:

Hey wow, this is so gnarly!  I spent the last little chunk of my life learning a bunch of commandline stuff like awk, sed, grep and a lot of versions and other utilities, things like ImageMagick.  I thought I was the only one actually taking that seriously for making utilities like this.  The last year of my life has involved obsessively hoarding and learning all the CLI stuff I can find, to automate as much as possible, combined with a little Python for Blender and MAXscript for 3DS.

 

I'm currently working on something that will convert lights in MAP files to DAE Collada files, for import into Blender and 3DSMAX.  After that, a solution to read from MTRs and setup basic (or potentially not so basic) materials to automatically take care of monkeywork between Radiant and that software, by using MaterialIDs coordinating with headers and propagating its only little contextual base of assets for itself.  It's convoluted, but should get the job done.  It wouldn't be so far off what you've done here. but I hope to expand it along the way and make it easy for people to finally render their maps in Eevee, etc. and beyond.

Sounds very interesting! Let us know how it goes. Something I was thinking about: It would be fun to see how TDM maps would look with raytracing... granted our engine will surely never have such a thing unless a complete technological miracle happened, importing it to Blender and rendering in Cycles could give us an interesting view :)

As for such scripts I write them when needed which isn't all that often. Without Google (or DuckDuckGo in my case) I wouldn't be able to do it in Bash's case, both as it's more complex and I use it more rarely compared to JavaScript.

Edited by MirceaKitsune
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

  • Recent Status Updates

    • snatcher

      TDM Modpack v2.6 released!
      · 0 replies
    • peter_spy

      Perhaps an unpopular opinion: TDM team might benefit from someone with actual QA experience; someone with naturally and professionally developed curiosity, who is interested in how and why things work, how they break At least to me it's kind of mind-boggling how untested some rather important features are (first the absence alert feature for items, now the rope +body carry behavior).
      · 4 replies
    • nbohr1more

      The Dark Mod is hosting an Ask Me Anything thread on the PC Gaming reddit forum:  https://www.reddit.com/r/pcgaming/comments/10nfcwj/hello_we_are_the_international_development_team/
      Feel free to join the discussion there
      · 2 replies
    • stgatilov

      Bumped into an interesting piece of wisdom called Hyrum's Law:
      With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.
      · 5 replies
    • The Black Arrow

      I love playing The Dark Mod when it's cold in my place. Bonus points when it's a bit (or even very) dark and it's raining, too.
      · 2 replies
×
×
  • Create New...