TDM Texture Mod

[Hugo Lobo]

After playing various Dark Mod FM’s, I become hooked to the game, like I was with Thief (and its FM’s). So, like I do to all games I love, I tried to find ways to improve it. Since my first contact with DM and after playing T1 and T2 with the fantastic “HD Mod”, it became apparent that graphically, TDM struck me strange. Of course, it is clearly a BIG improvement over T3 and it’s not worse then Thief 2014  -- if you remove all those post processing effects, the textures are actually very bad for today standards – but it could be a little bit better. So I started to change a texture here, a texture there, whenever I found a texture that could be improved.

Initially, I made this for my own amusement, while i was playing, but as the changes increased, I started to think I it would only be fair to share it with the community.

As a note, I really appreciate the amount of work done by the contributors to TDM. It’s amazing how an open source project of a game whose genre is unfortunately condemned to target a niche player base could attract so many talented people to work together and create what essentially is the Thief 4 we never had. So this is in no way a mean to disrespect the contributors and their work.

 

What changed and how

Currently, around 530 files were changed. The changes end up in one of the following categories:

NOTE: “texture quality” noted below is subjective and represents only my point of view. Again, It is in no way a mean of disrespect for the original author and its work.     

• The texture is good but is in a low resolution – upscale it using AI image enhancement methods.
• The texture is poor and low res, with poor AI upscale results – try to replace it using various free PBR or raw Image sources (1) or create my own. If necessary, adjust the image using (colors, saturation, contrast, …)
• The texture has a good resolution and its not quite good but can be improved – improve using gimp (ex: on textures with bur, use sharpen, noise reduction or/and other features)  (2)  
• The texture depicts an horrible stew – change it to a decent and delicious stew, because my Portuguese roots forced me to do it.

    Additionally, specular and normal maps were added to some textures.

(1) Free textures and PBR sites already discussed on this forum (texturehaven.com, 3dtextures.me, cc0textures.com and so on).
(2) Finding the right texture is not always easy. I always tried to follow the same “feel” and appearance of the original image, but i confess that this is not always the case. Again, very subjective.

 

New version 2021.01.08

* Around 170Mb of textures processed

• Some tree barks enhanced
• Stucco change
• more enhancements on doors, paint paper, fire places, ground textures, curtains
•  .. and much more

 

Screenshots and Comparisons

It’s obviously undoable to show the comparison for all changed textures, so keep in mind that the following screenshots are just a very small example of the whole project. Also, very important, keep in mind that there is so much you can do with screenshots and in game the differences are much more clear than what is shown below.

Sir Talbot's Collateral
https://imgsli.com/MTI2NDE
https://imgsli.com/MTI2MzY
https://imgsli.com/MTI2Mzc
https://imgsli.com/MTI2Mzg
https://imgsli.com/MTI2Mzk
https://imgsli.com/MTI2NDA

WS3: Cleighmoor
https://imgsli.com/MTI1OTE

WS1: In the North

https://imgsli.com/MTI2MTY
https://imgsli.com/MTI2MTc

WS2: Home Again
https://imgsli.com/MTI2MjM
https://imgsli.com/MTI2MjQ
https://imgsli.com/MTI2Mjc
https://imgsli.com/MTI2Mjg
https://imgsli.com/MTI2Mjk
https://imgsli.com/MTI2MzE
https://imgsli.com/MTI2MzM

 

New (version 2021.01.08)

Briarwood Manor
https://imgsli.com/MzUwNDY
https://imgsli.com/MzUwNDg
https://imgsli.com/MzUwNTE
https://imgsli.com/MzUwNjc

The Builder's Influence
https://imgsli.com/MzUxNjM
https://imgsli.com/MzUxNjQ

No honor among thieves: forest
https://imgsli.com/MzU0ODE
https://imgsli.com/MzU1NTQ
https://imgsli.com/MzU1NTk

 

How to install

1) download the pk4 file from here

2) drop it on your TDM game folder (where all the other pk4 files are)

3) Play!

 

Uninstall

Just remove z_TDM_HD.pk4 file from your TDM install folder.

Disclaimer

If you are a purist, please don’t use this texture mod. Don’t bash it for not being “exactly the same as the original ones but hires”.

If you find some texture that is copyrighted, please let me know and i will replace it.

Fell free to suggest changes, but please don't make requests. Understand that i am doing this while playing and if i start feeling that i'm working instead, i will probably start to loose my interest.

 

PS: I really don't know if this is the right thread to make this post. Let me know if i need to change it to another thread.

 

Edited January 8, 2021 by Hugo Lobo
wrong info

[peter_spy]

Any efforts to upgrade textures and materials are appreciated, but the question is, what was your workflow? Did you use dds diffuse and specular textures as a base?

Second thing, I see some of the changes actually made some materials worse in terms of quality, e.g. here the wooden floor looks less sharp for some reason and here the bust material actually looks less detailed. Sometimes it's hard to pinpoint how to set the quality level for a texture because it's used for many objects and with different scale, so a tiling texture on a wall might look bad applied on an object.

[freyk]

I love your work, Hugo.
And thanks to upload the screenshots to a picture comparison site.

About the installation, please use standards.
Never give novice user an 7zip package, a zip-package is more of a standard on systems.
I prefer adding one big pk4 file to an fm folder or overwrite the original files in the main folder, after a backup.

Edited February 29, 2020 by freyk

[Dragofer]

This looks great, I'll be sure to take a closer look at this at a later point.

For now - if you want your .pk4's to overwrite stock .pk4's you only need to name them so they come later in the alphabet. So z_TDM_HD.pk4 or tdm_z_hd.pk4 in the main darkmod folder would do the trick for making this apply universally to FMs (unless the FMs provide their own versions of the files). And yes, it'd be appreciated if you provided this as a .pk4, which is simply a renamed .zip.

[Hugo Lobo]

Hi @peter_spy! 

Please note that i never said that this was mean to replace (upgrade) the original textures and as you surely read from my post, this was just for fun, not a professional work. I just shared this because i think it's better to share than to be only on my disk forever. So, about the workflow, more that what i already pointed in "What changed and how", i really can give you more.

About the screenshots you mentioned, I personally like the my versions better but this is a matter of taste (you can go as technical as you like about it, but in the end, its always a matter of taste). So if you want, install it and see the real differences (like i said, screenshots are screenshots), Another thing is that as you know, the textures are shared by multiple meshes and so, what looks perfect in one mesh could not look that good in another. Your are right about the marble texture though. My idea was to have a cleaner one (the file name is called "smooth_marble_greenish.dds" for god sakes... its supposed to be smooth  ).

About the floor you mentioned, the issue is that it does not use the original or an upscaled version of the normal map, but a new one. You can check the old vs new floor here. Why i did not use the original or an upscaled version of it? Because of this. I really don't think that is how a wooden floor should look like in the real world. Too much details.. it looks like plastic to me.  

Again, thank you for your reply and feedback peter_spy.

[Hugo Lobo]

Thank you @freyk! Tree reasons i did not offer the pk4 files:

- the download size would be much bigger

- sharing as i did will allow you to see what was exactly changed by me

- i think it is not fair to the original texture authors (or legal, probably?) to just replace the original textures with my textures and just deliver the modded pk4 files

About the 7zip files... its open source and pretty much everybody can use it (windows, linux, mac, ...). But i will take that in consideration in the future.

[Hugo Lobo]

Thank you @Dragofer for you tip. I didn't know that. I will see if i can make them work that way and re-upload the modded pk4 files.

[Springheel]

I like many of the changes.  But I'm curious about the size of the new textures.  Some of them seem like they've been upscaled to 2048x2048, which certainly will look better, but will do a number on loading and performance. 

[VanishedOne]

15 hours ago, Hugo Lobo said:

    Additionally, displacement, specular and normal maps were added to some textures.

By displacement, do you mean heightmaps (the ones with filenames ending _h that get invoked with heightmap() to convert them to normal maps at runtime)? If the engine supports displacement mapping that definitely passed me by.

[peter_spy]

5 hours ago, Hugo Lobo said:

So, about the workflow, more that what i already pointed in "What changed and how", i really can give you more.

I was asking this simple question because .dds images are heavily compressed (usually with DXT1 mode), with some inevitable artifacts. Unless there's an ML dataset used to account for this, you're working on a lossy image and if you save to the same .dds format, you're adding another compression on top of it.

As for the wood texture, this is how often wood looks like in games. Normalmaps are mostly supposed to represent microdetails, and if a surface looks too shiny, you change a specularmap, not normalmap. In general, I'd be wary of AI upscaling in this case, it's not a simple case as with Thief 1-2, or even 3. You have to think with surfaces and materials here, not with just textures. Btw. I hope you remember that normalmap is not a bitmap in traditional sense, so you can't apply traditional 2d image techniques to it (e.g. sharpening).

[Hugo Lobo]

8 hours ago, Springheel said:

Some of them seem like they've been upscaled to 2048x2048, which certainly will look better, but will do a number on loading and performance. 

That is always in my mind when i change some texture with another (bigger) one. Specially if it reaches 2048x2048 or more. I didn't notice any performance issues with them, but i didn't benchmark it. From my experience on multiple FMs, the lags that are there also happen with the original textures and result from the lighting system, not from textures. My rule is simple: if i don't notice performance issues and the new texture quality is noticeable ingame compared to the original one, i allow it to be 2048x2048. You can see one example in the guard screenshot.

6 hours ago, VanishedOne said:

By displacement, do you mean heightmaps (the ones with filenames ending _h that get invoked with heightmap() to convert them to normal maps at runtime)? If the engine supports displacement mapping that definitely passed me by.

I stand corrected and will update the post. There is no displacement maps in this mod. Only normal and spec maps. 

5 hours ago, peter_spy said:

Btw. I hope you remember that normalmap is not a bitmap in traditional sense, so you can't apply traditional 2d image techniques to it (e.g. sharpening).

It's certainly not advisable

Edited February 29, 2020 by Hugo Lobo

[nbohr1more]

I like those high contrast wallpapers!

That said, I think the bumpy wallpaper design was meant to be a showcase of the lighting system.

Bumpy wall papers work well in dark areas where AI or the player have movable light sources.

We currently have done some tentative work to get RGTC normal map compression working so perhaps we will

have more VRAM for better diffuse textures eventually. The down side (of course) is a major reason for getting

better texture compression was to lower Load Times. Keeping the long load times while increasing texture quality is

sure to irk many long time complainers about that issue...

[peter_spy]

You can have bigger textures these days and the only downside so far is be the first loading time, as the game loads everything at map start (this will change in the future too). 1 GB VRAM cards became mainstream years ago, and this basically allows all your textures to be 1k-2k without performance drops.

[HMart]

6 hours ago, VanishedOne said:

By displacement, do you mean heightmaps (the ones with filenames ending _h that get invoked with heightmap() to convert them to normal maps at runtime)? If the engine supports displacement mapping that definitely passed me by.

I seem to recall talks in the old forum about implementing parallax occlusion mapping in TDM, but i'm not really sure if the work was ever done.  

So IMO the team should seriously discuss this again, why, because is a petty that something that has the potential to improve some textures depth even more is just forgotten and today GPUs are much faster for it. Also especially when there's already made shaders for it for Doom 3, Prey 1 and Quake 4, but those are ARB shaders. But the idtech 4 engine i'm using, fhdoom, supports relief mapping and is a GLSL engine, someone from the team could look at it and transfer it to TDM. I would if I knew enough about rendering code, that I don't.  

Edited February 29, 2020 by HMart

[duzenko]

I must be half-blind because I don't see significant difference on those screenshots

[LDAsh]

There are a lot of "HQ Texture Mod" packs for everything coming out (and a lot more to come!) thanks to the growing popularity of Topaz Gigapixel AI.

 

I see a future of a lot of wasted HDD space for very little discernible improvement.  We already have texture filtering since ~1997, any software that attempts to take those sharp original texels and blur them into each other is doing so in the face of that.  That's what folks need to understand.  Any 'upscaling' is inevitably going to take artistic effort and the same skills as it did to originally create the asset.

[Hugo Lobo]

56 minutes ago, LDAsh said:

I see a future of a lot of wasted HDD space for very little discernible improvement.  We already have texture filtering since ~1997, any software that attempts to take those sharp original texels and blur them into each other is doing so in the face of that.  That's what folks need to understand.  Any 'upscaling' is inevitably going to take artistic effort and the same skills as it did to originally create the asset.

There will be a time in the near future that you will not need to AI upscale textures manually. The engine will do it for you while playing. Currently, the process is too heavy and slow for this to happen, but give some some years and it will become common.

When you say "Any 'upscaling' is inevitably going to take artistic effort and the same skills as it did to originally create the asset. " i suppose you are talking about human effort? It this is the case, my opinion is that to make a statement like that, you first have to understand what art is and if it can be replicated by computers. The answer is yes, of course. We already know this. Specially because "art" is such a subjective thing.   

[LDAsh]

I would like to politely and respectfully disagree.

Yeah, there are some emulators that already attempt to do this kind of stuff with textures, but the results are never pretty when weighed against the performance hit.  It's interesting to toy with, but usually get turned off for long serious play sessions.

A human being is always going to bring better and more significant details into the scene than a computer program ever can.  A human already has all of the filters and effects the computer can use, and infinitely more.  The real question is speed, time and money - that's the only reason for considering these approaches.

Here's a comparison of Gigapixel in use, it's basically like using a median filter and sharpen/soften combination, much like the HQ*X/'SuperEagle' filters you can find in modern emulators, which FYI many people think makes games look ugly:-
https://media.indiedb.com/images/downloads/1/191/190440/F.png
Judge for yourself, if it's worth 4X the video memory demand or not, compared to what dedicated artists could do with the same demands.  Once again, quite a huff made over AI that doesn't pay off on closer inspection quite as much as the runaway imagination of it.  The thing is - why not batch-process every texture in the game, if humans are never going to do it.

In the future, I can see developers forcing the end-user to compile textures for themselves during install, using something similar to Allegorithmic Substance, if players want 8K or even 16K+ textures without needing to download 500GB worth of content - but that's not upscaling and that's during install, not runtime, because there would be no need for that even if it was possible.

 

Edited March 7, 2020 by LDAsh

[Hugo Lobo]

18 hours ago, LDAsh said:

Yeah, there are some emulators that already attempt to do this kind of stuff with textures, but the results are never pretty when weighed against the performance hit.  It's interesting to toy with, but usually get turned off for long serious play sessions.

It's a matter of personal taste. I use upscaling filters all the time on dosbox and some console emulators - mostly xbrz - and i am happy with it.

18 hours ago, LDAsh said:

A human being is always going to bring better and more significant details into the scene than a computer program ever can.  A human already has all of the filters and effects the computer can use, and infinitely more.  The real question is speed, time and money - that's the only reason for considering these approaches.

Are you sure about this? Search for "ai created paintings" or "ai created textures" and you will be surprised.

Also, it's a fact that more and more, artists (be it texture artists, game developers, ....) will rely on AI to make their works. For instance, the process of creating a game area (lets say a city and its details) manually is very time consuming. AI is helping with it.

 

18 hours ago, LDAsh said:

Here's a comparison of Gigapixel in use, it's basically like using a median filter and sharpen/soften combination, much like the HQ*X/'SuperEagle' filters you can find in modern emulators, which FYI many people think makes games look ugly:-

https://media.indiedb.com/images/downloads/1/191/190440/F.png
Judge for yourself, if it's worth 4X the video memory demand or not, compared to what dedicated artists could do with the same demands.  Once again, quite a huff made over AI that doesn't pay off on closer inspection quite as much as the runaway imagination of it.  The thing is - why not batch-process every texture in the game, if humans are never going to do it.

 

Ai neural network upscaling has little to do with "median filter and sharpen/soften combination" or HQ*X/'SuperEagle scaling algorithms. Please check you facts. I could post a link or two about the matter, but you can easily google it. But i agree with you about that screenshot. That is really bad upscaling right there.  Be it because of the software itsef, the upscaling algorithm or that the machine did not have yet time to learn how it should upscale a particular texture material, i don't know. I never used Gigapixel.

[LDAsh]

Yeah, I've seen the Mandelbrot-like art created by AI, and some more advanced stuff after AI has been fed with references to guide itself, to the point of outright ventriloquism.

We're a very long way away from replacing a team of game developers with software.  It seems you've been reading a lot of the fantastical articles and are all excited, like many others, but I'm just saying that I'm personally far from being overwhelmed by much of it.  It's mostly boomer-noob media parroting Silicon Valley geeks looking to bolster more investment opportunities.  They are just mostly other tools to use and that is all.  I don't want you to get worked up over it so I won't reply anymore, we can easily just disagree.

I just want to clarify about the upscaling techniques (as relevant to the thread) - if a software can automagically perform a task (or a million) it's because a human programmed it to, because another human already developed that technique.  It's not something mystical, and that kind of thinking bugs me.  Hence why I mentioned emulator filters and Gigapixel results, they don't look so similar by accident.  If that's me being misinformed, well, I'll just carry on actually manually working, and wish you the best of luck.

[NeonsStyle]

Thank you

[MirceaKitsune]

Judging from the screenshots, I honestly feel this one should be merged into vanilla TDM and replace the originals, or at least some of the textures in it should. Occasional low-res textures are an annoyance and such a patch is super welcome IMHO. I'm kind of afraid to try it because I might get too used to it and not want to give it up if an update ever breaks them

[nbohr1more]

2 hours ago, MirceaKitsune said:

Judging from the screenshots, I honestly feel this one should be merged into vanilla TDM and replace the originals, or at least some of the textures in it should. Occasional low-res textures are an annoyance and such a patch is super welcome IMHO. I'm kind of afraid to try it because I might get too used to it and not want to give it up if an update ever breaks them

See here:

for a reason why we might "not" want to bump the VRAM budget for all texture assets...

 

[MirceaKitsune]

51 minutes ago, nbohr1more said:

See here:

for a reason why we might "not" want to bump the VRAM budget for all texture assets...

 

I see what you mean. This is why texture compression or at least downscaling would be a good idea. Hopefully a way around it can be found someday.

If not I can live with this being a mod, especially since you only need to put the pk4 in and that's it! Just hope there's enough guarantee that it can be maintained. Maybe it can become an official addon, still separate but maintained by the devs? I probably should not throw so many ideas at once again xD

[Hugo Lobo]

18 hours ago, MirceaKitsune said:

Judging from the screenshots, I honestly feel this one should be merged into vanilla TDM and replace the originals, or at least some of the textures in it should. Occasional low-res textures are an annoyance and such a patch is super welcome IMHO. I'm kind of afraid to try it because I might get too used to it and not want to give it up if an update ever breaks them

Thank you for your kind words MirceaKitsune. Have fun with the mod! And don't be afraid of using it. It is not likely that some future version would break it. That said, i am monitoring the new pk4 versions every time a new version of the game comes out, so we don't get funny stuff while playing.

On a side note, after some months playing other things, i returned to TDM and so my quest to enhance more textures. I will upload a new version with more screenshots in a near future.