Jump to content
The Dark Mod Forums

Convert normal maps to RGTC?


duzenko
 Share

Recommended Posts

It's interesting that people like to mention the forceHighQuality flag and very rarely remember about uncompressed/highquality flags

I think you should not use the former if you don't know the difference with the latter

Link to comment
Share on other sites

uncompressed / highQuality — Do not compress this image in medium quality mode
forceHighQuality — Do not compress this image in low quality mode

Given that we do not expose the concepts of "medium quality" or "low quality" in TDM, I'm not sure how this difference is applied in practice.

Link to comment
Share on other sites

32 minutes ago, OrbWeaver said:

uncompressed / highQuality — Do not compress this image in medium quality mode
forceHighQuality — Do not compress this image in low quality mode

Given that we do not expose the concepts of "medium quality" or "low quality" in TDM, I'm not sure how this difference is applied in practice.

There's a cvar to ignore highQuality flag that does not apply to forceHighQuality

In reality forceHighQuality should not be used on anything that highQuality is enough for because

  • it robs the end user of options
  • it's less typing and parsing
Link to comment
Share on other sites

I implemented RGTC compressor in SIMD (generic and SSE2), as well as mipmapping.
The compressor uses min/max value of a block as min/max ramp points with 6 intermediate ramp points between them, and snaps each color value to the closest ramp point (well, almost closest sometimes).

On the shitty i3-5005U CPU which I have now, compressing 2K x 2K normal map takes 28 ms (with all mipmaps included). This means 7 ms per 1K x 1K texture, or 7 ns per pixel.
Given that loading the 2K x 2K TGA image from PK4 file takes about 200 ms, I think 30 ms is negligible in comparison.

The only question is quality. It seems that naive compressor should be good enough for RGTC.
I guess I'll wrap up the changes and commit them, so that someone else could test it on hard cases if there are known ones. It took me several days here to download 10 MB installer of DDS Viewer, so I won't be able to test it myself 😥

  • Like 1
  • Thanks 3
Link to comment
Share on other sites

That sounds awesome! Can we have a reminder of what the cvar is to test the new implementation? Though I assume all existing normal maps in the assets need to be converted, so I take it the next dev snapshot might not benefit just by enabling the setting unless someone batch-converts every normal texture as well.

Link to comment
Share on other sites

I have committed all my changes to source code repo.
New software RGTC compressor is enabled for everyone. I did not provide a way to disable it, but you can easily find it if you want.

There is nothing like image_mipmapMode in the code already, maybe @duzenko removed it later.

  • Thanks 2
Link to comment
Share on other sites

Compression on, 0.7GB VRAM used

[url=https://postimg.cc/jnK3Mz6F][img]https://i.postimg.cc/hv7FPsXc/image.png[/img][/url]

image.png

Compression off, 0.9GB VRAM used

[url=https://postimg.cc/pmhNhJsf][img]https://i.postimg.cc/13MydW6j/image.png[/img][/url]

image.png

Spoiler

FFS what's going on with image hosting links?

 

  • Like 1
Link to comment
Share on other sites

27 minutes ago, duzenko said:

@stgatilovI think we want to ignore image_useCompression here

It has been only affecting non-normal textures before AFAIR

Yes, it makes sense.
But make sure to disable it woth in HandleCompression and in SelectInternalFormat.

I'd say we can probably refactor SelectInternalFormat into two parts: the one which is O(1) and return format without any analysis, and the one which can additionally check the image data for final internal format. In this case I'd call the first function in HandleCompression instead of writing the same conditions again.

Link to comment
Share on other sites

If I understand correctly, the advantage of doing RGTC compression in software is performance, because it can be done in parallel rather than image-by-image on the GPU?

If so, I guess any testing should focus on performance (particularly the time to load a new mission from scratch), rather than GPU memory usage which should be the same whether RGTC compression is done in software or by OpenGL.

Link to comment
Share on other sites

7 minutes ago, OrbWeaver said:

If I understand correctly, the advantage of doing RGTC compression in software is performance, because it can be done in parallel rather than image-by-image on the GPU?

If so, I guess any testing should focus on performance (particularly the time to load a new mission from scratch), rather than GPU memory usage which should be the same whether RGTC compression is done in software or by OpenGL.

With dynamic image loading driver-side compression is just too slow and results in stuttering

Even if not used widely ATM it's already technologically superior

Link to comment
Share on other sites

39 minutes ago, stgatilov said:

Yes, it makes sense.
But make sure to disable it woth in HandleCompression and in SelectInternalFormat.

Huh, it seems I was wrong about that

Completed: At revision: 9563 

Quote

I'd say we can probably refactor SelectInternalFormat into two parts: the one which is O(1) and return format without any analysis, and the one which can additionally check the image data for final internal format. In this case I'd call the first function in HandleCompression instead of writing the same conditions again.

Also a good idea

Link to comment
Share on other sites

2 hours ago, MirceaKitsune said:

This is really great! Any noticeable difference in loading time too? That's one of the main aspects I was interested in, TDM still has some really long load times compared to many other engines.

Try image_preload 0

Link to comment
Share on other sites

47 minutes ago, duzenko said:

Try image_preload 0

Will keep it in mind. IIRC though that means the game will stutter whenever an object with a previously unloaded texture comes into view, so I'd likely get slowdowns during the game instead.

Link to comment
Share on other sites

1 hour ago, MirceaKitsune said:

Will keep it in mind. IIRC though that means the game will stutter whenever an object with a previously unloaded texture comes into view, so I'd likely get slowdowns during the game instead.

The current svn performs quite well actually

Link to comment
Share on other sites

Faster conversion is definitely welcome but I still think we should consider pre-compressing some of these:

1) Detailed wood textures. It is unlikely anyone will fiddle with image program functions on these

2) Normals baked from models ( on model assets such as bellows )

Of course, as soon as I say this some mission author will make an exotic material which requires the uncompressed images... We can still handle those rare exceptions on a case by case basis

Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Link to comment
Share on other sites

I bet you guys already know this but if you don't, just for kicks I set image_preload 0 and image_useCompression 0 and the missions I tried load way quicker! Is obvious why but I underestimated how impactful this options were!

And I didn't felt any performance loss from it. I hope this is true for all! 

It seems I will play with those off going forward. :) 

Btw I also found that using preload 0 and compression ON, causes the game to start with no textures for a few micro seconds, similar to some UE3 games! TDM is now among the big league engines! ;D kd

Is not a big problem and the same happens with compression off but for a lower time, unfortunately, the game has visible pauses, when opening doors and the game as to load and compress the textures on the fly. 

 

edit: I didn't played the latest version with stgatilov new RGTC compressor in SIMD, so things could be different on it. 

Edited by HMart
Link to comment
Share on other sites

I tried loading a number of larger missions last night. I definitely saw shorter load times. The candle animation for A Score to Settle never completed so quickly. I'll need to use a timer to get definitive results.

Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Link to comment
Share on other sites

On 8/20/2021 at 2:13 AM, HMart said:

I bet you guys already know this but if you don't, just for kicks I set image_preload 0 and image_useCompression 0 and the missions I tried load way quicker! Is obvious why but I underestimated how impactful this options were!

And I didn't felt any performance loss from it. I hope this is true for all! 

It seems I will play with those off going forward. :) 

Btw I also found that using preload 0 and compression ON, causes the game to start with no textures for a few micro seconds, similar to some UE3 games! TDM is now among the big league engines! ;D kd

Is not a big problem and the same happens with compression off but for a lower time, unfortunately, the game has visible pauses, when opening doors and the game as to load and compress the textures on the fly. 

 

edit: I didn't played the latest version with stgatilov new RGTC compressor in SIMD, so things could be different on it. 

With uncompressed assets on ssd it's quite fast but then again full preload is not too slow either

Link to comment
Share on other sites

8 hours ago, duzenko said:

With uncompressed assets on ssd it's quite fast but then again full preload is not too slow either

oh yes a good internal ssd or even a nvem/M.2 ssd can be fast even with preload on, but I have the game in a external USB 3.0 2.5 HDD, so preload may impact me more I assume. 

Link to comment
Share on other sites

17 hours ago, HMart said:

oh yes a good internal ssd or even a nvem/M.2 ssd can be fast even with preload on, but I have the game in a external USB 3.0 2.5 HDD, so preload may impact me more I assume. 

Preload is definitely preferred for HDDs

Link to comment
Share on other sites

The new dev build is available, so now everyone can test the new approach.

To be honest, I forgot to add a cvar to enable/disable the new RGTC code, so on the new dev build it is always on.
I'm afraid you'll have to install old/new versions side-by-side if you want to make a comparison.

And if you decide to compare, note that OS has a very large cache.
Basically, all the unused physical RAM is used as cache for the recently loaded files, so the whole TDM easily fits into it.
So if you load TDM second time without reboot, then the data will be loaded from RAM instead of HDD/SSD.
Make sure to not compare cold and hot loading times to each other!

  • Like 1
Link to comment
Share on other sites

23 minutes ago, stgatilov said:

The new dev build is available, so now everyone can test the new approach.

To be honest, I forgot to add a cvar to enable/disable the new RGTC code, so on the new dev build it is always on.
I'm afraid you'll have to install old/new versions side-by-side if you want to make a comparison.

image_useNormalCompression does that?

You mean there's no more driver compression, only ours or none

  • Like 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share


  • Recent Status Updates

    • STiFU

      We are taking our son on his very first holiday trip to see the sea for the first time. 🙂 Will be back in a week.
      · 0 replies
    • Gilkar

      When I was a young man my father was so ignorant I could hardly stand to have him around. As I grew older I was amazed at how much the old man had learned in such a short time.
      · 1 reply
    • jaxa

      RTX 3090 Super, RTX 3070 Ti 16 GB, RTX 2060 12 GB
      https://wccftech.com/nvidia-launching-rtx-3090-super-rtx-3070-ti-16gb-and-rtx-2060-12gb-by-january-2022/
      · 0 replies
    • duzenko

      CPU benchmark time - compiling DarkRadiant (2nd run)
      i5 8600K 6C/6T@4.4GHz DDR4 2x2133MHz 9MB cache
      Parallel builds: 1. 3:57 Parallel builds: 6 (default). 2:28 r5 1600AF 6C/12T@3.3GHz DDR4 1x2666MHz 16 MB cache, temp folder on HDD
      Parallel builds: 1. 5:05 Parallel builds: 4. 2:47 Parallel builds: 6. 2:55 Parallel builds: 12 (default). 2:57
      · 6 replies
    • nbohr1more

      Status updates are back so it is also a good time to return to contests!
      https://forums.thedarkmod.com/index.php?/topic/21095-christmas-connections-contest-2021
       
      · 0 replies
×
×
  • Create New...