Convert normal maps to RGTC?

[STiFU]

Do you have a converter for RGTC @duzenko? I'd like to conduct an experiment so see how loading times are affected. I assume their is no code support needed to load RGTC normals at this point?

[duzenko]

3 hours ago, STiFU said:

Do you have a converter for RGTC @duzenko? I'd like to conduct an experiment so see how loading times are affected. I assume their is no code support needed to load RGTC normals at this point?

Good question that I can't answer fully ATM. D3 offers two automatic conversion options:

• via the r_writeCompress_something cvar that is supposed to let the GL driver do the conversion and then simply bit copy the result from the VRAM. You'd think that driver's on-the-fly compression is fast but image quality might suffer.
• using an external command-line tool. Supposedly this is the recommended option. We still need to decide which 3rd party compressor we want to use and how to wire it to TDM (command line params, etc)

Feel free to try either of these but keep in mind that texture loading is only a fraction of total map loading time.

I'd like to get the RGTC conversion completed but afraid to touch it before 2.08 is out

[STiFU]

40 minutes ago, duzenko said:

Feel free to try either of these but keep in mind that texture loading is only a fraction of total map loading time.

How can you tell? Did you profile map loading? If so, I'd be very interested to see the results!!

Quote

I'd like to get the RGTC conversion completed but afraid to touch it before 2.08 is out

Yes, this is definitely a 2.09 task.

[duzenko]

20 minutes ago, STiFU said:

How can you tell? Did you profile map loading? If so, I'd be very interested to see the results!!

Yes, this is definitely a 2.09 task.

Unfortunately VS2019 profiling does not work for me

But image_preload 0 will do the job - try it.

[stgatilov]

 

17 hours ago, STiFU said:

Do you have a converter for RGTC @duzenko? I'd like to conduct an experiment so see how loading times are affected. I assume their is no code support needed to load RGTC normals at this point?

If you have time to have fun, you can try to implement your own in something like Python. All those texture compression formats are so simple compared to PNG and JPG, that writing a basic converter is a one-day problem

13 hours ago, duzenko said:

Unfortunately VS2019 profiling does not work for me

I'm afraid CPU profiling is not the right way to conduct experiments in this case. There are many things involved which you won't be able to properly measure with profiler: disk accesses, OpenGL driver work, GPU transfers, etc. I suggest doing more complicated measurements with temporarily disabling some parts in the code. I guess most of the time should be taken by texture loading, models loading, and sound loading (also covered in zip compression unless you are on SVN).

[duzenko]

2 hours ago, stgatilov said:

 I guess most of the time should be taken by texture loading, models loading, and sound loading (also covered in zip compression unless you are on SVN).

.map/.proc/.cm files can take a while to load, as well as AAS parsing

Then, there are also scripts, def's, materials

[HMart]

On 3/26/2020 at 8:01 PM, duzenko said:

Good question that I can't answer fully ATM. D3 offers two automatic conversion options:

• via the r_writeCompress_something cvar that is supposed to let the GL driver do the conversion and then simply bit copy the result from the VRAM. You'd think that driver's on-the-fly compression is fast but image quality might suffer.
• using an external command-line tool. Supposedly this is the recommended option. We still need to decide which 3rd party compressor we want to use and how to wire it to TDM (command line params, etc)

Feel free to try either of these but keep in mind that texture loading is only a fraction of total map loading time.

I'd like to get the RGTC conversion completed but afraid to touch it before 2.08 is out

What about ATI/AMD compressionator? It was the tool that idsoftware used to compress normal maps for Doom 3 and splash damage for quake wars, you can find the old version used in Doom 3 here.

Or perhaps ODCompress from Overdose?

[duzenko]

48 minutes ago, HMart said:

What about ATI/AMD compressionator? It was the tool that idsoftware used to compress normal maps for Doom 3 and splash damage for quake wars, you can find the old version used in Doom 3 here.

Or perhaps ODCompress from Overdose?

Does either of them support RGTC?

[New Horizon]

I don't see RGTC in the Overdose menu and I doubt compressonator would support it since it's an old program.

[HMart]

3 hours ago, duzenko said:

Does either of them support RGTC?

Don't know a thing about texture compression but latest compressionator "Can compress to a wide range of compression formats including ASTC, ATC, ATInN, BCn, ETCn, DXTn and swizzled DXTn formats, and supports conversion of textures between 8bit fixed, 16bit fixed and 32bit float formats."

 

Taken from this link:   "ASTC provides a single-channel and two-channel version, equivalent to RGTC's formats."

  

Edited March 29, 2020 by HMart

[nbohr1more]

As I recall, RGTC is just another name for Microsoft's BC4 or 5

This seems to confirm it:

RGTC1_RED = BC4U, RGTC1_SIGNED_RED = BC4S, RGTC2_RG = BC5U, RGTC2_SIGNED_RG = BC5S

 

[duzenko]

6 hours ago, HMart said:

Don't know a thing about texture compression but latest compressionator "Can compress to a wide range of compression formats including ASTC, ATC, ATInN, BCn, ETCn, DXTn and swizzled DXTn formats, and supports conversion of textures between 8bit fixed, 16bit fixed and 32bit float formats."

 

Taken from this link:   "ASTC provides a single-channel and two-channel version, equivalent to RGTC's formats."

  

Can't see any reference of a compression tool on that link

[HMart]

4 hours ago, duzenko said:

Can't see any reference of a compression tool on that link

Nor was that my intention I was just showing that I've read that ASTC is equivalent to RGTC but I don't know if is really true or not. 

Ha now I know what you meant by that, the first quote in dark gray from my reply above was from the latest AMD compressonator page, the link to it is on my reply above with the links for the tools, the other quote in white, is from another link, the "taken from this link" was referring to the white quote only.    

Edited March 29, 2020 by HMart

[revelator]

compressonator was opensourced and now seems to support it https://github.com/GPUOpen-Tools/Compressonator

[revelator]

It uses ASTC which is aproximatly the same as RTCG.

The downside is that ASTC is an opengl ES feature so might be a bit lobotomized.

[OrbWeaver]

I haven't been keeping up with the normal map compression changes, but I am confused about the current correct way to produce a DDS normal map.

Exporting from GIMP, both DXT1 and DXT5 work fine, but have the blocky artifacts I associate with DXT-compressed normal maps. "RXGB" produces a normal map which is clearly missing one dimension, so I guess the DXT5 swizzling has been disabled.

From the link @nbohr1more posted it seems like "BC5 / ATI2 (3dc)" (which exists in the GIMP plugin) ought to be equivalent to RGTC, but this only gave me garbage results. I don't know if this is the wrong format to choose or if it is a limitation of the GIMP plugin itself.

[MirceaKitsune]

dds is a standard format nowadays, and since those pesky patents on it expired it should be safe to use in FOSS projects. Gimp seems to work fine with it (importing and exporting) so it should be okay for artists editing the graphics. Especially if it's going to improve those huge loading times and RAM usage, it does feel worth considering to me.

[OrbWeaver]

We're already using DDS format, and have been doing so since the initial release of the mod.

The recent comments in this thread are about using RGTC compression within DDS for normal maps, which should give better quality than regular DXT1 but is not necessarily as widely supported by editing tools.

[MirceaKitsune]

6 hours ago, OrbWeaver said:

We're already using DDS format, and have been doing so since the initial release of the mod.

The recent comments in this thread are about using RGTC compression within DDS for normal maps, which should give better quality than regular DXT1 but is not necessarily as widely supported by editing tools.

I see. Are Gimp / Krita / Photoshop able to load and save that, or at least plan on being able to? If yes then I'd say go for it.

Gimp is the image editing software I actively use so I gave it a try. I can see there's no export option for RGTC which does raise a concern for this plan.

[duzenko]

1 hour ago, MirceaKitsune said:

I see. Are Gimp / Krita / Photoshop able to load and save that, or at least plan on being able to? If yes then I'd say go for it.

Gimp is the image editing software I actively use so I gave it a try. I can see there's no export option for RGTC which does raise a concern for this plan.

You don't _need_ to convert your custom textures to RGTC

This thread is about the core assets only

[MirceaKitsune]

16 minutes ago, duzenko said:

You don't _need_ to convert your custom textures to RGTC

This thread is about the core assets only

I know. I was thinking whether this format allows them to be editable and accessible by mainstream editing tools, in case someone wants to look at them or make their own versions for development purposes. I'd think that should be possible, so people who need access to them won't need to struggle with conversion tools.

[OrbWeaver]

5 hours ago, MirceaKitsune said:

I see. Are Gimp / Krita / Photoshop able to load and save that, or at least plan on being able to? If yes then I'd say go for it.

Gimp is the image editing software I actively use so I gave it a try. I can see there's no export option for RGTC which does raise a concern for this plan.

Supposedly yes (I believe it's equivalent to "3DC+" in the GIMP plugin), however as I mentioned I have not been able to get this to work at all with the latest build. Only plain DXT-compressed normal maps render correctly.

[duzenko]

2 minutes ago, OrbWeaver said:

Supposedly yes (I believe it's equivalent to "3DC+" in the GIMP plugin), however as I mentioned I have not been able to get this to work at all with the latest build. Only plain DXT-compressed normal maps render correctly.

How did you test this?

[OrbWeaver]

Just now, duzenko said:

How did you test this?

I took a normal map which was working, then saved it using the GIMP plugin in 3DC+ format (I also tried using the Compressonator with various formats which looked like RGTC, e.g. "ATI2N"). After this, the normal map would either show only a single channel (e.g. horizontal bumps are visible but vertical bumps are missing), or no bumps at all; I don't recall exactly which.

[duzenko]

11 hours ago, OrbWeaver said:

I took a normal map which was working, then saved it using the GIMP plugin in 3DC+ format (I also tried using the Compressonator with various formats which looked like RGTC, e.g. "ATI2N"). After this, the normal map would either show only a single channel (e.g. horizontal bumps are visible but vertical bumps are missing), or no bumps at all; I don't recall exactly which.

Could you upload a same texture for me too look at?