Improved DDS File Handling

[greebo]

I hadn't the time to do this yet. The fix is in SVN now.

[AluminumHaste]

I hadn't the time to do this yet. The fix is in SVN now.

 

Appreciated, my map runs a lot better now too, less memory and half the load time :wub:

You are doing AMAZING work on this tool Greebo, ID should hire you to work on their editor after this is all done, their's sucks balls compared to DarkRadiant.

Hell even going back to Quake 3 editing with the last release of GTKRadiant is such a huge step back from DR that I end up ripping my hair out.

[greebo]

Thanks for the words.

 

Realistically though, the work we're doing here is nothing in comparison with the work id is doing on their new "studio" developed for the Rage engine. Have you seen those editor demo videos?

[Tels]

Thanks for the words.

 

Realistically though, the work we're doing here is nothing in comparison with the work id is doing on their new "studio" developed for the Rage engine. Have you seen those editor demo videos?

 

Yeah, but does it run in Linux? Still, you are doing awesome work, please keep it up!

[greebo]

Yeah, but does it run in Linux?

Most probably not. The capability to run in Linux is admittedly one of the strong points of DarkRadiant.

 

I also got the impression (I think I read that on d3world), that John Carmack was somewhat losing interest in developing for the Linux platform. Would be a shame anyhow. (Not that this would be the Large Gaming Market in the first place, it's rather a symbolic gesture, in my opinion.) But still, the tools they're developing look really great, and they're showing at which rate a development studio can produce new stuff.

[Tels]

I tried the new version under Linux and it compiled fine. But it doesn't work, it segfaults when loading blackhear_manor.map.

 

Watching the loading screen, the primitives finish, and then with the entities it crashes. Sorry, have not further info atm.

 

Loading the map with the old version worked, it took 2 minutes 16 seconds and consumed 1128 Megabytes (since my laptop only has 1Gbyte, it swapped out, hence the very long loading time).

[Flanders]

Loading blackheart manor gives me the same on my windows system.

[greebo]

Somebody, report it on the tracker please, I'll look into that sometime.

[Tels]

Somebody, report it on the tracker please, I'll look into that sometime.

 

Done:

 

http://bugs.angua.at/view.php?id=1583

[Tels]

Done:

 

http://bugs.angua.at/view.php?id=1583

 

Done now a test with the newest version, which now loads Blackheart Manor without crash

 

Here are the values:

 

20500 te 20 0 836m 703m 22m S 7 70.2 1:58.41 darkradiant

 

Meaning it uses only 836 Mbyte of memory. It took about 1 minute and 54 seconds to load the map.

 

Compare to:

 

Loading the map with the old version worked, it took 2 minutes 16 seconds and consumed 1128 Megabytes (since my laptop only has 1Gbyte, it swapped out, hence the very long loading time).

 

 

The time is not much less than before, because as you can see from the 703 resident memory value above, my system still swapped out 150 Mbyte and this slowed down the loading stage of the entities. With 1.5 Gbyte memory the map would probably load much faster. Anyway, good work!

 

Here is one question that popped into my head: Do the editor textures in JPG format actually help? Because if we already have JPG for most textures, why does the DDS loading make so much difference? Also, since JPG need to be decompressed and then recompressed as DDS with mipmaps, wouldn't they actually slow down things down, despite them being 256x256 instead of 512x512 etc?

 

If we could get rid of the JPG files, we would save a ton of files on SVN and releases, and if that makes actually map loading faster and use less memory, that would be a win all around. Greebo, do you have any insight into this?

[SneaksieDave]

@Tels: I was actually wondering exactly the same thing, but I was reluctant to stir it up after such a good improvement. It is probably worth looking into.

[Springheel]

Before anyone leaps in and deletes all the .jpgs, keep in mind that modeling programs cannot load .dds files, so modelers are using the editor versions of textures when they model.

[Flanders]

Before anyone leaps in and deletes all the .jpgs, keep in mind that modeling programs cannot load .dds files, so modelers are using the editor versions of textures when they model.

A lot of modelling programs can handle dds files. And if not, why not convert the dds into something else so you can have a larger/sharper image for uv mapping?

[Springheel]

A lot of modelling programs can handle dds files.

 

Lightwave doesn't. The last time I looked for a decent plugin I couldn't find one for my version, though I admittedly did not spend a lot of time looking.

 

I'm not saying this is a reason not to remove the .jpg files, if they serve no other purpose. I just don't want anyone to leap in and do it without talking to the modelers so we can figure something else out.

[Tels]

Lightwave doesn't. The last time I looked for a decent plugin I couldn't find one for my version, though I admittedly did not spend a lot of time looking.

 

I'm not saying this is a reason not to remove the .jpg files, if they serve no other purpose. I just don't want anyone to leap in and do it without talking to the modelers so we can figure something else out.

 

That's a good point to take into consideration. One thing tho, the JPGs only have 256x256 resolution "to keep space down" and this is very blurry, so modellers probably are better of to use the DDS or even the TGAs? Or does this not matter much for modeling? (you can tell I never done it

 

Lightwave:

 

This:

 

http://forums.cgsociety.org/archive/index.php/t-155051.html

 

talks about this containing one, but I can't check:

 

http://www.dstorm.co.jp/english/plugin/object.htm#DirectX2

 

(scroll to top to see list of plugins)

[greebo]

I can see the point about the modeling program - I've been using the stone textures for my models as well, and without JPG editor images, I would have been quite lost. Blender doesn't support DDS files out of the box, as far as I could see.

 

As for the question about JPG vs DDS: yes, DDS editor images would mean faster loading times, as they can be directly pushed to the GL routines without prior decompression.

 

To get a better feeling about what we're talking about an actual benchmark would be very helpful, loading Saint Lucia using JPG vs loading it using DDS editor images.

[Flanders]

Blender doesn't support DDS files out of the box, as far as I could see.

 

It does. Maybe not early versions of blender, but it's not so hard to get the latest blender version .

[greebo]

Really? I'm using Blender 2.48 (64 bit), and it always dumps an "unknown file format" to the console when trying to use a DDS image.

[Tels]

To get a better feeling about what we're talking about an actual benchmark would be very helpful, loading Saint Lucia using JPG vs loading it using DDS editor images.

 

It should be possible to do this by cloning the entire darkmod directory, then deleting all .JPG files, then loading the map there versus the original. I can attempt this and post back here, but it might need an hour time to copy the entire folder.

 

Edit: Just to be clear, there are three methods we can try:

 

* current: most editor images are 256x256 JPG files (uses little disk space)

* editor images are 256x256 DDS images (uses very probably more disk space)

* editor images are non-existant, the normal texture (512x512 or 1024x1024) are used directly (uses no additional disk space)

 

Which one needs how much loading time and uses how memory is not yet known.

 

I am attempting the last one as this is easiest to try. The second variant needs a script to convert all JPG to DDS, which I don't have time right now

[Flanders]

Really? I'm using Blender 2.48 (64 bit), and it always dumps an "unknown file format" to the console when trying to use a DDS image.

 

I have 2.48a 32 bit and I'm sure it can handle dds files. I don't remember installing any plugins for it.

[greebo]

Ok, I'll be sure to give the 32-bit version a shot next time I'll do some modeling.

[Tels]

Okay, here is the first data:

 

First, there seems to be another memory leak when switching maps. For instance, when you load cath.map right after start, DR uses 404 Mbyte of memory.

 

If you instead load saintlucia.map, DR uses 865 Mbyte of memory. When you then just load cath.map, DR uses 883 megabyte - which is even more than saintlucia and a lot more than 400 Mbyte. Maybe old data from the first map is not cleared away properly?

 

Then I did a test loading SL with the JPG files in place, and without them. It looks like in that case DR refuses to load the diffusemaps in place, and instead just displays "shader not found". Greebo, would it be possible to make DR fall back to the diffusemap instead if the qer_editorimage cannot be found?

 

Here is the data:

 

Saintlucia:

16768 te		20   0  759m 626m  22m S   10 62.5   1:27.19 darkradiant

With:

16992 te		20   0  865m 727m  20m S	3 72.6   1:46.22 darkradiant

 

That means the editor image textures use up about 100 mybte or roughly 1/8 of all the memory. Since we didn't actually use any textures in their place, this is not practical, but it puts up an upper limit on how much memory we can save. And 1/8 isn't much

 

I wonder where the rest of the memory goes. There seem quite a few textures with no JPG version yet, so at least that would be a way to improve things.

 

Now to find out how much memory using the DDS in place would need means I need to strip out the qer_editorimage lines from all material files, and thats gonna take a me a while. Stand by

[Tels]

I used the following monster to remove the qer_editorimage lines in all material files:

 

find materials/ -type f -print0 -name "*.mtr" |xargs -0 perl -p -i.bak -e 's#qer_editorimage.*(models|darkmod).
*##'

 

The result is inconclusive, however. Memory consumption is still 759 Mbytes, however, all the textures where DDS files are used are now appearing white, very probably because my laptops shitty graphic can't render DDS files. I don't know what the memory consumption would be if my laptop could render DDS.

 

Anyway, that alone is a reason to not get rid of the JPG files, because my laptop is the only machine where DR still works (it no longer compiles on my 64bit PC for several months

[Tels]

Hm,

 

instead of getting rid of the JPGs I think it would also be possible to improve my check-materials.pl script. It could be more stable and then be used to generate the missing JPG files.

 

Also, this technique:

 

http://www.imagemagick.org/Usage/annotating/#wmark_image

 

might be used to draw some fake bumpmapping on the editor textures. That would give them more depth, and it would also mean that the stone textures where the features are only defined by the normal map, could be generated automatically.

 

Unfortunately, I don't know yet how to convert a normal map into a bumpmap....