Jump to content
The Dark Mod Forums

Improved DDS File Handling


Recommended Posts

I restructured the texture handling code such that DarkRadiant can take advantage of those precompressed DDS texures. The texture loader doesn't decompress the DDS files anymore to push it to the GL system, but directly downloads the mipmaps now.

 

I have assembled a new snapshot for users to test.

 

http://72.8.59.188/TheDarkMod/DarkRadiant/...t-0.9.9pre1.exe

 

Please grab the snapshot and try to run a "diff" on this version (0.9.9pre1) against the latest release (0.9.8) in terms of memory usage and loading speed. Saint Lucia might be a good indicator, but any map with bold shader usage is suitable.

 

(I have 4 GB RAM myself, so I can only use Vista's built-in memory resource managers for testing (which indeed show a difference), but my swap file is hardly used in either case. That's why I'm asking for feedback.)

Link to comment
Share on other sites

I have some test results I think everyone will be pleased with. ^_^

 

This is using 0.9.8.1 and 0.9.9pre1.

 

Starting DR fresh, memory consumption with no map file loaded:

109,784 (Kb)

109,800

109,780

(Etc. Results were always consistent.)

 

Saint Lucia tests

0.9.8.1:

* Fully loaded, unlit mode: 1.41 Gb. (At this point, the actual memory reading for the DR process is not beneficial, as I'm using the hell out of the page file which is affected by every process on the system. Regardless, this is still of some use.)

 

* Render mode: it never got there. PF size went up to 1.99 Gb and then DR crashed out, 'terminated in an unusual way.' I'm not sure if this is a DR problem or a result of my memory settings (I have a 1.5Gb page file with 1Gb RAM, which is greater than 2Gb), but whatever. It didn't work.

 

0.9.9pre1 (are you ready for this?):

* Fully loaded, unlit mode: 1.08 Gb (!)

 

* Render mode: 1.74 Gb, no crash, could use cam to look around (though it was slow and consumed more memory at that point; point is, it worked.)

 

I'm a little confused by this great success, to be honest. I had thought that DDS files were only used for render mode? If so, why the great decrease (almost 1/3!) in memory consumption for unlit mode?

 

Note that I forgot to time the loads for SL. It takes extremely long -- long enough to go make breakfast and still have a few minutes left after -- for each step (loading, going to render mode, crashing), and in the meantime I cannot use my system at all lest I disturb the paging process, so I didn't repeat it.

 

Instead, I did further tests, with timing, on more manageable maps. :) The map used was bc_manor_12, which is large enough to give noticeable result, but small enough to stay within RAM, so I could watch the DR process and not need to use the page file at all. More good results:

 

0.9.8.1

Map loaded: 338,480 @ 47sec -> render mode: 520,688 @ 19sec (maybe slower because first time?)

338,624 @ 47sec -> render mode: 520,832 @ 16sec

338,456 @ 47sec -> render mode: 520,824 @ 16sec

 

0.9.9pre1

221,200 @ 43sec -> render mode: 385,664 @ 16sec

221,248 @ 43sec -> render mode: 385,700 @ 15sec

221,264 @ 43sec -> render mode: 385,704 @ 15sec

 

Slightly faster, and much better use of memory (again, about 1/3). Looks like a pretty conclusive slam-dunk to me. Awesome stuff, greebo. :wub:

Link to comment
Share on other sites

Very good! M y laptop will be pleased (it started to swap when loading SL because it has only one Gb memoty :)

 

I will do some timings another day.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

I'm a little confused by this great success, to be honest. I had thought that DDS files were only used for render mode? If so, why the great decrease (almost 1/3!) in memory consumption for unlit mode?

DDS files can be used in non-lighting rendermode just as fine. The only thing I changed is the way DDS files are loaded to GL.

 

The way it has been before was like this:

- Find DDS texture file

- Load the texture file into memory (eat 700 kB)

- Decompress the texture file to its full size (into some additional 4 MB)

- Load the file into OpenGL and create mip maps on the fly.

- Release the 700 kB chunk and the 4 MB chunk*.

 

The way it is now is:

- Find DDS texture file

- Analyse the DDS header and see which mipmaps are there

- Allocate memory (700 kB)

- Load mipmaps into memory, one after the other, leave data unchanged.

- Load mipmaps into OpenGL

- Release the 700 kB chunk.

 

* which was not fully working before, I could fix another bug here (delete vs delete[] call).

Link to comment
Share on other sites

I uploaded another pre-release build (the last one for today already), which contains the fixes for the "model" spawnargs, the MediaBrowser memory leak and the mousewheel crash in the ModelPreview.

 

http://72.8.59.188/TheDarkMod/DarkRadiant/...t-0.9.9pre2.exe

 

Very cool, gonna recompile tonight and then check it out :)

 

Compile doesn't finish, it throws some C++ error about templates *sigh*

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Very cool, gonna recompile tonight and then check it out :)

 

Compile doesn't finish, it throws some C++ error about templates *sigh*

 

I am getting the following error under Linux:

 

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../include -I../../libs -I/usr/include/gtk-2.0 -I/usr/lib/gtk-2.0/include -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/freetype2 -I/usr/include/libpng12 -I/usr/include/pixman-1 -g -O2 -DPOSIX -MT dds.lo -MD -MP -MF .deps/dds.Tpo -c dds.cpp -fPIC -DPIC -o .libs/dds.o

dds.cpp: In function ‘DDSImagePtr LoadDDSFromStream(InputStream&)’:

dds.cpp:58: error: template argument for ‘template<class _Alloc> class std::allocator’ uses local type ‘LoadDDSFromStream(InputStream&)::MipMapInfo’

dds.cpp:58: error: trying to instantiate ‘template<class _Alloc> class std::allocator’

dds.cpp:58: error: template argument 2 is invalid

dds.cpp:58: error: invalid type in declaration before ‘;’ token

dds.cpp:59: error: request for member ‘resize’ in ‘mipMapInfo’, which is of non-class type ‘int’

dds.cpp:69: error: invalid types ‘int[size_t]’ for array subscript

dds.cpp:104: error: request for member ‘size’ in ‘mipMapInfo’, which is of non-class type ‘int’

dds.cpp:105: error: invalid types ‘int[size_t]’ for array subscript

make[3]: *** [dds.lo] Error 1

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

I'd need to look further into this, maybe there is some #include missing or I forgot to add a typename specifier.

 

However, I don't have a working Linux environment available. I switched laptops and haven't bothered to set up Ubuntu since then, the old partition has been wiped from my old laptop.

Link to comment
Share on other sites

* Render mode: it never got there. PF size went up to 1.99 Gb and then DR crashed out, 'terminated in an unusual way.' I'm not sure if this is a DR problem or a result of my memory settings (I have a 1.5Gb page file with 1Gb RAM, which is greater than 2Gb), but whatever. It didn't work.

 

This is to be expected if you are running a 32bit Windows, because there the per-process memory is limited to 2Gbyte.

 

Windows splits the adressable 4Gbyte into 2Gbyte for the system, and 2 Gbyte for each process.

 

Here is a link that explains all that in detail :)

 

http://blogs.technet.com/markrussinovich/a...17/3155406.aspx

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

I am getting the following error under Linux:

 

I just moved the local definition of struct MipMapInfo out of LoadDDSFromStream() (in plugins/image/dds.cpp). GCC doesn't seem to allow that. It compiles now but, for me, the result crashes with a segmentation fault. That could have something to do with Boost on my system. I have to look into that.

Link to comment
Share on other sites

Ok, if moving the declaration out of the function body is doing the trick, I can do that. I was thinking about some refactoring to do that step anyway, so if this issue is resolved by that step, I'll just go ahead and do it.

Link to comment
Share on other sites

It's running fine now with this little fix (under openSuSE10.3 compiled from SVN).

Memory consumption is much smaller and I think the lighting render mode performs a lot better on my Linux notebook (this thing has an integrated ATI chip that barely runs Doom 3).

Link to comment
Share on other sites

This is to be expected if you are running a 32bit Windows, because there the per-process memory is limited to 2Gbyte.

Wow! I wondered if that was it, but dismissed it for not believing it. That sounds really, really bad, if that means SL is already pushing the limits for what DR can open and work with in XP. :unsure:

Link to comment
Share on other sites

Wow! I wondered if that was it, but dismissed it for not believing it. That sounds really, really bad, if that means SL is already pushing the limits for what DR can open and work with in XP. :unsure:

 

Yeah, basically XP (and any other 32bit OS) will die a slow death due this limit alone.

 

Windows 98 has a 256 MByte memory limit per process IIRC, so you can see why nobody can use it anymore except in some rare cases :)

 

However, DR wastes massive amounts of memory and I am confident that greebo can squeeze quite a bit more out of it. Keeping every vertex as its own object, f.i. surely must waste memory, esp. if you have billions of them :)

 

Also, D3 with SL uses "only" about 500 MByte so we have quite a bit room for bigger maps :)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Interesting, that would indeed suggest there is a lot more to be had. rubchin.gif

 

D3 has the advantage to not need to cache things outside the current portals rendered, tho. However, as I wrote, I am confident hat DR still wastes a lot of memory :)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

I am getting the following error under Linux:

 

Any news on this, I am still getting the error:

 

dds.cpp: In function ‘DDSImagePtr LoadDDSFromStream(InputStream&)’:
dds.cpp:58: error: template argument for ‘template<class _Alloc> class std::allocator’ uses local type ‘LoadDDSFromStream(InputStream&)::MipMapInfo’
dds.cpp:58: error:   trying to instantiate ‘template<class _Alloc> class std::allocator’
dds.cpp:58: error: template argument 2 is invalid
dds.cpp:58: error: invalid type in declaration before ‘;’ token
dds.cpp:59: error: request for member ‘resize’ in ‘mipMapInfo’, which is of non-class type ‘int’
dds.cpp:69: error: invalid types ‘int[size_t]’ for array subscript
dds.cpp:104: error: request for member ‘size’ in ‘mipMapInfo’, which is of non-class type ‘int’
dds.cpp:105: error: invalid types ‘int[size_t]’ for array subscript
make[3]: *** [dds.lo] Error 1

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

The change is not yet in the repository but you can do that yourself. It's a simple fix. Do you know C++?

 

Not really ;) And I don't have access to the repository, so I can't check in the change even if you give it to me verbatim :D

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

I can't do that either but if you just want to try the newest version here's the fix:

 

In file "plugins/image/dds.cpp" at lines 60 to 65 you find the definition "struct MipMapInfo { ... };". You have to move these lines a few lines up and out of the function body of LoadDDSFromStream() so that its not a local definition. Should be simple. :)

Link to comment
Share on other sites

I can't do that either but if you just want to try the newest version here's the fix:

 

In file "plugins/image/dds.cpp" at lines 60 to 65 you find the definition "struct MipMapInfo { ... };". You have to move these lines a few lines up and out of the function body of LoadDDSFromStream() so that its not a local definition. Should be simple. :)

 

ah ok, thanx! I am about to travel until the 25th so I can't try it right now but I'll see if it is still nec. afterwards.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recent Status Updates

    • Ansome

      Finally got my PC back from the shop after my SSD got corrupted a week ago and damaged my motherboard. Scary stuff, but thank goodness it happened right after two months of FM development instead of wiping all my work before I could release it. New SSD, repaired Motherboard and BIOS, and we're ready to start working on my second FM with some added version control in the cloud just to be safe!
      · 1 reply
    • Petike the Taffer  »  DeTeEff

      I've updated the articles for your FMs and your author category at the wiki. Your newer nickname (DeTeEff) now comes first, and the one in parentheses is your older nickname (Fieldmedic). Just to avoid confusing people who played your FMs years ago and remember your older nickname. I've added a wiki article for your latest FM, Who Watches the Watcher?, as part of my current updating efforts. Unless I overlooked something, you have five different FMs so far.
      · 0 replies
    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
×
×
  • Create New...