Jump to content
The Dark Mod Forums

Converting model .tgas to .dds


Springheel

Recommended Posts

Well it's a compressed format, so there's more work to do for the system... but that might be all super-speedy with the video card, and the fact that it's smaller probably helps.

 

Anyway, I just tried a DXT3 of parchment_load and it was black, so yes I guess something beyond what I have is needed for these (my guess is the ATI specific tool).

 

Edit:

The only time you will see increased load times is when doom 3 has to compress every TGA to DDS at load time

Ah, knew it came into play somewhere in there...

Link to comment
Share on other sites

  • Replies 143
  • Created
  • Last Reply

Top Posters In This Topic

Okay, I've converted guis/assets/mainmenu/parchment_load.tga --> DXT3 DDS (in the dds folders). See how that works; looks the same to me. From 4Mb down to about 1.3Mb (for whatever reason, a menu image needs mipmaps, otherwise it would've been 1Mb). If that doesn't work for anyone, we need to revert the add and delete.

Link to comment
Share on other sites

Okay, I've converted guis/assets/mainmenu/parchment_load.tga --> DXT3 DDS (in the dds folders). See how that works; looks the same to me. From 4Mb down to about 1.3Mb (for whatever reason, a menu image needs mipmaps, otherwise it would've been 1Mb). If that doesn't work for anyone, we need to revert the add and delete.

 

There was already dds/guis/assets/blanklevelshot.dds so it should work.

 

That it needs mipmaps is because the engine treates the menu as another 3D scene (consisting of a few flat triangles, tho) and normally renders it via the graphic card.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Well it's a compressed format, so there's more work to do for the system... but that might be all super-speedy with the video card, and the fact that it's smaller probably helps.

 

From my understanding of things, dds is understood natively by video cards and doesn't have to be de-compressed to run. A compressed TGA, on the other hand, does have to be de-compressed...so that creates more work. DDS is definitely the best choice. :)

Link to comment
Share on other sites

It just occured to me that we have a hard-coded list of GUI images in devel/manifests/base.txt - and this would need changing if the files are moved from TGA to DDS.

 

So I removed the list and teached devel/extract-assets.pl how to extract texture names from all the GUI files that are in the package, and then include the right version (TGA or DDS, whatever we find).

 

This turned up the problem that some files were mentioned like this:

 

 Warning! Cannot find any texture (DDS, TGA or JPG) for file 'guis/assets/mainmenu/slider_bg'
 Warning! Cannot find any texture (DDS, TGA or JPG) for file 'guis/assets/mainmenu/buttons_mainmenu/Settings'
 Warning! Cannot find any texture (DDS, TGA or JPG) for file 'guis/assets/mainmenu/buttons_mainmenu/Credits'
 Warning! Cannot find any texture (DDS, TGA or JPG) for file 'guis/assets/mainmenu/buttons_mainmenu/Quit_lit'
 Warning! Cannot find any texture (DDS, TGA or JPG) for file 'guis/assets/mainmenu/buttons_mainmenu/Credits_lit'
 Warning! Cannot find any texture (DDS, TGA or JPG) for file 'guis/assets/mainmenu/buttons_mainmenu/Settings_lit'
 Warning! Cannot find any texture (DDS, TGA or JPG) for file 'guis/assets/mainmenu/buttons_mainmenu/Quit'
 Warning! Cannot find any texture (DDS, TGA or JPG) for file 'guis/assets/mainmenu/buttons_mainmenu/Settings'
 Warning! Cannot find any texture (DDS, TGA or JPG) for file 'guis/assets/mainmenu/buttons_mainmenu/Credits'
 Warning! Cannot find any texture (DDS, TGA or JPG) for file 'guis/assets/mainmenu/buttons_mainmenu/Credits_lit'
 Warning! Cannot find any texture (DDS, TGA or JPG) for file 'guis/assets/mainmenu/buttons_mainmenu/Settings_lit'

 

Note the capital letters. I fixed this in the GUI files to be lowercase (the file exist as lowerase)

 

The reference to "slider_bg" was actually missing a part of the path inside the GUI file, fixed that too :)

 

Sneaksie: If the files referenced in the GUI files carry a .TGA extension, I *think* you need to remove that after converting them to DDS - not sure if doom will still find the DDS file if you explicitely request a .TGA file. Entries without any extension work just fine, tho.

 

So, this should work automatically now :) The generated manifest is still a bit incomplete and has problems (like the script including the particle definition name as a file name instead finding the file it was defined in) but I intend to fix this tomorrow.

 

Bed time for me now :D

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

One more strange thing

 

We actually have:

 

./guis/assets/objectives/background.jpg

and ./guis/assets/objectives/background.tga

 

I am sure the .JPG version can just be deleted.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Is the original script which was used to create the TGA _ed versions still around, and if so how does it work? I did another ~2 hours of this by hand tonight and started to hit disillusion state (it'll pass; it's just mind-numbingly strong right now). At first it seemed like a really good idea to give it the careful human touch, going where a script cannot, getting it done right the first time.................. but now I'm realizing there might be something to be said for getting "most" of it done by script, even if it does create errors, then finding and fixing the errors by hand. There's a lot of effing issues in these folders and I'd rather use human thought to fix those than devote it to something a script could handle.

 

What's needed:

The existing script to spit out medium quality (e.g., level 5 in photoshop) 256^2 (or half-size, whichever is smaller) JPGs instead of the 256^2 TGAs it did last time, for both brush and model materials.

 

This will represent around 70%-90% HD space reduction for editor versions (e.g., 384kb to 20kb). For items lacking editor versions at all (there are plenty) it will represent 95%-99% HD space reduction (e.g., 4Mb to 20kb) not to mention the main point of this: in-editor memory.

 

Problems which I'll gladly handle by hand, as cleanup after a script run:

1. many materials have ordering screwed up, for instance type_desc1_desc2_ed_tiling_1d

(the editor designation is within the name instead of at the end)

2. special cases where the diffuses still reside in the /textures folders, no DDS files exist

3. cases where _ed.tga files already exist (aka everywhere; the script could handle this easy I assume)

4. something else I ran into a lot of times over the past two hours, but now forget (brain fry)

 

Can it do this? Please say yes following here:

Link to comment
Share on other sites

Can it do this? Please say yes following here:

 

I don't know we had a script. It would be possible to write one, of course.

 

The best idea is probably to write a script that makes sure that for each material, the correct _ed.jpg file exists. However, it takes me a few hours to get it right and right now I'd rather first finish the script that builds the manifest for the release :)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

I don't know we had a script. It would be possible to write one, of course.

 

The best idea is probably to write a script that makes sure that for each material, the correct _ed.jpg file exists. However, it takes me a few hours to get it right and right now I'd rather first finish the script that builds the manifest for the release :)

 

Ok, this didn't let me rest, so I checked in devel/check-materials.pl. To run it, you need Perl:

 

* Either get it from http://strawberryperl.com/

* or from http://activestate.com/store/download.aspx...15-08d58c2648ca

 

Then open a DOS box, go to the darkmod directory and run the script like so:

 

perl devel/check-materials.pl

 

So far for the good news. The bad news is:

 

* the script currently only checks, not correcting things

* it outputs the following statistics:

 

 Found 1124 materials without a qer_editorimage.
Found 577 materials without a JPG qer_editorimage.

 

That's some serious work ahead of you, Sneaksie (or the script is faulty :)

 

Edit: The script couldn't handle qer_editorimage definitions without an extension, fixed it, this reduced the number of non-JPG by half. Still bootloads of work tho.

 

Making the script automatically create the JPGs is a bit more work. Need coffee first.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Making the script automatically create the JPGs is a bit more work. Need coffee first.

 

Ok, teached the script to automatically create JPGs with "convert" and then check them in with "svn add".

 

This, however, will only work on Linux I am afraid.

 

Currently testing this on a cloned repo, but we should have about 500 more JPGs in a few minutes :)

 

What is missing are the materials that do not have a qer_editor entry at all. I could teach the script to generate the JPGs, and check them in, but someone would need to go through the materials and manually add the "qer_editorimage" line. I'd rather not have it hacked it per script until my scripts can correctly and properly rewrite the material definition files - and the code can currently only read them. Making it write is planned, but actually a lot of work. Sorry :(

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Well, if your script can spit out a list of which materials need to be updated, this would already make our lives easier I reckon. Maybe it could also provide the line

 

qer_editorimage textures/darkmod/blah/blah_ed

 

so that a human only needs to copy&paste it into the correct location.

Link to comment
Share on other sites

Well, if your script can spit out a list of which materials need to be updated, this would already make our lives easier I reckon. Maybe it could also provide the line

 

qer_editorimage textures/darkmod/blah/blah_ed

 

so that a human only needs to copy&paste it into the correct location.

 

Yeah, except it will be over 1000 locations... which is not feasable unless we have a few trained monkeys to do it :D

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Okay, this will take a while. Cloning the repo takes about 15..20min solid on my laptop, and everytime I make a mistake in the script, I need to repeat that step :)

 

However, I am confident that in a few hours we have _ed.jpg files for all materials. I also have some ideas how to insert the qer_editorimage lines into the material definitions. Hunting down the "diffusemap" line and inserting it just before that line *should* work. But it needs extensive testing :)

 

Stand by....

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

RANT:

 

Consistency is good. Naming things with the same schema is important. Why? Because then you can recognize things by their name. Right now, our own definition files are just a giant mess. For instance, there is no real reliable way to determine what type of material a material is, because every now and then some generic name like "book1" is used, instead of "tdm_readable_book1".

 

This makes automating any work next to impossible, because all you do is coding in exceptions upon exceptions and in the end, you miss something, anyway :(

 

Grrrrrrrrrrrrrrrrrrr. End of rant.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Hit another snag: Some of our "no edit version" materials have DDS versions of the diffuse - and ImageMagick can handle these only from version 6.3.9 on but Ubuntu still has 6.3.7. Updating it...

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

I'm reading this as I go:

Found 1124 materials without a qer_editorimage.

Found 577 materials without a JPG qer_editorimage.

Wow! It's worse than I thought. But on the positive side, that means there's a lot of memory savings to be had, yay. :) Does that first statistic include TGA editor versions? The reason I ask is because the old script is what automatically generated 256^2 _ed.TGAs. Does it include only brush materials, or also those used for models?

 

Ok, teached the script to automatically create JPGs with "convert" and then check them in with "svn add".

Whoa, now that's what I'm talking about. :D Medium quality? (so there's actually good HD savings)

 

I could teach the script to generate the JPGs, and check them in, but someone would need to go through the materials and manually add the "qer_editorimage" line.

Interesting. Last night I had a hell of a time sleeping due to this (very silly, but I couldn't put it out of my head); I got mixed dreams of dog packs (WTF? Tines from A Fire Upon the Deep) discussing how to do the task, wondering if a batch conversion for Photoshop exists (it does for paintshop pro), etc. Even the usual "get up, take a piss, get a drink, walk around" didn't clear it from my head. So even though I'm not done reading yet, I'm kinda charged to hear what I've heard so far, and to seek solutions if needed.

 

Hunting down the "diffusemap" line and inserting it just before that line *should* work.

Part of what I was thinking about, tossing and turning last night. I wonder how hard it would be for a program of my level. I'd done simple file I/O before, but I'd probably have to research it all over again.

 

Naming things with the same schema is important.

You got that right for sure. When I first started running into names with _ed_ in the middle instead of on the end, or names where one material is "slate_" and the next material is "rough_slate_", I wanted to cry. That said, if it's choking the script, don't worry about it; I'm going to go through them all by hand to fix and verify anyway, and I'll pick up those exceptions.

 

Thanks for all the help on this! Thanks also for the links. I'm going to get perl and look at some of the scripts and try to learn something. It's a valued skill in my field.

Link to comment
Share on other sites

It appears that Photoshop can in fact do batch conversions (and more) with Actions. Just in case we need it.

 

Care to elaborate? That sounds interesting, and I wasn't aware of that ability.

Link to comment
Share on other sites

I found this: http://en.allexperts.com/q/Adobe-Photoshop...ert-PDF-JPG.htm

 

It sounds like it's a matter of defining Actions (it's one of the little tab interface things) and Recording on them what they should do. Haven't tried it yet, but it sounds much more powerful than PSP's batch convert (simpler to use, little power).

Link to comment
Share on other sites

It appears that Photoshop can in fact do batch conversions (and more) with Actions. Just in case we need it.

 

Don't worry, ImageMagick 6.4.1 (installed on my system) should handle this just fine. It is not the problem of actually converting the files, but finding out *what* filename to convert to *waht other* filename that is here the problem.

 

As for the script, the latest version seems to run fine, but I am still trying it on clones of the repo since it ever gets some things wrong.

 

The biggest missing thing is the code insertin qer_editorimage and let me tell you, this is *not* easy. The old plan, to make my parser robust enough to write out definition files again takes too much work for now.

 

The next best plan, to just use some regexps and modify the material file text is, well, very very ugly, as it is really hard to figure out where actually inserting the text. Still working on it.

 

Btw, here is one more comment on JPG vs TGA for editor files.

 

In my opinion using JPG is fine, but with a _high_ quality. Really, HD space is cheap, our repo is over 6 Gbyte, and if we have 100 Mbyte more or less won't hurt us. However, the 256x256 editor images are already quite fuzzy, and making them with heavy compressed JPGs will certainly not help their quality. And it will not save memory, bcause the JPEG has to be decompressed into memory, anway.

 

So in the end we end up with the worst of both worlds, We get fuzzy 256x256 textures, plus compression artefacts.

 

The only other option, "lighting mode" is no real alternative,because is way too slow, and due to the bug that ambient lights don't show, looks even worse to judge the texture quality than the "full lit mode".

 

So let's not overly "optimize" for diskspace.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Counterpoint: If a particular image is hard to use, it can be created at higher quality by hand at any time. But if we're using a blanket statement to cover the vast majority of these images which have no real difference in appearance between high and medium (I know first hand; I've done perhaps more than a hundred of them this week by hand), then it's ultimately nothing more than wasted space and wasted effort (why go to all this effort at all, if savings ends up being only 25%). See the images below. One is the high quality version; one the medium. They don't have enough difference between them IMO to warrant the high quality version being three times the size of the other. The high quality version offers only half size reduction compared to the original DDS. The med quality one, a bit more than 80% size reduction.

 

The point I'm trying to make there is yes, I agree that some of them could/should be higher quality. But definitely not the majority, quite the opposite, and it represents a lot of space savings lost for no reason. If certain select textures (unlike the one pictured here) warrant higher quality, I (or whoever) can gladly do those exceptions to the rule, by hand.

 

The alternative is that I must go through and pick out the ones that are wasting space because they're little more than a grey blur or a reddish smear or yet another blurry grainy wood, and that's going to waste a ton of my time, because it is the majority. I just say do the opposite: assume med, and adjust for high.

 

HD space is not cheap for everyone, and especially when it balloons to this large for a mod, and part of the cost is the trouble of installing a new one. Not everyone wants to be bogged down with that for the sake of playing a mod.

post-58-1212075563_thumb.jpg

post-58-1212075570_thumb.jpg

Link to comment
Share on other sites

Counterpoint: If a particular image is hard to use, it can be created at higher quality by hand at any time. But if we're using a blanket statement to cover the vast majority of these images which have no real difference in appearance between high and medium (I know first hand; I've done perhaps more than a hundred of them this week by hand), then it's ultimately nothing more than wasted space and wasted effort (why go to all this effort at all, if savings ends up being only 25%). See the images below. One is the high quality version; one the medium. They don't have enough difference between them IMO to warrant the high quality version being three times the size of the other. The high quality version offers only half size reduction compared to the original DDS. The med quality one, a bit more than 80% size reduction.

 

The point I'm trying to make there is yes, I agree that some of them could/should be higher quality. But definitely not the majority, quite the opposite, and it represents a lot of space savings lost for no reason. If certain select textures (unlike the one pictured here) warrant higher quality, I (or whoever) can gladly do those exceptions to the rule, by hand.

 

HD space is not cheap for everyone, and especially when it balloons to this large for a mod, and part of the cost is the trouble of installing a new one. Not everyone wants to be bogged down with that for the sake of playing a mod.

 

That is all nice and dandy, but you are talking mainly about relative savings. In the case you posted, that is 84 vs 36 Kbyte, or a saving of 48 Kbyte. If we save 50 Kbyte on each editor image, we save all in all about 2144 * 50 = 107200 Kbyte, or about 100Mbyte. (Edit: Given the fact that we have 2144 materials and each of them needs an editor image, which I don't know exactly, btw)

 

Given our current repo size of 6.something Gigabyte, 100Mbyte is not really worth thinking about. Eidt: What I mean is we only need to convert 50 TGAs to DDS to save easily more space than that. That should have priority.

 

You are right, it might effect people who just want to play the mod, because they also have to download the JPGs, but we could always make a seperate download just with the _ed files, normal players than save something to download and anyone serious into editing could download and drop them in and enjoy less memory while editing.

 

Btw, I have _no_ idea what you "medium" vs. "high" quality settings actually are, my script uses the standard "xx%" settings that almost every program supports and you can pass that as a parameter. The default is 85%, but if you are unhappy, you can easily re-run the script with a different percentage and it will recreate the JPGs from the TGA/DDS files. (This is not yet implemented, tho. I am still lagging behind and this discussion doesn't really help either)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Yes, the inevitable debate. As I've mentioned, I'm just referring to Photoshop's implementation of JPG quality (they have a slider on the save dialog). The approximated 100Mb means plenty IMO, especially when it represents no appreciable or needed image quality difference for perhaps 90%+ of the textures, but do whatever you wish.

Link to comment
Share on other sites

Yes, the inevitable debate. As I've mentioned, I'm just referring to Photoshop's implementation of JPG quality (they have a slider on the save dialog). The approximated 100Mb means plenty IMO, especially when it represents no appreciable or needed image quality difference for perhaps 90%+ of the textures, but do whatever you wish.

 

I have just run the following commands after the script created all the _ed versions of the JPEG files:

 

te@te-laptop:~/.doom3/darkmod.test3$ find -name "*_ed.jpg" -exec du -b '{}' \; >size.txt
te@te-laptop:~/.doom3/darkmod.test3$ vi sum.pl
te@te-laptop:~/.doom3/darkmod.test3$ cat size.txt |perl sum.pl
57424866

 

The result is that all editor JPGs consume 57Mbytes. Total. Nothing much can be saved here.

 

(The release will contain not all of these JPGs since we do not release all of our materials/textures, anyway)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recent Status Updates

    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
    • OrbWeaver

      I like the new frob highlight but it would nice if it was less "flickery" while moving over objects (especially barred metal doors).
      · 4 replies
    • nbohr1more

      Please vote in the 15th Anniversary Contest Theme Poll
       
      · 0 replies
×
×
  • Create New...