Jump to content
The Dark Mod Forums

Image duplication check


duzenko

Recommended Posts

This is another suggestion on improving load times.

When loading a map, TDM calculates an md4 checksum on each texture image.

The checksum is only used by a diagnostic tool (namely reportImageDuplication).

Not sure how many map designers have used it because e.g. NHAT has 64 duplicates, 17 of them being the TDM GUI.

Anyway I think it makes sense to add a config param to skip md4 calculation by default.

Then if any map designer wants to test image duplication they can toggle the param.

My test with NHAT:
md4 checksum on: 21.2 sec
md4 checksum off: 19.2 sec

Your tests will show less difference because I already don't use pk4's.

I can prepare a diff patch if this idea is accepted.

Edited by duzenko
Link to comment
Share on other sites

"A Score to Settle".

 

Here's the complete list if I start TDM and run "reportImageDuplication"

Images with duplicated contents:
fonts/english/carleton_glow/carleton_0_48 == fonts/english/carleton_bold/carleton_0_24
fonts/english/mason_glow/masonalternate_1_48 == fonts/english/carleton_bold/carleton_0_48
fonts/english/mason/masonalternate_4_48 == fonts/english/mason_glow/masonalternate_4_48
fonts/english/carleton_condensed/carleton_0_12 == fonts/english/mason/masonalternate_5_48
fonts/english/carleton_condensed/carleton_5_48 == fonts/english/mason_glow/masonalternate_4_48
fonts/english/andrew_script/andrew_script_0_24 == fonts/english/andrew_script/andrew_script_0_12
fonts/english/andrew_script/andrew_script_1_24 == fonts/english/andrew_script/andrew_script_0_12
fonts/english/andrew_script/andrew_script_0_48 == fonts/english/andrew_script/andrew_script_0_12
fonts/english/andrew_script/andrew_script_1_48 == fonts/english/andrew_script/andrew_script_0_12
fonts/english/andrew_script/andrew_script_2_48 == fonts/english/andrew_script/andrew_script_0_12
fonts/english/andrew_script/andrew_script_3_48 == fonts/english/andrew_script/andrew_script_0_12
fonts/english/andrew_script/andrew_script_4_48 == fonts/english/andrew_script/andrew_script_0_12
guis/assets/mission_success/background_success == guis/assets/mainmenu/background_empty
guis/assets/briefing/prevpage == guis/assets/briefing/nextpage
guis/assets/objectives/parchment_untitled == guis/assets/mainmenu/background_empty
guis/assets/objectives/box_default == guis/assets/briefing/nextpage
guis/assets/objectives/nextpage == guis/assets/briefing/nextpage
guis/assets/objectives/prevpage == guis/assets/briefing/nextpage
guis/assets/purchase_menu/parchment_shop == guis/assets/mainmenu/background_empty
fonts/english/medusa/medusa_0_12 == fonts/english/andrew_script/andrew_script_0_12
fonts/english/medusa/medusa_0_24 == fonts/english/andrew_script/andrew_script_0_12
Result : 21 collisions out of 230 images
Link to comment
Share on other sites

I think there is a bug with the check procedure.

Here's my patch that kinda works

 

 

Index: renderer/RenderSystem_init.cpp
===================================================================
--- renderer/RenderSystem_init.cpp	(revision 6638)
+++ renderer/RenderSystem_init.cpp	(working copy)
@@ -986,6 +986,8 @@
 		int		h1 = 0; int w1 = 0;
 
 		R_LoadImageProgram( image1->imgName, &data1, &w1, &h1, NULL );
+		if (!data1)
+			continue;
 
 		for ( int j = 0 ; j < i ; j++ ) {
 			idImage	*image2 = globalImages->images[j];
@@ -1013,10 +1015,11 @@
 
 				R_LoadImageProgram( image2->imgName, &data2, &w2, &h2, NULL );
 				
-				if ( w1 != w2 || h1 != h2 || memcmp( data1, data2, w1*h1*4 ) ) {
+				if (!data2 || w1 != w2 || h1 != h2 || memcmp( data1, data2, w1*h1*4 ) ) {
 					R_StaticFree( data2 );
 					continue;
 				}
+				R_StaticFree(data2);
 
 				common->Printf( "%s == %s\n", image1->imgName.c_str(), image2->imgName.c_str() );
 				session->UpdateScreen( true );

 

 

  • Like 1
Link to comment
Share on other sites

Here's the full diff patch with the new cvar and the fix from the previous post.

Using the map Illusionist tower I get 7.2 vs 6.2 sec in the console output for "all images loaded".

BTW how do you get a text copy of the console?

 

 

Index: renderer/Image.h
===================================================================
--- renderer/Image.h	(revision 6638)
+++ renderer/Image.h	(working copy)
@@ -385,6 +385,7 @@
 	static idCVar		image_downSizeBumpLimit;	// downsize bump limit
 	static idCVar		image_ignoreHighQuality;	// ignore high quality on materials
 	static idCVar		image_downSizeLimit;		// downsize diffuse limit
+	static idCVar		image_blockChecksum;		// duplicate check 
 
 	// built-in images
 	idImage *			defaultImage;
Index: renderer/Image_init.cpp
===================================================================
--- renderer/Image_init.cpp	(revision 6638)
+++ renderer/Image_init.cpp	(working copy)
@@ -72,6 +72,7 @@
 idCVar idImageManager::image_downSizeBumpLimit( "image_downSizeBumpLimit", "128", CVAR_RENDERER | CVAR_ARCHIVE, "controls normal map downsample limit" );
 idCVar idImageManager::image_ignoreHighQuality( "image_ignoreHighQuality", "0", CVAR_RENDERER | CVAR_ARCHIVE, "ignore high quality setting on materials" );
 idCVar idImageManager::image_downSizeLimit( "image_downSizeLimit", "256", CVAR_RENDERER | CVAR_ARCHIVE, "controls diffuse map downsample limit" ); 
+idCVar idImageManager::image_blockChecksum("image_blockChecksum", "0", CVAR_RENDERER | CVAR_ARCHIVE | CVAR_BOOL, "Perform MD4 block checksum calculation for later duplicates check");
 // do this with a pointer, in case we want to make the actual manager
 // a private virtual subclass
 idImageManager	imageManager;
Index: renderer/Image_load.cpp
===================================================================
--- renderer/Image_load.cpp	(revision 6638)
+++ renderer/Image_load.cpp	(working copy)
@@ -1635,7 +1635,8 @@
 		// build a hash for checking duplicate image files
 		// NOTE: takes about 10% of image load times (SD)
 		// may not be strictly necessary, but some code uses it, so let's leave it in
-		imageHash = MD4_BlockChecksum( pic, width * height * 4 );
+		if (globalImages->image_blockChecksum.GetBool())
+			imageHash = MD4_BlockChecksum( pic, width * height * 4 );
 
 		GenerateImage( pic, width, height, filter, allowDownSize, repeat, depth );
 		timestamp = timestamp;
Index: renderer/RenderSystem_init.cpp
===================================================================
--- renderer/RenderSystem_init.cpp	(revision 6638)
+++ renderer/RenderSystem_init.cpp	(working copy)
@@ -968,7 +968,9 @@
 void R_ReportImageDuplication_f( const idCmdArgs &args ) {
 	int	count = 0;
 
-	common->Printf( "Images with duplicated contents:\n" );
+	common->Printf("Images with duplicated contents:\n");
+	if (!globalImages->image_blockChecksum.GetBool())
+		common->Printf("Warning: image_blockChecksum set to 0, results invalid.\n");
 
 	for ( int i = 0 ; i < globalImages->images.Num() ; i++ ) {
 		idImage	*image1 = globalImages->images[i];
@@ -986,6 +988,8 @@
 		int		h1 = 0; int w1 = 0;
 
 		R_LoadImageProgram( image1->imgName, &data1, &w1, &h1, NULL );
+		if (!data1)
+			continue;
 
 		for ( int j = 0 ; j < i ; j++ ) {
 			idImage	*image2 = globalImages->images[j];
@@ -1013,10 +1017,11 @@
 
 				R_LoadImageProgram( image2->imgName, &data2, &w2, &h2, NULL );
 				
-				if ( w1 != w2 || h1 != h2 || memcmp( data1, data2, w1*h1*4 ) ) {
+				if (!data2 || w1 != w2 || h1 != h2 || memcmp( data1, data2, w1*h1*4 ) ) {
 					R_StaticFree( data2 );
 					continue;
 				}
+				R_StaticFree(data2);
 
 				common->Printf( "%s == %s\n", image1->imgName.c_str(), image2->imgName.c_str() );
 				session->UpdateScreen( true );

 

 

Edited by duzenko
Link to comment
Share on other sites

In Image_load.cpp:

 

// build a hash for checking duplicate image files
// NOTE: takes about 10% of image load times (SD)
// may not be strictly necessary, but some code uses it, so let's leave it in
if (globalImages->image_blockChecksum.GetBool())
    imageHash = MD4_BlockChecksum( pic, width * height * 4 );
"but some code uses it, so let's leave it in"

 

What code uses it, and how is that code affected by not doing this here?

Link to comment
Share on other sites

With image_blockChecksum set to 1, reportImageDuplication gives me a result of "0" (same conditions as before, with A Score to Settle loaded).

 

If I change the cvar to 0, I get the same results, but with the added warning that the results are invalid.

 

Can I assume that the results I got before your patch were wrong because of the bug you corrected in the patch?

Link to comment
Share on other sites

Practical test using a stopwatch (from shop's "Start Mission" to the "Mission Loaded" GUI).

 

AStS takes 32s to load with image_blockChecksum set to 1.

 

AStS takes 31s to load with image_blockChecksum set to 0.

 

So there is a small load savings with the patch.

Link to comment
Share on other sites

In Image_load.cpp:

 

// build a hash for checking duplicate image files
// NOTE: takes about 10% of image load times (SD)
// may not be strictly necessary, but some code uses it, so let's leave it in
if (globalImages->image_blockChecksum.GetBool())
    imageHash = MD4_BlockChecksum( pic, width * height * 4 );
"but some code uses it, so let's leave it in"

 

What code uses it, and how is that code affected by not doing this here?

 

AFAIK only the diagnostic tool uses it.

It will not find any duplicates.

 

With image_blockChecksum set to 1, reportImageDuplication gives me a result of "0" (same conditions as before, with A Score to Settle loaded).

 

If I change the cvar to 0, I get the same results, but with the added warning that the results are invalid.

 

Can I assume that the results I got before your patch were wrong because of the bug you corrected in the patch?

I think most of them were wrong because it did not check if R_LoadImageProgram returned valid data.

I still have some duplicates with Illusionist Tower and they do look the same.

Edited by duzenko
Link to comment
Share on other sites

In R_ReportImageDuplication_f(), if the results are invalid with image_blockChecksum set to 0, and the user is warned, why bother letting the rest of the method run? Why not exit immediately?

In case image_blockChecksum was toggled after images were loaded. Then they would still have block checksums on them. There is always hope!!!

Link to comment
Share on other sites

Practical test using a stopwatch (from shop's "Start Mission" to the "Mission Loaded" GUI).

 

AStS takes 32s to load with image_blockChecksum set to 1.

 

AStS takes 31s to load with image_blockChecksum set to 0.

 

So there is a small load savings with the patch.

There is more than one load time trick I used on my installation. Each adds a little but all combined work very well.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recent Status Updates

    • Petike the Taffer  »  DeTeEff

      I've updated the articles for your FMs and your author category at the wiki. Your newer nickname (DeTeEff) now comes first, and the one in parentheses is your older nickname (Fieldmedic). Just to avoid confusing people who played your FMs years ago and remember your older nickname. I've added a wiki article for your latest FM, Who Watches the Watcher?, as part of my current updating efforts. Unless I overlooked something, you have five different FMs so far.
      · 0 replies
    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
    • OrbWeaver

      I like the new frob highlight but it would nice if it was less "flickery" while moving over objects (especially barred metal doors).
      · 4 replies
×
×
  • Create New...