Loading Saves just... hangs after close+reopen the game

[chumbucket91]

This has been a recurring issue for me over the past year or so and I'm finally fed up enough with it to make a forum post about it. I've not been able to find anything else with either a forum or discord search, so I'm hoping I can get some troubleshooting assistance here.

Sometimes, on larger maps with lots of custom assets and/or scripting, when I'm attempting to load a save file the game just... hangs. At like 1% loaded, in the loading screen. This happens to me a lot with Seeking Lady Leicester, but also has started happening to my most recent playthrough of The Lieutenant 2.

Some relevant notes:

 - I am on the latest beta branch, but this has been happening to me since at least 2.13, so I don't think its a beta-specific bug.

 - I'm on Windows 10.

 - Starting a fresh run of the map does not ever reproduce this problem (I can reliably fresh-load a map even when I can't save-load it), and it doesn't happen until I close the game and reopen it (so, if I start a new mission and play a bit, I can reliably load save files if I get caught or whatever. Its not until I save after a long gaming session and try to pick up where I left off later that I occasionally hit this bug).

 - Usually I get around this issue with a complete reinstall of TDM? But my most recent reinstall didn't do it for my playthrough of Lieutenant 2.

 - Once this happens to me, there's no way for me to close the game or go back to the main menu. I have to alt+F4 or Task Manager kill TDM.

 - When I'm loading a save or a new map and things are working correctly, I am unable to open the dev console during the load. However, when I get stuck in this bad state, the dev console is openable. I've attached a screenshot of the contents of the dev console just in case its illuminating, although there are no scary looking errors or warning text that I can see. If we want me to try any commands when I get into this state I'd be happy to poke around.

Let me know if this explanation makes sense, or if there's anything I should try or any debug info I should poke around the game's folder for.

Edited February 26 by chumbucket91

[datiswous]

18 hours ago, chumbucket91 said:

 - When I'm loading a save or a new map and its working correctly, I am unable to open the dev console

Do you know how to open it?

[chumbucket91]

Yes, generally the keybind is ctrl+alt+` (key next to number 1 for non-QWERTY/non-US keyboards), and I can open it in the main menu and ingame reliably. I have to use it to dmap the FM that I've been working on for awhile now, and I also use it to noclip around other people's FMs when I want to see "behind the curtains", so to speak. I was able to take that screenshot with it opened, which I hope is proof that I can use it haha.

Its only when a save is loading, and loading correctly, that the keybind is unresponsive. Is there another way to interrupt the game during load that can open the console, that I am unaware of?

Edited February 26 by chumbucket91

[datiswous]

18 hours ago, chumbucket91 said:

Its only when a save is loading, and loading correctly, that the keybind is unresponsive. Is there another way to interrupt the game during load that can open the console, that I am unaware of?

I think during loading a save the console doesn't work. I tried console command:

loadGame "savegamename"

(Where "savegamename" is the name of your savegame)

But it doesn't load it inside the console.

 

Starting TDM from a console window ("cmd") might give you access to the console. This does work for me on Linux at least.

[chumbucket91]

Hey, that's worth a shot!
 So, when the load gets stuck and I have access to a console, I attempted to

loadGame "sav"

and the load restarted, but got stuck at the exact same place. HOWEVER, there was some interesting warning text in the console when this happened!

WARNING:unknown destination `FlareBox::rect` of set command at \fms\highex\highex.pk4/guis/map/fsx.gui:<line number changes>

I've attached a screenshot - you can see me running the command at the very top of the console history, and then warning text.

@Frost_Salamander since you're the FM author - do you happen to recognize that particular gui script? I'm sure its not necessarily fresh in your memory but maybe you could shed some light on whatever that flareBox rect thing is, which might help me get to the bottom of my hang issue?

[Frost_Salamander]

7 minutes ago, chumbucket91 said:

Hey, that's worth a shot!
 So, when the load gets stuck and I have access to a console, I attempted to

loadGame "sav"

and the load restarted, but got stuck at the exact same place. HOWEVER, there was some interesting warning text in the console when this happened!

WARNING:unknown destination `FlareBox::rect` of set command at \fms\highex\highex.pk4/guis/map/fsx.gui:<line number changes>

I've attached a screenshot - you can see me running the command at the very top of the console history, and then warning text.

@Frost_Salamander since you're the FM author - do you happen to recognize that particular gui script? I'm sure its not necessarily fresh in your memory but maybe you could shed some light on whatever that flareBox rect thing is, which might help me get to the bottom of my hang issue?

It's probably a result of me copying and pasting some GUI file to make my loading screen.  I think it's this stuff here: https://github.com/FrostSalamander/fsx/blob/7e9bb0c837fc479503072f7dc443067ea59700aa/guis/map/fsx.gui#L30

I have no idea what this stuff means.  I can't remember if the warning was always there or not either.  I have a vague recollection of someone mentioning it, but not sure if it was that mission or not.

In subsequent FMs I removed that section of the GUI for my loading screens, like in LT3 here: https://github.com/FrostSalamander/lt3/blob/main/guis/map/lt3.gui

Maybe try editing that GUI file and removing that whole section and see what happens?

[chumbucket91]

Gah, it was a red herring. I altered fsx.gui to no longer have the entire windowDef Action section and, while the warnings are gone, the save still gets stuck while loading.

Thanks for the assist, at any rate

[grodenglaive]

I could be wrong, but I think that you can only load a game save if you are on the same TDM version that the game was saved under. It should give you a warning though.

Black-screen hanging has been an issue for me on the latest beta anytime there's a scripting error.

 

[chumbucket91]

Save version compatibility isn't the cause of this - this currently broken save was made in the same beta branch that I'm using to try to load it.

 

I have noticed the error message they added while upgrading between 2.14's beta branches though, and I just wipe all my saves and start from scratch when it happens.

Edited February 27 by chumbucket91

[datiswous]

18 hours ago, Frost_Salamander said:

I have no idea what this stuff means

I think what that sequience is suposed to do is showing an animation during loading of the mission. So depending on the position of the loadingbar it shows something else. I saw a mission where you saw leaves falling from a tree for example.

The flarebox might be something you didn't copy with it.

Edited February 27 by datiswous

[stgatilov]

To begin with, can you share savegame file and condump (preferably using "logfile 1" cvar) of how it happens?
Also, which exact TDM version do you use it, and which mission?

[chumbucket91]

2 hours ago, stgatilov said:

To begin with, can you share savegame file and condump (preferably using "logfile 1" cvar) of how it happens?
Also, which exact TDM version do you use it, and which mission?

You got it!

My current TDM version is 2.14 #11195 according to the main menu, which I believe corresponds to beta 5 or 6. This is not the first time this has happened to me, though - this also happened in 2.13 stable, and earlier 2.14 betas.

This is currently happening in Lieutenant 2: High Expectations .

I set `logfile 1` by hand in the main menu, and then attempted to load the save, and then had to alt+F4 to escape the application. I've attached the resulting log file to this post. There's... not much to it.
My Darkmod.cfg file contains a line that reads 

seta logFile "1"

but I can't find a corresponding qconsole.log anywhere in my install directory or appdata for that? Does that log file have more info or a different name? Or does that line not write a log file for some reason?

I've uploaded the content of my save folder, which includes only the one save that's causing the issue, to google drive here: https://drive.google.com/file/d/1RXRkV563gnigTzNAVnIzYU7bHcKYXORv/view?usp=sharing Let me know if this link doesn't work for whatever reason.

qconsole.log

Edited February 27 by chumbucket91

[Frost_Salamander]

16 minutes ago, chumbucket91 said:

You got it!

My current TDM version is 2.14 #11195 according to the main menu, which I believe corresponds to beta 5 or 6. This is not the first time this has happened to me, though - this also happened in 2.13 stable, and earlier 2.14 betas.

This is currently happening in Lieutenant 2: High Expectations .

I set `logfile 1` by hand in the main menu, and then attempted to load the save, and then had to alt+F4 to escape the application. I've attached the resulting log file to this post. There's... not much to it.
My Darkmod.cfg file contains a line that reads 

seta logFile "1"

but I can't find a corresponding qconsole.log anywhere in my install directory or appdata for that? Does that log file have more info or a different name? Or does that line not write a log file for some reason?

I've uploaded the content of my save folder, which includes only the one save that's causing the issue, to google drive here: https://drive.google.com/file/d/1RXRkV563gnigTzNAVnIzYU7bHcKYXORv/view?usp=sharing Let me know if this link doesn't work for whatever reason.

qconsole.log 4.87 kB · 1 download

He's asking you to run the condump command.  If you can open the console when it's hanging, type 'condump console.log'.  You'll then find the console.log file in the root of your TDM installation.

[chumbucket91]

Ah, ok - that makes more sense. Here's the result of `condump console.log`, which includes lots more detail.

console.log

[stgatilov]

It loads fine for me on Windows and Linux.

However, I see out-of-bounds errors if I load it in Debug build. The first one happens during loading "Desktop" \ "InventoryIcon" \ "InventoryGroupText" GUI window in "guis/tdm_inv.gui" overlay. Something does not match between where you made this save and where I'm trying to load it.

I also checked your installation according to the log, and found two differences.
1) [C] D:\Installed Games\The Dark Mod\tdm_autosave.pk4 (1 files - 0xc429b81c) --- this is script-only mod, should be OK
2) [M] D:\Installed Games\The Dark Mod\fms\highex\highex.pk4 (152 files - 0xd0043d5b) --- the actual mission installed
With the latest version of the mission, I have different checksum locally:
  [M] G:\TheDarkMod\darkmod_temp\fms\highex\highex.pk4 (152 files - 0x673a3873)

Note that savegames are often incompatible between different version of the mission just like they are incompatible between different versions of core scripts or engine. And this is not checked by the game.

[stgatilov]

Also, since I see the issue inside inventory icon, could you please pay attention if the issue is somehow related to the inventory icon displayed in the bottom-right corner?...

Like maybe the issue happens when you have something there, or when you have some specific item (like the key) there?

[chumbucket91]

Quote

I also checked your installation according to the log, and found two differences.
1) [C] D:\Installed Games\The Dark Mod\tdm_autosave.pk4 (1 files - 0xc429b81c) --- this is script-only mod, should be OK

This mod is actually one that I made, and it calls `saveGame` every five-10 minutes with an auto generated name. https://forums.thedarkmod.com/index.php?/topic/22647-quick-and-dirty-autosave-mod/#comment-499784 The save file I uploaded that we are looking at is not an autosave, and that mod has been compatible and functioning correctly for several versions now. It is, indeed, a tiny script-only mod.

Quote

2) [M] D:\Installed Games\The Dark Mod\fms\highex\highex.pk4 (152 files - 0xd0043d5b) --- the actual mission installed
With the latest version of the mission, I have different checksum locally:
  [M] G:\TheDarkMod\darkmod_temp\fms\highex\highex.pk4 (152 files - 0x673a3873)

Note that savegames are often incompatible between different version of the mission just like they are incompatible between different versions of core scripts or engine. And this is not checked by the game.

The checksum changed because I made the adjustment to the loading screen gui recommended in frost salamander's post earlier in this topic ( https://forums.thedarkmod.com/index.php?/topic/23119-loading-saves-just-hangs-after-closereopen-the-game/#findComment-506750 ). This mission is, flare box warning notwithstanding, bitwise identical to the latest version in the repo available to the mission downloader. Also, this issue was present before I followed that advice from frost salamander. I suppose I can go undo that change/redownload the .pk4 since it didn't fix the issue anyways.

I will revert to a clean install of both the latest TDM beta and this particular mission, just to clear some red herrings out of our debugging work.

Quote

Also, since I see the issue inside inventory icon, could you please pay attention if the issue is somehow related to the inventory icon displayed in the bottom-right corner?...

Interesting! I'll go test this.

[chumbucket91]

1 hour ago, chumbucket91 said:

This mod is actually one that I made, and it calls `saveGame` every five-10 minutes with an auto generated name. https://forums.thedarkmod.com/index.php?/topic/22647-quick-and-dirty-autosave-mod/#comment-499784 The save file I uploaded that we are looking at is not an autosave, and that mod has been compatible and functioning correctly for several versions now. It is, indeed, a tiny script-only mod.

NOT SO FAST MR CHUMBUCKET how dare you think your own code is fool proof!

Ok, so slight story time: I wiped my install of TDM and then reinstalled the latest beta and a fresh copy of High Expectations. Then, I redownloaded my save file from google drive and put it in the FM's save folder - and was able to load the save file! After that, I redownloaded all of the other FMs I had and re-installed my WIP FM into my folder, and I could still load my save. Then I changed all of my config settings back to the way I had them before, and could still load my save.

Then I redownloaded my autosave.pk4 mod - and my save loading freezing problem returned as described in my OP. We have a culprit!

I have a new question for this thread, then, because I would very much like my autosave mod to function and not cause save-loading to hang for no discernable reason. This "mod" is a single .script file located in the \script\ folder. Its entire content is the following:

#ifndef __TDM_AUTOSAVE__
#define __TDM_AUTOSAVE__

/* ---------------------------------------------------------------------------------------------
 *
 * This file is added to The Dark Mod's Script Inclusion chain in tdm_main.script
 *
 * This is intended for players who wish to install addons with custom scripts that should be available
 * in all missions. Mission authors must use tdm_custom_scripts instead.
 * 
 * --------------------------------------------------------------------------------------------- */

#define MAX_AUTOSAVES 5
#define AUTOSAVE_INTERVAL_SECONDS 300

void autosave()
	{
		float save_count = 1;
		do {
			sys.wait(AUTOSAVE_INTERVAL_SECONDS);

			string fname = "__autosave_" + save_count;
			$player1.saveGame(fname);
			sys.println("Autosaving '" + fname + "'");
			save_count += 1;
			if (save_count > MAX_AUTOSAVES) {
				save_count = 1;
			}
		} while (1);
	}

void user_addon_init_autosave()				//If any of your addon scripts need to be initialised at map start, add their init function here. See the line that has been commented out with // for an example.
	{
		thread autosave();
		sys.waitFrame();		// contingency in case no addons are specified
	}

#endif

what have I done wrong with this script? 

Edited February 28 by chumbucket91

[Frost_Salamander]

3 minutes ago, chumbucket91 said:

NOT SO FAST MR CHUMBUCKET how dare you think your own code is fool proof!

Ok, so slight story time: I wiped my install of TDM and then reinstalled a fresh beta and a fresh copy of High Expectations. Then, I redownloaded my save file from google drive and put it in the FM's save folder - and was able to load the save file! After that, I redownloaded all of the other FMs I had and re-installed my WIP FM into my folder, and I could still load my save. Then I changed all of my config settings back to the way I had them before, and could still load my save.

Then I redownloaded my autosave.pk4 mod - and my save loading freezing problem returned as described in my OP. We have a culprit!

I have a new question for this thread, then, because I would very much like my autosave mod to function and not cause save-loading to hang for no discernable reason. This "mod" is a single .script file located in the \script\ folder. Its entire content is the following:

#ifndef __TDM_AUTOSAVE__
#define __TDM_AUTOSAVE__

/* ---------------------------------------------------------------------------------------------
 *
 * This file is added to The Dark Mod's Script Inclusion chain in tdm_main.script
 *
 * This is intended for players who wish to install addons with custom scripts that should be available
 * in all missions. Mission authors must use tdm_custom_scripts instead.
 * 
 * --------------------------------------------------------------------------------------------- */

#define MAX_AUTOSAVES 5
#define AUTOSAVE_INTERVAL_SECONDS 300

void autosave()
	{
		float save_count = 1;
		do {
			sys.wait(AUTOSAVE_INTERVAL_SECONDS);

			string fname = "__autosave_" + save_count;
			$player1.saveGame(fname);
			sys.println("Autosaving '" + fname + "'");
			save_count += 1;
			if (save_count > MAX_AUTOSAVES) {
				save_count = 1;
			}
		} while (1);
	}

void user_addon_init_autosave()				//If any of your addon scripts need to be initialised at map start, add their init function here. See the line that has been commented out with // for an example.
	{
		thread autosave();
		sys.waitFrame();		// contingency in case no addons are specified
	}

#endif

what have I done wrong with this script? 

If there is a problem with the script, maybe you're the latest victim of this bug: https://bugs.thedarkmod.com/view.php?id=6686

I can't remember if it's been fixed already though. @stgatilov?

[chumbucket91]

"Hang" definitely describes my issue - and since I'm in a loading screen, I really can't send the game engine any input to make it crash, which would be the second part of that bug description.

It seems likely that there is some kind of bug with my script that only happens on loading a save. Its a map-start autoload that spins off a thread so I can only imagine what that bug might be, and the console isn't giving me any info.

[stgatilov]

I think saveGame script event is a very dangerous thing.

All the proper ways to save game are called from the main thread outside of parallel section.
But the game script is called immediately in the frontend thread, which is not supposed to work properly. At the very least it needs to run GL commands to take screenshot, which it certainly cannot do on frontend thread.

[stgatilov]

I think I fixed it here: https://bugs.thedarkmod.com/view.php?id=6470

Right now the fix is only in SVN trunk.