2.10 Crashes - May be bow \ frontend acceleration related

[Frost_Salamander]

I tried both A New Job and St. Lucia, and couldn't get it to crash on either one of them, after trying for a good while too.

With High Expectations I got it to crash a couple of more times.  Seems easier to reproduce there for some reason.

[stgatilov]

As a matter of face, I cannot reproduce the issue on High Expectations as well.

[Frost_Salamander]

3 minutes ago, stgatilov said:

As a matter of face, I cannot reproduce the issue on High Expectations as well.

hmm.  okay maybe I can try it again in the debug build with a breakpoint there and see if I can tell what's clearing that variable...

[stgatilov]

Maybe you can check if you can reproduce it without Frontend Acceleration.

And if you can, then try also com_smp 0.

[snatcher]

@stgatilov

I can reproduce this bug. TDM 2.11a. No mods.

There isn't a specific mission, spot or situation but I identified some patterns and I eventually get the game to crash.

In tdm_weapon_arrow.script I get the last print out in the console right before this sys.wait(). TDM then crashes. This perhaps doesn't mean much because many things are going on elsewhere at this point in time.

 

I don't want to go the enable/disable settings route to further troubleshoot the problem: I can make the game crash. How do I get more more info for you from, I don't know, the binaries?

Windows event log:

Faulting application name: TheDarkModx64.exe, version: 2.0.11.0, time stamp: 0x63d6c10d
Faulting module name: TheDarkModx64.exe, version: 2.0.11.0, time stamp: 0x63d6c10d
Exception code: 0xc0000005
Fault offset: 0x0000000000415e1f
Faulting process id: 0x26f4
Faulting application start time: 0x01d9d66c90204ea7
Faulting application path: H:\GAMES\The Dark Mod\TheDarkModx64.exe
Faulting module path: H:\GAMES\The Dark Mod\TheDarkModx64.exe
Report Id: 3f3944ab-8451-4c1b-835a-11c984202c47
Faulting package full name: 
Faulting package-relative application ID: 

[snatcher]

It probably is more clear this way.

Press "attack" at that moment and 5P is the last line in the console before the crash. Always.

void weapon_arrow::Idle()
{

sys.println(" ***** 5 *****");

	weaponReady();

sys.println(" ***** 5A *****");

	player owner = getOwner();

sys.println(" ***** 5B *****");

	// Switch to lower if weapon is not allowed right now
	if (owner.getImmobilization("") & IM_ATTACK_RANGED)
	{

sys.println(" ***** 5C *****");

		weaponState( "Lower", ARROW_IDLE_TO_LOWER );

sys.println(" ***** 5D *****");

	}

sys.println(" ***** 5E *****");

	playCycle( ANIMCHANNEL_ALL, "idle" );

sys.println(" ***** 5F *****");

	while( 1 ) {

sys.println(" ***** 5G *****");

		if ( WEAPON_LOWERWEAPON ) {

sys.println(" ***** 5H *****");

			weaponState( "Lower", ARROW_IDLE_TO_LOWER );

sys.println(" ***** 5I *****");

		}

sys.println(" ***** 5J *****");

		if ( WEAPON_ATTACK ) {

sys.println(" ***** 5K *****");

			weaponState( "Fire", ARROW_IDLE_TO_FIRE );

sys.println(" ***** 5L *****");

		}

sys.println(" ***** 5M *****");

		if (owner.getImmobilization("") & IM_ATTACK_RANGED)
		{

sys.println(" ***** 5N *****");

			weaponState( "Lower", ARROW_IDLE_TO_LOWER );

sys.println(" ***** 5O *****");
		}

// LAST PRINT OUT BEFORE THE CRASH
sys.println(" ***** 5P *****");

		// waitFrame();
		sys.wait(0.0166667);

sys.println(" ***** 5Q *****");

	}

sys.println(" ***** 5R *****");

}

[Daft Mugi]

5 hours ago, snatcher said:

I can reproduce this bug. TDM 2.11a. No mods.

There isn't a specific mission, spot or situation but I identified some patterns and I eventually get the game to crash.

That's awesome that you found a way to reliably reproduce this crash!

What steps are you taking while playing the game to reproduce the crash?

If I can reproduce it reliably as well, I'll run a debug build and get a backtrace on Linux.

I'm not familiar with Windows, but if you want to do the same, I'd look into running the debug build and getting a backtrace. And, perhaps try https://wiki.thedarkmod.com/index.php?title=Analyze_a_Memory_Dump

[snatcher]

Anything but reliable, unfortunately.

Sorry but I wouldn't waste time discussing the how: it happens by chance very rarely and I cannot guarantee anything except that I will do my best to crash my game as many times as I can. I need direction on how to get useful data out of the crash. Can someone prepare an ad-hoc exe, perhaps? That link you shared scares the hell out of me. As a start, I didn't receive a crash dump

[Daft Mugi]

13 minutes ago, snatcher said:

Anything but reliable, unfortunately.

Sorry but I wouldn't waste time discussing the how: it happens by chance very rarely and I cannot guarantee anything except that I will do my best to crash my game as many times as I can.

Oh, man. Well, thanks for putting in the time to make it crash.

14 minutes ago, snatcher said:

I need direction on how to get useful data out of the crash. Can someone prepare an ad-hoc exe, perhaps?

I thought that the TDM installer includes a debug build, but I was remembering wrong. Then, I thought the wiki had information about how to compile a debug build, but I wasn't able to find it. I'm sorry I can't help with a Windows debug build, so another dev will need to chime in here.

17 minutes ago, snatcher said:

That link you shared scares the hell out of me. As a start, I didn't receive a crash dump

Perhaps you can save a crash dump and send it to a dev?
https://wiki.thedarkmod.com/index.php?title=Save_a_Memory_Dump_for_debugging_Crashes

[snatcher]

12 minutes ago, Daft Mugi said:

Perhaps you can save a crash dump and send it to a dev?

https://wiki.thedarkmod.com/index.php?title=Save_a_Memory_Dump_for_debugging_Crashes

Thanks, awesome, I now have a modest 4,35GB dump

I guess I have to become friends with that previous link of yours.

[Daft Mugi]

11 minutes ago, snatcher said:

Thanks, awesome, I now have a modest 4,35GB dump

Does it compress well using Zip or 7-Zip?

[snatcher]

~650MB

Will see if I can figure something out myself. Thanks for the support.

[stgatilov]

Crashdump should compress very well.
Please share it somewhere

It is version 2.10, right?

[snatcher]

5 minutes ago, stgatilov said:

It is version 2.10, right?

2.11a

5 minutes ago, stgatilov said:

Please share it somewhere

Does the Crashdump contain any information that could be considered private? I will share it with you only just in case.

[stgatilov]

11 minutes ago, snatcher said:

Does the Crashdump contain any information that could be considered private? I will share it with you only just in case.

As far as I know, it contains complete virtual memory contents of TheDarkMod process at the moment of crash:
https://learn.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps

Maybe it has some stuff in the header (like OS build, CPU info), but I can't say anything for certain.
It should not contain any data from other programs of OS, as far as I know.

Note that kernel-level dumps are a very different thing:
https://learn.microsoft.com/en-us/troubleshoot/windows-client/performance/generate-a-kernel-or-complete-crash-dump

They indeed contains memory contents of OS and probably other programs, so don't send them to anyone untrusted (or just don't send them).

[Daft Mugi]

13 minutes ago, snatcher said:

Does the Crashdump contain any information that could be considered private? I will share it with you only just in case.

Not sure. It might include your username, where the exe is located, and system hardware.

On Linux, if I'm concerned, I do:

strings <file> | grep -i <search term>

That'll at least match against plain text but not binary data.

Perhaps you could use similar tools on Windows.

[snatcher]

It's been shared with @stgatilov for analysis.

[snatcher]

After a detailed review of this whole topic it seems stgatilov detected something relevant on May 14 and perhaps it's been corrected in recent builds... Nevertheless I remain available to share more crashdumps with different settings if required.

[Araneidae]

Don't know if it's helpful, but with the latest dev build (dev16818-10434) I had two or three crashes when head-shotting zombies while playing In The Black (forgotten the mission, will try and work it out).  I was going to report it, because it seemed quite repeatable, all the crashes were in exactly the same location ... but then it went away.

Edited August 25, 2023 by Araneidae
Remove reference to wrong mission

[stgatilov]

I think I have already seen such a state.
The crash happens in idWeapon::BeginAttack because "idEntity *e" is NULL.

Player's weapon has two attachments:

1. "idStaticEntity_atdm:attachment_aimer_8" --- this one is alive and OK.
2. Some second entity which has already died by this moment.

So the question is: what is this second entity, and is it OK that it is dead at the moment of attach?

UPDATE: The second attachment usually is something like "idStaticEntity_atdm:attachment_broadhead_30"...

[snatcher]

Good. Any ideas on how to troubleshoot this further?

[stgatilov]

16 hours ago, snatcher said:

Good. Any ideas on how to troubleshoot this further?

I built a custom TDM 2.11 executable with console logging when entity is attached and destroyed:

• https://drive.google.com/drive/folders/1gdXD9nugu4rgHwCIx-KX3ZefdteHizjI?usp=sharing

So you can do the following:

1. Unpack both files to TDM game directory.
2. Run thegame with new executable.
3. Set "logfile 2" so that all console text is recorded.
4. Reproduce the crash.
5. Find qconsole.log in the FM directory and post it here.

When this is over, make sure to revert cvar "logfile" back to its default value, and run tdm_installer to recover old executable if you didn't save it.

[snatcher]

Here: https://www.mediafire.com/file/psetzfhwqht5l84/qconsole_202308271900.zip/file

[stgatilov]

According to the last three events, someone deleted the arrow from script:

Spoiler

[116336] Attach: player1_weapon <- idStaticEntity_atdm:attachment_aimer_7
Stack trace (hash = FFFF20A8):
  idCommonLocal::PrintCallStack           G:/TheDarkMod/darkmod_211_src/framework/Common.cpp:618
  idAnimatedEntity::Attach                G:/TheDarkMod/darkmod_211_src/game/Entity.cpp:8724
  idWeapon::Attach                        G:/TheDarkMod/darkmod_211_src/game/Weapon.cpp:3612
  idWeapon::GetWeaponDef                  G:/TheDarkMod/darkmod_211_src/game/Weapon.cpp:1064
  idPlayer::Weapon_Combat                 G:/TheDarkMod/darkmod_211_src/game/Player.cpp:4164
  idPlayer::UpdateWeapon                  G:/TheDarkMod/darkmod_211_src/game/Player.cpp:4422
  idPlayer::Think                         G:/TheDarkMod/darkmod_211_src/game/Player.cpp:7551
  idGameLocal::RunFrame                   G:/TheDarkMod/darkmod_211_src/game/Game_local.cpp:3366
  idSessionLocal::RunGameTic              G:/TheDarkMod/darkmod_211_src/framework/Session.cpp:3061
  idSessionLocal::RunGameTics             G:/TheDarkMod/darkmod_211_src/framework/Session.cpp:3107
  idSessionLocal::FrontendThreadFunction  G:/TheDarkMod/darkmod_211_src/framework/Session.cpp:3157
  <lambda_4da128457a8cbff73e848802e63deb86>::<lambda_invoker_cdecl>  G:/TheDarkMod/darkmod_211_src/framework/Session.cpp:3234
  BaseThreadInitThunk                     :0
  RtlUserThreadStart                      :0
[116336] Attach: player1_weapon <- idStaticEntity_atdm:attachment_broadhead_8
Stack trace (hash = FFFF5B10):
  idCommonLocal::PrintCallStack           G:/TheDarkMod/darkmod_211_src/framework/Common.cpp:618
  idAnimatedEntity::Attach                G:/TheDarkMod/darkmod_211_src/game/Entity.cpp:8724
  idWeapon::Attach                        G:/TheDarkMod/darkmod_211_src/game/Weapon.cpp:3612
  idWeapon::GetWeaponDef                  G:/TheDarkMod/darkmod_211_src/game/Weapon.cpp:1108
  idPlayer::Weapon_Combat                 G:/TheDarkMod/darkmod_211_src/game/Player.cpp:4164
  idPlayer::UpdateWeapon                  G:/TheDarkMod/darkmod_211_src/game/Player.cpp:4422
  idPlayer::Think                         G:/TheDarkMod/darkmod_211_src/game/Player.cpp:7551
  idGameLocal::RunFrame                   G:/TheDarkMod/darkmod_211_src/game/Game_local.cpp:3366
  idSessionLocal::RunGameTic              G:/TheDarkMod/darkmod_211_src/framework/Session.cpp:3061
  idSessionLocal::RunGameTics             G:/TheDarkMod/darkmod_211_src/framework/Session.cpp:3107
  idSessionLocal::FrontendThreadFunction  G:/TheDarkMod/darkmod_211_src/framework/Session.cpp:3157
  <lambda_4da128457a8cbff73e848802e63deb86>::<lambda_invoker_cdecl>  G:/TheDarkMod/darkmod_211_src/framework/Session.cpp:3234
  BaseThreadInitThunk                     :0
  RtlUserThreadStart                      :0
[116384] Destroy: idStaticEntity_atdm:attachment_broadhead_8
Stack trace (hash = 00001C57):
  idCommonLocal::PrintCallStack           G:/TheDarkMod/darkmod_211_src/framework/Common.cpp:618
  idEntity::~idEntity                     G:/TheDarkMod/darkmod_211_src/game/Entity.cpp:1545
  idStaticEntity::`scalar deleting destructor'  :0
  idClass::Event_Remove                   G:/TheDarkMod/darkmod_211_src/game/gamesys/Class.cpp:1093
  idClass::ProcessEventArgPtr             G:/TheDarkMod/darkmod_211_src/game/gamesys/Class.cpp:1049
  idEvent::ServiceEvents                  G:/TheDarkMod/darkmod_211_src/game/gamesys/Event.cpp:619
  idGameLocal::RunFrame                   G:/TheDarkMod/darkmod_211_src/game/Game_local.cpp:3405
  idSessionLocal::RunGameTic              G:/TheDarkMod/darkmod_211_src/framework/Session.cpp:3061
  idSessionLocal::RunGameTics             G:/TheDarkMod/darkmod_211_src/framework/Session.cpp:3107
  idSessionLocal::FrontendThreadFunction  G:/TheDarkMod/darkmod_211_src/framework/Session.cpp:3157
  <lambda_4da128457a8cbff73e848802e63deb86>::<lambda_invoker_cdecl>  G:/TheDarkMod/darkmod_211_src/framework/Session.cpp:3234
  BaseThreadInitThunk                     :0
  RtlUserThreadStart                      :0

 

The first two events happen inside GetWeaponDef when player switches from blackjack to bow.
This should be the only place messing with the weapon.
Two attachments (aimer + arrow) are added here.

But then the arrow gets destroyed inside ServiceEvents, which is unexpected.
Normally, it should be destroyed in the same GetWeaponDef at the very beginning, before creating new attachments.
This is either triggered by script code, or by C++ code which posts delayed event.
 

[stgatilov]

@snatcher, I uploaded second debug executable:

• https://drive.google.com/file/d/1vP_XcRAo8-YQeibMsprT72ANZFKZo8Iq/view?usp=sharing

This time I hope to learn who posts the event for deleting attachment...