2.10 Crashes - May be bow \ frontend acceleration related

[stgatilov]

23 hours ago, vozka said:

Savefile for The Black Mage here: https://drive.google.com/file/d/1CHXidcfVLtxcF2JebQjRB5kXfebOL1pE/view?usp=sharing

This is the non-crashing bug, where the bow merely doesn't work. I have a water arrow in my hands in the savefile and when I try to fire it, it doesn't work and the animation is broken.

Created and fixed: 5928

After inspecting this savegame and finding the most likely cause, I found a reliable way to reproduce it.
It only happens if you load the game in a fresh TDM process, and if you saved game while holding/stringing the bow.
Hopefully, this won't happen in future dev builds and in 2.11.

Also, there is a minor probability that this change will fix crashes too, if they happened due to thread-unsafety of cvars system.

[datiswous]

On 3/21/2022 at 7:11 PM, nbohr1more said:

Yep. Linux is a bit of a pain.

The last time I tried replicating a crash, I invoked "gdb thedarkmod.x64" and then "run" and when the application crashed, I invoked "step" to see subsequent operations.

After doing more digging, most Linux distros should have kdump installed and the /etc/kdump.conf should show the path where crashes are supposed to be written.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/kernel_administration_guide/kernel_crash_dump_guide

The kdump data is also supposedly able to be parsed by gdb.

https://developers.redhat.com/blog/2021/04/30/the-gdb-developers-gnu-debugger-tutorial-part-1-getting-started-with-the-debugger#next_up

If you have working info for Linux crashdumps, maybe you can update the wiki article?

https://wiki.thedarkmod.com/index.php?title=Save_a_Memory_Dump_for_debugging_Crashes

[TheUnbeholden]

I've also experienced bow related crashes in Hazard Pay. I don't have MSI Afterburner + RTSS. I have Bow Aimer or Front End Acceleration disabled. Its weird and a shame that this has happened, its with a clean install of TDM because the damned installer didn't work for me.

[vozka]

I encountered the bow crash again, in a different mission. It's in Who Watches The Watcher?, it happened again when shooting an NPC with a broad arrow (a different NPC than in Hazard pay, a guard, so it's not zombie-related). 

I noticed one new thing - the bow shot that crashes the game has broken animation. Something happens when "loading" the bow, the (broad) arrow seems to vanish and maybe the bow is at a weird angle as well - I'm not sure since it happens pretty fast and I only saw twice. Then I think the arrow release still works, and as soon as the arrow (dunno if it's visible by then or not) hits the enemy, the game immediately crashes. 

 

I am only able to reproduce it randomly. I have a save created shortly before the crash happened and I manage to reproduce it once in 2 tries, and then in the next 15-20 tries I was not successful again, so I gave up. I only managed to reproduce with broad arrows, not with water, gas or fire arrows that I had on me, but that may have been by chance.

[irisx]

I had a series of bow-related crashes in the new mission The Lieutenant 2: High Expectations, on linux. In a specific part of the map, selecting any bow weapon and then pressing attack/fire caused an immediate crash --just as the animation starts.

To avoid it I had to reload and shuffle a couple of times through weapons.

[Frost_Salamander]

15 minutes ago, irisx said:

I had a series of bow-related crashes in the new mission The Lieutenant 2: High Expectations, on linux. In a specific part of the map, selecting any bow weapon and then pressing attack/fire caused an immediate crash --just as the animation starts.

To avoid it I had to reload and shuffle a couple of times through weapons.

what part of the map was it?  I can try to reproduce on Windows...

[irisx]

8 minutes ago, Frost_Salamander said:

what part of the map was it?  I can try to reproduce on Windows...

The first time it happened was standing on the roof trusses of the cathedral trying to shoot broadheads downwards. The second time was on the cathedral groundfloor trying to shoot moss arrows, near the 1st row of seats.

In both cases, just reloading and selecting the same weapon using keys did not fix it: I still got the crash when I tried to shoot. I had to quickly scroll-wheel through all weapons, then select bow again to get through.

In any case, I think it's a bit random, certainly cannot reproduce it consistently.

[snatcher]

It happens, very rarely, in any mission. Most of the times with the bow, a few times with the sword. I am unable to willingly reproduce it in any way.

[Frost_Salamander]

I just tried it a whole bunch of times until it crashed.  Oddly, the time I drew the bow before the time it crashed, the arrow was missing from the animation.

@stgatilov is this stuff of any use?

crash dump: https://drive.proton.me/urls/A6T13TEFJR#6YzRcvVFsMW9

debugger screenshot (link to better image )

 

Edited May 11, 2023 by Frost_Salamander

[stgatilov]

10 hours ago, Frost_Salamander said:

debugger screenshot (link to better image )

The crash inside idDict access might be race condition.
idDict use global string pool without mutex, so two threads cannot access it simultaneously.
But the culprit is on the main thread then: frontend thread runs gameplay code which can legally access idDict-s.

I will download the dump and see myself what's going on.

Maybe we should add assert for threadID inside idDict/idStrPool?
Maybe even leave the check enabled in release, so that if such a problem happens, it leads to Error/crash in 100% cases...

[vozka]

19 hours ago, Frost_Salamander said:

I just tried it a whole bunch of times until it crashed.  Oddly, the time I drew the bow before the time it crashed, the arrow was missing from the animation.

I'm pretty sure this happened to me at least once as well, but not every time the crash happened, if this brilliantly specific information helps any. It was not in the mission mentioned above (The Lieutenant 2), it was in various different missions.

[stgatilov]

On 5/11/2023 at 10:47 PM, Frost_Salamander said:

crash dump: https://drive.proton.me/urls/A6T13TEFJR#6YzRcvVFsMW9

debugger screenshot (link to better image )

 

I suppose you built your own executable.
So crashdumps generated on it can only be opened with PDB file that was generated alongside it.

It is possible to hack PDB file to "match" EXE file (works fine if you were using Release build of 2.11), but unfortunately it does not work for a crashdump, and I don't have your executable that crashdump was generated for.

So either attach a PDB or at least EXE.
If you rebuilt TDM executable after recording crashdump, then I guess crashdump is no longer openable.

[Frost_Salamander]

16 minutes ago, stgatilov said:

I suppose you built your own executable.
So crashdumps generated on it can only be opened with PDB file that was generated alongside it.

It is possible to hack PDB file to "match" EXE file (works fine if you were using Release build of 2.11), but unfortunately it does not work for a crashdump, and I don't have your executable that crashdump was generated for.

So either attach a PDB or at least EXE.
If you rebuilt TDM executable after recording crashdump, then I guess crashdump is no longer openable.

I did build it, yes.  Here is the EXE: https://drive.proton.me/urls/H7CHQ4XY60#Q3aBT26LEq6e

where would the PDB be?  If I can find that I can send it.  Apologies, I don't know my way around this stuff too well.

[stgatilov]

8 minutes ago, Frost_Salamander said:

I did build it, yes.  Here is the EXE: https://drive.proton.me/urls/H7CHQ4XY60#Q3aBT26LEq6e

Well, this is not a release build, so it is unlikely I can use my PDB.
Which build it is, at least?

Quote

where would the PDB be?  If I can find that I can send it.  Apologies, I don't know my way around this stuff too well.

It is near executable. It has .pdb extension.

[Frost_Salamander]

This should be a release build, I checked out the 2.11 tag if I recall correctly.

Here is the PDB: https://drive.proton.me/urls/94KE71F310#d1P16mz5aO7B

[stgatilov]

Spoiler

Here you can select configuration.

The default one is "Debug" for some reason.
It is interesting, because we almost never use pure Debug these days

The problem is that "Debug", "Debug Editable", "Debug Fast" all use FASTLINK setting for debug information, which makes PDB file useless. Only "Release" configuration can generate crashdumps that can be passed to someone else for analysis (with PDB file of course).

 

It looks like I can see some stack traces, and the main thread seems to "wait for frontend".
It would be interesting to see what is wrong with e->spawnArgs value, but my debugger does not show anything.

[Frost_Salamander]

4 minutes ago, stgatilov said:

  Reveal hidden contents

Here you can select configuration.

The default one is "Debug" for some reason.
It is interesting, because we almost never use pure Debug these days

The problem is that "Debug", "Debug Editable", "Debug Fast" all use FASTLINK setting for debug information, which makes PDB file useless. Only "Release" configuration can generate crashdumps that can be passed to someone else for analysis (with PDB file of course).

 

It looks like I can see some stack traces, and the main thread seems to "wait for frontend".
It would be interesting to see what is wrong with e->spawnArgs value, but my debugger does not show anything.

oh sorry I complete misunderstood what you were talking about . I guess I thought that PDB would only be generated if you had 'debug' build config (see, told you I didn't know what I was doing).

So for this to be useful, I would need to rebuild it with 'release' config, and then make it crash again (and generate new crashdump)?

[stgatilov]

5 minutes ago, Frost_Salamander said:

So for this to be useful, I would need to rebuild it with 'release' config, and then make it crash again (and generate new crashdump)?

Yes.

I think it would be even easier if you take clean 2.11 installation and record crashdump on it.
This way our release PDB would match your crashdump automatically.
 

[Frost_Salamander]

2 minutes ago, stgatilov said:

Yes.

I think it would be even easier if you take clean 2.11 installation and record crashdump on it.
This way our release PDB would match your crashdump automatically.
 

OK - so no local build, etc even needed.  Didn't know that.

I'll give it a go on my usual release install.  It only took a few minutes last time to make it crash, but I may have got lucky...

[stgatilov]

7 minutes ago, Frost_Salamander said:

I'll give it a go on my usual release install.  It only took a few minutes last time to make it crash, but I may have got lucky...

It can be easier to trigger on Debug build.

Also, the ability to record crashdumps might depend on some Windows OS settings.
You can first try to execute "crash" console command and see if you can record a crashdump from it. Just to be sure everything is prepared properly.

[Frost_Salamander]

5 minutes ago, stgatilov said:

It can be easier to trigger on Debug build.

Also, the ability to record crashdumps might depend on some Windows OS settings.
You can first try to execute "crash" console command and see if you can record a crashdump from it. Just to be sure everything is prepared properly.

OK here's a new one from my normal TDM installation: https://drive.proton.me/urls/FENCGJ3HP0#oTMvDlmSLfux

It didn't actually happen while drawing the bow this time, it was in between draws when I was just walking around.

[stgatilov]

"player1_weapon" entity usually has two attachments: so-called aimer, and arrow.

In this crashdump, the aimer is here, but arrow attachment is dead: it was deleted beforehand, without clearing the attachment reference. And since this code does not check e for NULL, it calls a method on NULL object, which certainly crashes.

I'll of course add a check for NULL here, but it would be interesting to understand what kills arrow attachment, and why this problem does not happen every time.

UPDATE: Since everyone reported the arrow to be missing just before the crash, I suppose the real issue is that arrow entity is occasionally lost...

[Frost_Salamander]

Interesting.  Maybe that explains why last time I got the crash dump, the arrow was missing from the animation when I was drawing the bow?

I'm not sure what else I can try, but let me know if there is anything I can do to help.  BTW to make this happen, I just started High Expectations and sat there at the start of the map drawing and cancelling the bow repeatedly.  I was using the water arrow because those are the only arrows you have at the start.

[stgatilov]

How do you cancel shot?
Does the same issue happens to you on other maps?

[Frost_Salamander]

Just now, stgatilov said:

How do you cancel shot?

while the bow is fully drawn (and you are still holding down the fire button), just hit tilde key (put away weapon), or some other weapon key.

1 minute ago, stgatilov said:

Does the same issue happens to you on other maps?

I haven't tried to reproduce it on any other maps but I can try...