Linux crash: High Expectations

[Frost_Salamander]

A couple of users have pretty consistent crashes with High Expectations. I have also experienced crashes with Windows on my own machine, but very rarely.  I have a debug build of 2.11, but I haven't been able to get it to crash under that yet.

I've attached the logs that @irisx has provided, and @DavyJones has the exact same issue.  The only thing that stood out to me was in the qconsole.log:

The ambient 'snd_tunnels' (underground_forelone_loop_z) for location 'location_tunnels' is now playing.
Getting threadname failed, reason: No such file or directory (2)
--------- Game Map Shutdown ----------

@irisx did you say you also have some trace file?

Do any other Linux users have this problem (or not)?  The issue seems to occur about 1.5 seconds after the FM starts, or as soon as you start to provide input (move the mouse or start walking, etc).

 

qconsole.log Darkmod.log

[DavyJones]

As for me:

If I start mission and do nothing and even dont touch a mouse - crash after 1.5 sec.

If I do something, for example just rolling a mouse or walk back/forward - all seems ok

[irisx]

I confirm the same behaviour that @DavyJones is reporting: starting from a fresh install of the mission and not doing anything like moving the mouse or pressing a key, results in an immediate crash.

If I move around upon start, most of the time (but not always) the mission continues normally. Once this happens, there are no crashes upon subsequent restarts or reloads.

In addition to the above logs, I have generated a strace -F trace file of the crash, in case somebody (e.g. @nbohr1more ) could get some useful information from it. It is around ~15MB (gzipped), any suggestions on how to send it?

[nbohr1more]

@irisx I believe our bug-tracker will accept files of that size:

https://bugs.thedarkmod.com/my_view_page.php

Otherwise we will gladly accept a Google Drive or Dropbox ( etc ) link.

[irisx]

Thanks for the fast reply, this Google Drive link should work:

High Expectations Linux crash - strace -F log

[nbohr1more]

2 hours ago, irisx said:

Thanks for the fast reply, this Google Drive link should work:

High Expectations Linux crash - strace -F log

Thanks!

Are you running TDM with elevated privileges?

 

[irisx]

Quote

Are you running TDM with elevated privileges?

Well, none that I'm aware of

I am not running as root nor have any suid/sgid executables in the TDM folders. On the other hand, nVidia's binary modules have been known to do strange stuff from time to time, and they do run with elevated privileges.

[nbohr1more]

Something is causing ALSA audio errors. One thing that modern Linux versions have issues with is running audio as a privileged user.

Do you use pipewire or pulse?

[nbohr1more]

Hmm:

 

[pid 88402] write(2, "[ALSOFT] (EE) Failed to set Ster"..., 61[ALSOFT] (EE) Failed to set Stereo, got 5.1 Surround instead
) = 61

 

Please post your Darkmod.cfg and the results of:

( Open console ) CTRL+ALT+ ~  ( tilde )

condump audiocrash.txt

[jonri]

I gave it a try, and mine crashed with the same segfault the first time I tried.  I've attempted to reproduce it while running under GDB and haven't gotten it to crash again yet.  Each time I started it it up the same way without moving.

I did a little source code snooping and the only places I found that would print out that particular message "Getting threadname failed" would be where Sys_GetThreadName is called to print out some information when a mutex is busy, deadlocked, or not locked when it should be; or also when a thread is destroyed.

If I manage to catch a backtrace I'll share it here or on the bug tracker.

[stgatilov]

1.5 seconds after start is probably the time when script functions are called.
Or some other post-spawn activity.

Someone can also share LInux debug symbols for 2.11 release (they should be in assets repo somewhere inside devel).
This way we can probably see stacktrace.

Also keep in mind that the threadname message might be not the error itself, but some issue happening when game tries to throw exception or stop something after the issue has already happened.

[irisx]

@nbohr1more thanks, here are the files.

Darkmod.cfg audiocrash.txt

[irisx]

20 hours ago, nbohr1more said:

Do you use pipewire or pulse?

Just in case it's not visible in the logs, I'm using pulseaudio.

[Heuli]

I have on Linux Ubuntu and cinnamon Linux desktop no up crashes Intel CPU 10 gen Geforce 1060 6gb ram 16 rams SSD my computer boots in 11 seconds about 5000 MB write 6000 read I thought that the SSD is too fast so that some scripts are not started properly but no falls

[Frost_Salamander]

On 5/2/2023 at 9:16 AM, stgatilov said:

Someone can also share LInux debug symbols for 2.11 release (they should be in assets repo somewhere inside devel).
This way we can probably see stacktrace.

Who would need to do what for this? Who can access the assets repo?  Once we got the files, what needs to happen then? It sounds like a useful suggestion, but not really sure how to progress it...

Edited May 3, 2023 by Frost_Salamander

[nbohr1more]

The source code has the debug symbols, you just have to compile your own executable (etc) rather than use the one from the installer. I tried to replicate on my Linux Mint desktop while in debug but thus far cannot get it to crash.

[stgatilov]

6 hours ago, nbohr1more said:

The source code has the debug symbols, you just have to compile your own executable (etc) rather than use the one from the installer. I tried to replicate on my Linux Mint desktop while in debug but thus far cannot get it to crash.

@nbohr1more can provide you debug symbols from the assets SVN repo (for 2.11 release only, not for dev builds). They are zipped inside devel/release directory. I don't have my main PC available right now, can't do it.

Then you can run "gdb ./thedarkmod.x64", within gdb execute "add-symbol-file ./thedarkmod.x64.debug", and run the program to reproduce your issue.

[nbohr1more]

5 hours ago, stgatilov said:

@nbohr1more can provide you debug symbols from the assets SVN repo (for 2.11 release only, not for dev builds). They are zipped inside devel/release directory. I don't have my main PC available right now, can't do it.

Then you can run "gdb ./thedarkmod.x64", within gdb execute "add-symbol-file ./thedarkmod.x64.debug", and run the program to reproduce your issue.

@AluminumHaste can you attach this? If not I can upload this tonight.

[irisx]

8 hours ago, stgatilov said:

can provide you debug symbols from the assets SVN repo (for 2.11 release only, not for dev builds). They are zipped inside devel/release directory. I don't have my main PC available right now, can't do it.

Then you can run "gdb ./thedarkmod.x64", within gdb execute "add-symbol-file ./thedarkmod.x64.debug", and run the program to reproduce your issue.

Maybe I could try this too, since I'm more or less able to consistently reproduce the crash?

[Frost_Salamander]

I got it to crash on my Windows debug build. I clicked 'debug', and in VS I was able to see the stack trace:

https://drive.proton.me/urls/B06A4E8MV4#2lezsq0gsgfd

I think I might know what that is.  The entity in question is a door (atdm:arched01_111x40_left) that I didn't want to be openable.  If I remember correctly, the usual tricks weren't working (making it a func_static made it disappear, and making it non-frobable the AI were still using it).  So I changed the spawnclass to an idStaticEntity. Because it was a prefab, I think I thought it was a custom brush door as well.  I see now it's just using a model, so I can probably just change it to that.

Anyways, those are all my excuses.  I'll fix this and send out a new version that someone can test.

 

 

Edited May 4, 2023 by Frost_Salamander

[AluminumHaste]

9 hours ago, nbohr1more said:

@AluminumHaste can you attach this? If not I can upload this tonight.

Yes sorry, missed this message.

 

[nbohr1more]

Here are the debug symbols :

https://drive.google.com/file/d/1MrMuhiYGtiiE8ZtRD1t5j8Yf6nohbOnE/view?usp=sharing

Here is a debug executable :

https://drive.google.com/file/d/1gJia18obG5AuUmXirOyoRItbZvvxJThS/view?usp=sharing

[stgatilov]

16 hours ago, Frost_Salamander said:

I think I might know what that is.  The entity in question is a door (atdm:arched01_111x40_left) that I didn't want to be openable.  If I remember correctly, the usual tricks weren't working (making it a func_static made it disappear, and making it non-frobable the AI were still using it).  So I changed the spawnclass to an idStaticEntity.

My guess would be:

• the door generates visual stims that AI can notice
• the stim has "AIUse" spawnarg which specifies its type, which is 7 = "AIUSE_DOOR"
• the AI code casts the stim source entity to CFrobDoor, which it is not -> crash

Here is the place with non-checking cast:

	else if (aiUseType == EAIuse_Door)
	{
		// grayman #2866 - in the interest of reducing stim processing for closed doors,
		// add a check here to see if the door is closed. Otherwise, a closed door will
		// ping every AI w/in its radius (500) but the AI won't shut it down until it
		// can see the door. A guard not patrolling will receive endless pings unless
		// we shut it down here.

		CFrobDoor* door = static_cast<CFrobDoor*>(stimSource);
		if ( !door->IsOpen() )
		{
			door->DisableStim(ST_VISUAL); // it shouldn't be pinging anyone until it's opened
			return;
		}

Supposedly, you changed the entity C++ class from door to static entity, but did not remove the "AIUse" spawnarg.

Maybe we should add IsType check here and do Error if it is incorrect.

[Frost_Salamander]

The ability to change the spawnclass definitely feels like something that you should only do if you know what you are doing (which I clearly don't).

Also, @stgatilov any idea why the crash wouldn't happen every time, instead of sometimes?  The doors are there, and there is an AI that hangs around it the entire time.

[stgatilov]

2 hours ago, Frost_Salamander said:

Also, @stgatilov any idea why the crash wouldn't happen every time, instead of sometimes?  The doors are there, and there is an AI that hangs around it the entire time.

It's probably nothing more that goes wrong, and reading wrong memory by itself does not crash often (it is usually valid memory page, and depending on the build its contents can be quite deterministic).

It is not the first time where such issues cause crashes on Linux while Windows build seems to work fine.
Also, we might have different compile settings: Linux build settings are pretty standard, while Windows ones have e.g. security buffer checks disabled. Even calling conventions are very different between Windows and Linux.

Most of C++ worshippers don't look any further than "we get undefined behavior here"