darkmod process instantly crashes when launched from DarkRadiant / Game Connection menu

[MirceaKitsune]

The ability to launch and control TDM from DarkRadiant via the Game Connection interface is very helpful. But at least on Linux (Manjaro / KDE Plasma in my case) it continues to be bugged by crashes and only work intermittently, and last night it suddenly stopped working for good. Hoping we can debug what must be happening before the next release. DR is 3.7.0 (Git commit d9145c3015d945014daeb829f22d8cdb94a2aa8f) while TDM is dev16650-10157.

DR successfully contacts the engine each time and attempts to start it up, I can even see the mouse cursor temporarily disappear and the process briefly present in the system monitor. The issue is when the darkmod executable is launched by Radiant, it often crashes immediately after. This might thus be a problem in the TDM engine more than in DR.

I'm dealing with two types of crashes so far: Until now it was a cooldown effect, I can't start TDM from DR more often than once every 3 or so minutes. If start it up too frequently I see a brief flash of the screen but the main menu doesn't get to open, about 10 seconds later DR gives me the error "Timeout when connecting to just started TheDarkMod process". I have no idea where the information on how frequently I'm starting it is even stored: I can shut down both DR and TDM to ensure neither have open processes running, yet even if I start DR up again and launch TDM from the Game Connection menu too soon it remembers and throws this error.

Since yesterday on top of that, the process now permanently fails with another error. In its case I even see the main menu for a split second before TDM instantly closes. DR throws the error "Game restart failed:  Connection lost unexpectedly". I even tried with fresh settings for both DR and TDM as well as restarting the machine, but despite not updating anything not even any new OS package updates nor changing firewall settings, this suddenly appeared and continues to occur.

[MirceaKitsune]

Just to be 100% certain, I temporarily disabled my firewall in the OS settings then launched DR and tried again: Exact same issue for both types of crashes so they aren't related to that. As I said I even tried with fresh settings for DR and temporarily removing Darkmod.cfg, it's baffling what must be happening and where the cause is stored.

[stgatilov]

Does anyone else have this problem?

Most likely this is a problem in the game.

1. Did you have DR + TDM connection working reliably in the past?
2. Can you check if it works relialbly with TDM 2.10?
3. Do you have the same problem when starting the game manually?
4. Do you have same problem if you start TDM manually with "thedarkmod.x64 +set com_automation 1" then connect from DR? Or when you start TDM as usual, then set com_automation 1 manually and connect from DR?
5. If TDM closes instantly, it means that it crashes. You might want to set "logFile 2", reproduce the issue, then find the right qconsole.log and attach it. If it crashes the hard way, then you can record crashdump maybe...
6. Does setting "com_smp 0" and "r_useParallelAddModels 0" help?

[MirceaKitsune]

5 hours ago, stgatilov said:

Does anyone else have this problem?

Most likely this is a problem in the game.

1. Did you have DR + TDM connection working reliably in the past?
2. Can you check if it works relialbly with TDM 2.10?
3. Do you have the same problem when starting the game manually?
4. Do you have same problem if you start TDM manually with "thedarkmod.x64 +set com_automation 1" then connect from DR? Or when you start TDM as usual, then set com_automation 1 manually and connect from DR?
5. If TDM closes instantly, it means that it crashes. You might want to set "logFile 2", reproduce the issue, then find the right qconsole.log and attach it. If it crashes the hard way, then you can record crashdump maybe...
6. Does setting "com_smp 0" and "r_useParallelAddModels 0" help?

1. I first tried it over a week ago. I only had the first problem with the cooldown based crash until yesterday, when suddenly it also started crashing all the time without me changing or updating anything.
2. Downgraded to 2.10 in the installer for a test: Exact behavior as 2.11 beta 1 which I'm now on.
3. No, never: I'm using "testmap mymap" from the console again now, always compiles and runs well.
4. Aha: Interestingly enough no issue this way! I used the console to launch TDM with automation, as well as launching TDM normally then setting "com_automation 1" in the console: Using the "(Re)Start game" button in DarkRadiant, TDM always compiles and goes to my map successfully. So it must be something when Radiant launches the engine executable.
5. Attached qconsole.log below. Generated for the timed crash... funny enough the new "connection lost unexpectedly" crash just fixed itself likely after the manual test for #4, I can use the feature again if I don't restart too often.
6. Doesn't seem to make a difference, at least for the timed crash which is the one I can still reproduce now.

Quote

ERROR: idTCP::Listen: bind: 98
ERROR:Automation cannot listen on port 3879

Just saw the log repeats that 5 times near the end right before proceeding to shut down. So the engine might be failing to listen to the proper port when it's launched by DR I imagine.

qconsole.log

Was able to reproduce the timed crash another way: Successfully launch TDM from DR once then quickly close it... now open TDM normally (still works fine), open the console and set "com_automation 1". The moment I set that cvar TDM will crash or shut down on me. And once more, if I just wait over 1 minute and try again it works this time. Wonder if the OS might be messing with the port or something, I get this even with the firewall disabled so not sure how... also whether I close TDM from the Quit button or Alt + F4 makes no difference.

Edited December 4, 2022 by MirceaKitsune

[stgatilov]

ERROR: idTCP::Listen: bind: 98
ERROR:Automation cannot listen on port 3879

Goggling shows this means EADDRINUSE on Linux.
So here the game fails to listen on its port because that port is already occupied by someone else.

Are you sure that the previous TDM process is dead by the time you try to spawn a new one?
At least on Windows, if you start loading mission then close TDM window, the window closes but the process silently continues working until loading is over.

Is there some other software which uses the same port?

[datiswous]

21 hours ago, MirceaKitsune said:

The issue is when the darkmod executable is launched by Dark Radiant

For me this also often, but not always leeds to wrong map loading or crashing. None of that when I manually activate game connection so I use that.

What I noticed is that if you have selected that the map is dmapped before loading it takes a lot of time before the dmap actually takes place (maybe like 8 seconds or so). In this time period, there is no hint that it will load.

Edited December 5, 2022 by datiswous

[MirceaKitsune]

Sorry for the delay. Not that I'm aware of, the port should normally be free. What I imagine must be happening is when TDM (launched by DR) is closed, Linux still sees the port as being busy for a while, despite even the process having been shut down. Never heard of such a thing before nor can easily imagine what must be happening, but at least it makes sense.

This might not be a TDM specific issue in this case... unless the engine fails to tell the OS to close the port in the way Linux expects? Though perhaps an improvement could be made: What if darkmod doesn't crash when the port is busy, but instead just doesn't receive signals until it's freed? Would a more dynamic approach to busy connection ports make sense?

[stgatilov]

It is pretty strange that it crashes.

But I can easily reproduce it on Windows.
Just open two TDM instances and set "com_automation 1" in both.
Obviously, it gets fatal error

That's not right.

[stgatilov]

In svn rev 10212, I replaced errors/crashing with 30 retries (once per second), ending with disabling com_automation.