Crashing on Training Mission_Builders Scene

[ricon]

Hi guys!

 

I am playing TDM. Congratulations to the team who did such great game!

 

Now, to the bug: i´ve played almost every mission on the Training Mission, and i am on the stealth part where i must use moss arrows and then knock out a guy. The problem is that when i try to run to the guy the game will freeze and shuts down

 

In the first few 2-3 tries nothing unusual would happen, but know i was really close to knok him down and the game, puff! Frozen...If i load the game again it will repeat that bug. I dont get it.

 

At last: i couldnt register myself here on the forum by using an hotmail account. I didnt receive any validation email.

[SiyahParsomen]

welcome to forum ricon!

 

If you explain your configuration and hardware, os etc. it'll be more easy to find a solution.

[Springheel]

Hmm, welcome to the forum. Not sure what might be causing that problem. When you say the game "shuts down" do you mean it crashes to desktop?

[ricon]

Well, first the game stops, freezes. Then i will see a window, from the Windows itself, saying that there was a problem and the program needs to be closed.

Also, i think that the game freezes when i run, so i need to check that. Let me try few more tests.

 

Sorry for not explaining myself better, but english is not my mother tongue.

 

EDIT:

 

No, its not from running. I loaded an older savegame, and run, etc etc and it was normal. Then, while in the stealth mission, but now other part of that mission (the one where i need to grab the keys from the walking guard), suddenly the game did the same thing again, but now with a diferent thing: i found a message saying RuntimeError, Doom3.exe needed to shut down because of the following error: "r6025 Pure virtual function call"

 

Any insight on my problem??

 

Thank you again

Edited January 1, 2012 by ricon

[nbohr1more]

Do you have an ATI video card?

 

If so, is Catalyst AI disabled?

[ricon]

My card is ATI Mobility Radeon HD 5650 with 1024 dedicated memory (i dont know if that is the word you use in english, "dedicated").

My computer uses an Intel Core i5 CPU M 450 2,4Ghz, 4GB RAM.

 

Sorry but i dont understand that much about computers, so whats the Catalyst AI? Is that a ingame option or an option from my video card? Should i put that on or off?

[nbohr1more]

Catalyst AI is a driver option for your video card.

 

You need to turn it off.

 

Unfortunately there is some question about whether this is possible for anything above the 4xxx series with the latest drivers.

 

You may need to resort to using a 3rd party tool or editing the driver and it's registry settings to accomplish this.

 

See this discussion to get an overview of the various workarounds people have tried:

 

http://forums.thedarkmod.com/topic/12398-disabling-catalyst-ai/

 

 

When The Dark Mod ships as it's own executable (targeted for v1.08 according to Bugtracker ?), this problem should no longer be an issue for ATI users as the driver

only applies the problematic "optimizations" when it sees Doom3.exe.

[ricon]

Thanks for your reply.

 

Since turning of Catalyst AI seems to demand some understanding on such stuff, i think i will prefer to wait for the 1.08 version (or anyone with its own executable).

[Serpentine]

Hmmm, sounds like one of those old not-power-of-two texture crashes on AMD cards. However nearly all of that should have been fixed in TDM, and it was also fixed in the driver (December 2010).

 

Make sure that your graphics driver is up to date.

 

As for the TDM executable, it most likely wont be a silver bullet either.

[Zunzster]

I'm also getting a bunch of crashes in the Builders section of the Training Mission.

 

I'm using TDM 1.07 with Doom 3 installed via Steam on a Windows 7 x64 machine with an NVIDIA graphics card. I've turned off the threaded optimisations as suggested. I've set the various image options to use compressed textures to reduce memory as suggested and I've patched Doom 3 to allow 4 Gb in case it was memory use. I was getting crashes during Quick Save as well leading to corrupted save files so I've set tdm_savegame_compress to 0 as suggested.

 

The crashes all appear to be Access Violations (0xc0000005) - most are in gamex86.dll at a pair of exception addresses. The others are at random high addresses outside all modules suggestive of overwritten stack frames.

 

Below is a sample:

 

Problem Event Name: APPCRASH

Application Name: Doom3.exe

Application Version: 1.0.0.1

Application Timestamp: 45ad459a

Fault Module Name: gamex86.dll

Fault Module Version: 0.0.0.0

Fault Module Timestamp: 4ed26a42

Exception Code: c0000005

Exception Offset: 002df4fa

OS Version: 6.1.7601.2.1.0.256.48

Locale ID: 5129

Additional Information 1: 89f2

Additional Information 2: 89f26f0bd4f5c9d709a560b9644b99b2

Additional Information 3: c168

Additional Information 4: c168d2090c2d3937a0c6a2c48501fdf2

 

The other address in gamex86.dll is at Exception Offset 00195a29 rather than 002df4fa.

 

Does anyone know where I might find a .MAP file for TDM's gamex86.dll to determine the faulting routine(s)?

[greebo]

Can you create a 32 Bit memory dump when the crash message is appearing, compress it and upload it somewhere?

 

Creating a 32 Bit Memory Dump on an x64 machine is done like this:

 

- Close any existing task manager windows you might have opened

- Open C:\Windows\SysWOW64\taskmgr.exe

- When D3 is crashing, switch to that Task Manager, right-click the DOOM3.exe process and save a dump file

- Compress it and upload it, please

[Zunzster]

As it happens, I still have the savegame which would will load successfully but which will immediately crash with combinations of those access violations when I try to quick save subsequently. So I can reliably repro the crashes at those exception addresses at will.

 

So I have 3 dump files now - one which has a fault at 00195a29, one which has a fault at 002df4fa which happened 2nd, and one which appears to be a stack fault.

 

Each dump is ~500Mb uncompressed and ~190Mb compressed. Eeps!

I can upload the quicksave as well if that helps with repro since it's only ~1.5Mb compressed.

 

I have access to our company ftpsite so I'll look at creating a folder on it for you with these in it and send you a username and password to access it.

[Zunzster]

OK, I've set up the ftp folder with the dumps and savegame and sent a private message with the access details.

[greebo]

Got the savegame, and I can confirm the crashes there. I haven't looked at the dumps yet, but there is indeed something very strange going on.

 

At the location you saved the game there is an idTriggerMulti, which is continuously triggering the GUI message (which is ok and intended). The triggering code causes a crash when trying to call idUserInterface->Trigger(), due to the vfptr being corrupted in some way. I can confirm that the idUserInterface* pointer is restored correctly (e.g. 0x1952b528) and it's working when loading the savegame, but in the trigger method the vfptr at that memory location 0x1952b528 is no longer the same as it was at load time, the vfptr is pointing to memory the process is not allowed to access. I have no idea how that came to be, I've never seen somthing like that happen before. I can only suspect that some other code that ran between loading and triggering is overwriting memory, and it happens to be that pointer. Even more curious: I can restore the vfptr manually by using VC++'s hex editor, and the game stops crashing, but the next time the trigger is called the vfptr is corrupted again.

[Zunzster]

Ah, a rogue memory overwrite of some kind - they can be nasty to track down since they can hit random places depending on alllocation patterns.

Given that this is a repeatable corruption with the savegame, can you put a debug register data breakpoint on the vfptr slot to catch the later write to it that corrupts it?

[greebo]

Got the problem. It's a rather fundamental problem in the Overlay system, as it's designed to actually destroy idUserInterface instances once the corresponding overlay is removed. It should not be allowed to do that though, since the same idUserInterface could be in use somewhere else, which was exactly the case in this scenario.

 

It's kind of surprising that this didn't crash much more often, since we use lots of those message GUIs throughout the training mission and we constantly call destroyOverlay() from the message scripts.

 

Thanks for providing the savegame (and the dumps, even though I didn't use them), it will be resolved in the next version. Tracker entry: http://bugs.angua.at/view.php?id=2976

[Zunzster]

I'm glad I was able to help and I'm impressed by the prompt attention and resolution. This is quite an amazing project!

 

If I understand you correctly, this was a 'use after free' gremlin and those can indeed be tricky bugs to catch since you often get away with them if the memory is question is not reused in the interim. I've not looked at the Doom3 source (yet!) but what kind of allocation management does it use? Traditional new and delete, class specific pools, or something more sophisticated?

 

At the risk of appearing to be 'trying to teach grandma to suck eggs', the usual technique I've used to catch these is an optional debug memory manager which 'poisons' the freed memory with a known value (e.g. the popular 0xdeadbeef) to avoid hiding use after frees. Extra points for seperately 'poisoning' the vtable pointer to point to a static VMT with a bunch of slots all pointing to a 'method called on freed object' exception thrower.

[Zunzster]

OK, I've decided to 'put my money where my mouth is' (so to speak) and take a stab at answering some of my own questions :-)

I've pulled down the TDM source, downloaded VC++ 2010 Express and I'll go do some exploring.

[Zunzster]

Damn, I've successfully built a gamex86.dll but you can't attach to process with VC++ 2010 Express edition.

[Zunzster]

Aha, you have to switch to Expert Settings to enable 'attach to process' in Visual Studio 2010 (and later). It might be worth updating the Wiki to that effect.

[greebo]

If I understand you correctly, this was a 'use after free' gremlin and those can indeed be tricky bugs to catch since you often get away with them if the memory is question is not reused in the interim. I've not looked at the Doom3 source (yet!) but what kind of allocation management does it use? Traditional new and delete, class specific pools, or something more sophisticated?

We're talking about id software here, which means that they do it all themselves and manually, using new & delete. No trace of shared_ptr constructs anywhere (my guess is because "it's slow" or something), we only use that in the TDM game code.

 

At the risk of appearing to be 'trying to teach grandma to suck eggs', the usual technique I've used to catch these is an optional debug memory manager which 'poisons' the freed memory with a known value (e.g. the popular 0xdeadbeef) to avoid hiding use after frees. Extra points for seperately 'poisoning' the vtable pointer to point to a static VMT with a bunch of slots all pointing to a 'method called on freed object' exception thrower.

This is what VC++'s debug heap routines are already doing, but DOOM3.exe hasn't been built with any of these debug options, it's using the release runtime. The idUserInterfaces are managed by DOOM3.exe, therefore the game DLL had no chance of detecting that.

 

That kind of problem will happen less likely now that we have a chance to build a complete debug build of the whole engine.

 

The compilation guide on the wiki is not fully up to date for VC++ 2010 since we've been using 2008 for the recent releases, but I'll update it as soon as the next TDM version is out, which is based on the GPL version of Doom3. You'll definitely need VC++ 2010 from then on, but people won't be able to compile the full engine using Express due to its lack of MFC support. Feel free though to sign up on the wiki and expand the existing article, if you like.

[Zunzster]

Duh, of course! TDM is plugging gamex86.dll into the pre-existing Doom3.exe which until the recent GPL release, you had no control over.

 

It's true that game programming culture has traditionally had quite a conservative streak. An obsession with 'performance issues' can also hide a reluctance to learn improved practices. Interestingly enough, John Carmack has come to much the same conclusion himself given his current advocacy of modern static analysis tools.

 

It's a pity that Express editions don't have MFC support since it will be hard to justify purchasing a full license of Visual Studio 2010 Professional.

 

However, I've found some links about adding ATL and MFC headers to Express editions to support at least compiling projects containing MFC code in Visual C++ 2008.

 

http://www.codeproject.com/KB/MFC/MFCinVisualStudioExpress.aspx

http://sambro.is-super-awesome.com/2009/05/15/atlmfc-dependancies-in-visual-studio-2008-express/

 

Essentially, they appear to be using the Windows DDK or Platform SDK to get the necessary headers and then plugging them into Visual Studio Express.

 

I might take a stab at trying those suggestions out when the time comes. Then we could include the details about the steps involved in the Wiki.

 

I confess I'm somewhat surprised that Doom3 actually uses the MFC given that Doom3 has both Mac and Linux editions. I assumed that id would have rolled their own UI code to abstract across the platforms. Obviously, they used MFC for that abstraction on Windows rather than using the raw Win32 API.

 

Where should I go to sign up for the Wiki? I had a look around on the Wiki itself but I can't seem to find it.

 

Also, is it possible to get read access to the SVN repository? I'm curious to see the source changes related to each bugtracker issue as a way of getting some understanding of the codebase.

[Serpentine]

The MFC stuff is just used for the built-in tools/editors (which are not build for linux/osx) - most of which are pretty awful/redundant. You can easily disable the building of them and use Express or even mingw (with some work).

 

As for svn access, talk to greebo really nicely (and just remember that at the moment the source is undergoing a lot of changes)

[greebo]

It's a pity that Express editions don't have MFC support since it will be hard to justify purchasing a full license of Visual Studio 2010 Professional.

 

However, I've found some links about adding ATL and MFC headers to Express editions to support at least compiling projects containing MFC code in Visual C++ 2008.

 

http://www.codeproje...dioExpress.aspx

http://sambro.is-sup...o-2008-express/

Yeah, I know these already, but that didn't work out, at least not for VC++ 2010 Express. The Windows 7 Driver Kit has really old ATL and MFC versions that are not quite compatible. I gave up after some time as it was too tiresome.

 

I confess I'm somewhat surprised that Doom3 actually uses the MFC given that Doom3 has both Mac and Linux editions. I assumed that id would have rolled their own UI code to abstract across the platforms. Obviously, they used MFC for that abstraction on Windows rather than using the raw Win32 API.

MFC is used for the main window and the tools/editors like the AF Editor, the Particle Editor and other stuff. Most of them have been quickly hacked together, that's probably why it has been enough for id to only support the Windows platform, since that's where 95% of the id people spend their time developing - I doubt they have anyone besides T. Besset using Linux or OSX systems. Given the amount of time and head-ache that can go into cross-platform editors I definitely understand that decision, from a project management point of view.

 

Still, even without the tools you still have to create the game's main window through MFC, so Express users seem to be out of luck with regards to compiling the engine project, at least for the moment being.

 

Where should I go to sign up for the Wiki? I had a look around on the Wiki itself but I can't seem to find it.

Ah, I think we disabled self-signup on the wiki due to spambots constantly screwing our pages. I manually created an account for you, there should be an e-mail on the way to you.

 

Also, is it possible to get read access to the SVN repository? I'm curious to see the source changes related to each bugtracker issue as a way of getting some understanding of the codebase.

We don't provide public read access to SVN at the moment. Unless the rest of the programming team also votes to have it public, I'd like to keep it that way.

[greebo]

That said, if you're actually planning to help with coding, I can of course create an account for you.