Zunzster

OK. It turns out this is a known issue and is fixed in pugixpath version 1.0 - TDM is currently using version 0.5. Given the version drift and potential interface and/or behaviour changes, I'm not sure if it's worth the churn of upgrading given that it's only a few lines to patch.

From a brief bit of googling, the standard appears to be silent on this issue so I'd say it's implementation defined whether passing NULL to std::copy is allowed. For anything other than an empty source, it will fault so I think what VC++ is doing is perfectly defensible and what pugixpath is doing is very dubious. Given that m_end is never NULL other than due to that specific line in operator=(), it is essentially breaking it's own invariant IMHO. A minor bit of bad design that happens to survive in practice most of the time only to leap up and bite innocent parties down the road :-) I've logged an issue with pugixpath on googlecode. http://code.google.c...s/detail?id=143 Maybe they'll agree and maybe they won't :-)

Thanks for the Wiki account, greebo - I've since added a note about turning on Expert Settings in Visual Studio Express 2010. From some commentary on the Doom3 GPL project on Github, https://github.com/TTimo/doom3.gpl/commit/d422788143f3dab35153480e76c437dd5f139484 I gather that you can get more up to date ATL and MFC headers via the Platform SDK (as opposed to the DDK) but that still doesn't help. Whilst that will now compile, it will fail to link since the 32-bit MFC libraries are not present in the Platform SDK. Curiously, the 64-bit MFC libraries are included so maybe there is hope for working with any eventual 64-bit DOOM 3 builds using the Express Editions. Also Serpentine said you can potentially stub out the MFC code if you're feeling dedicated. Thanks for clarifying the projects SVN repository policy.

I don't know whether this truly qualifies as a bug, maybe it's more of a gotcha, a difference of assumptions between pugixpath and the VC++ Debug STL implementation. When compiling gamex86 as a debug build, downloading the mission list will cause a "invalid null pointer" exception. The problem is triggered by the following line in MissionManager.cpp in the CMissionManager::LoadModListFromXml method. // Localisation packs pugi::xpath_node_set l10PackNodes = node.select_nodes("localisationPack"); This produces an empty xml node set from the query which it attempts to return via xpath_node_set::operator=() This routine sets m_begin and m_end to NULL before falling through and calling into xpath_node_set::append() with an empty set. m_begin = m_end = m_eos = 0; m_type = ns.m_type; if (ns.size() == 1) { m_storage = *ns.m_begin; m_begin = &m_storage; m_end = m_eos = &m_storage + 1; } else { append(ns.begin(), ns.end()); } This is arguably a bit dubious since if you look at the xpath_node_set constructor, m_begin and m_end are not NULL there but point within the m_storage single static node. The problem with them being NULL is that the Debug version of std::copy() in the VC library asserts that it's destination is not NULL even when the source iterators imply an empty collection. Changing xpath_node_set::operator=() to the following avoids the issue: m_begin = m_end = &m_storage; m_eos = &m_storage + 1; m_type = ns.m_type; if (ns.size() == 1) { m_storage = *ns.m_begin; m_end = &m_storage + 1; } else { append(ns.begin(), ns.end()); } As far as I can tell, the documentation on std::copy is silent on whether NULL is a valid destination iterator when the source range is empty. Given that pugixpath is an included package, maybe passing this issue upstream is a better course.

Duh, of course! TDM is plugging gamex86.dll into the pre-existing Doom3.exe which until the recent GPL release, you had no control over. It's true that game programming culture has traditionally had quite a conservative streak. An obsession with 'performance issues' can also hide a reluctance to learn improved practices. Interestingly enough, John Carmack has come to much the same conclusion himself given his current advocacy of modern static analysis tools. It's a pity that Express editions don't have MFC support since it will be hard to justify purchasing a full license of Visual Studio 2010 Professional. However, I've found some links about adding ATL and MFC headers to Express editions to support at least compiling projects containing MFC code in Visual C++ 2008. http://www.codeproject.com/KB/MFC/MFCinVisualStudioExpress.aspx http://sambro.is-super-awesome.com/2009/05/15/atlmfc-dependancies-in-visual-studio-2008-express/ Essentially, they appear to be using the Windows DDK or Platform SDK to get the necessary headers and then plugging them into Visual Studio Express. I might take a stab at trying those suggestions out when the time comes. Then we could include the details about the steps involved in the Wiki. I confess I'm somewhat surprised that Doom3 actually uses the MFC given that Doom3 has both Mac and Linux editions. I assumed that id would have rolled their own UI code to abstract across the platforms. Obviously, they used MFC for that abstraction on Windows rather than using the raw Win32 API. Where should I go to sign up for the Wiki? I had a look around on the Wiki itself but I can't seem to find it. Also, is it possible to get read access to the SVN repository? I'm curious to see the source changes related to each bugtracker issue as a way of getting some understanding of the codebase.

Aha, you have to switch to Expert Settings to enable 'attach to process' in Visual Studio 2010 (and later). It might be worth updating the Wiki to that effect.

Damn, I've successfully built a gamex86.dll but you can't attach to process with VC++ 2010 Express edition.

OK, I've decided to 'put my money where my mouth is' (so to speak) and take a stab at answering some of my own questions :-) I've pulled down the TDM source, downloaded VC++ 2010 Express and I'll go do some exploring.

I'm glad I was able to help and I'm impressed by the prompt attention and resolution. This is quite an amazing project! If I understand you correctly, this was a 'use after free' gremlin and those can indeed be tricky bugs to catch since you often get away with them if the memory is question is not reused in the interim. I've not looked at the Doom3 source (yet!) but what kind of allocation management does it use? Traditional new and delete, class specific pools, or something more sophisticated? At the risk of appearing to be 'trying to teach grandma to suck eggs', the usual technique I've used to catch these is an optional debug memory manager which 'poisons' the freed memory with a known value (e.g. the popular 0xdeadbeef) to avoid hiding use after frees. Extra points for seperately 'poisoning' the vtable pointer to point to a static VMT with a bunch of slots all pointing to a 'method called on freed object' exception thrower.

Ah, a rogue memory overwrite of some kind - they can be nasty to track down since they can hit random places depending on alllocation patterns. Given that this is a repeatable corruption with the savegame, can you put a debug register data breakpoint on the vfptr slot to catch the later write to it that corrupts it?

OK, I've set up the ftp folder with the dumps and savegame and sent a private message with the access details.

As it happens, I still have the savegame which would will load successfully but which will immediately crash with combinations of those access violations when I try to quick save subsequently. So I can reliably repro the crashes at those exception addresses at will. So I have 3 dump files now - one which has a fault at 00195a29, one which has a fault at 002df4fa which happened 2nd, and one which appears to be a stack fault. Each dump is ~500Mb uncompressed and ~190Mb compressed. Eeps! I can upload the quicksave as well if that helps with repro since it's only ~1.5Mb compressed. I have access to our company ftpsite so I'll look at creating a folder on it for you with these in it and send you a username and password to access it.

I'm also getting a bunch of crashes in the Builders section of the Training Mission. I'm using TDM 1.07 with Doom 3 installed via Steam on a Windows 7 x64 machine with an NVIDIA graphics card. I've turned off the threaded optimisations as suggested. I've set the various image options to use compressed textures to reduce memory as suggested and I've patched Doom 3 to allow 4 Gb in case it was memory use. I was getting crashes during Quick Save as well leading to corrupted save files so I've set tdm_savegame_compress to 0 as suggested. The crashes all appear to be Access Violations (0xc0000005) - most are in gamex86.dll at a pair of exception addresses. The others are at random high addresses outside all modules suggestive of overwritten stack frames. Below is a sample: Problem Event Name: APPCRASH Application Name: Doom3.exe Application Version: 1.0.0.1 Application Timestamp: 45ad459a Fault Module Name: gamex86.dll Fault Module Version: 0.0.0.0 Fault Module Timestamp: 4ed26a42 Exception Code: c0000005 Exception Offset: 002df4fa OS Version: 6.1.7601.2.1.0.256.48 Locale ID: 5129 Additional Information 1: 89f2 Additional Information 2: 89f26f0bd4f5c9d709a560b9644b99b2 Additional Information 3: c168 Additional Information 4: c168d2090c2d3937a0c6a2c48501fdf2 The other address in gamex86.dll is at Exception Offset 00195a29 rather than 002df4fa. Does anyone know where I might find a .MAP file for TDM's gamex86.dll to determine the faulting routine(s)?