Crash in 2.12

[joebarnin]

My new FM is in beta test, and some testers are using 2.12. They have discovered a hang/crash that only happens in 2.12 (crash doesn't happen in 2.11). I am able to recreate it in 2.12 now. I run TDM in the Visual Studio debugger (using dev16829-10455). After starting the mission, I just wait a minute or so and the screen freezes. Then, I wait another couple of minutes, and the debugger reports a failed assertion. The assertion is:

WARNING:ASSERTION FAILED! E:\games\darkmod_source212\idlib\containers\List.h(394): 'newsize >= 0'

The callstack is:

     TheDarkModx64_debug.exe!AssertFailed(const char * file, int line, const char * expression) Line 75    C++
>    TheDarkModx64_debug.exe!idList<idPlane>::Resize(int newsize) Line 394    C++
     TheDarkModx64_debug.exe!idList<idPlane>::AddGrow(idPlane obj) Line 765    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 351    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 385    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 385    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 385    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 385    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 385    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 385    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 385    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 385    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 385    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 385    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 385    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 385    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 385    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FloodLightThroughArea_r(idRenderWorldLocal::FlowLightThroughPortalsContext & context, int areaNum, const portalStack_s * ps) Line 385    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::FlowLightThroughPortals(idRenderLightLocal * light, idFlexList<int,128> * areaIds, lightPortalFlow_t * portalFlow) Line 495    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::AddLightToAreas(idRenderLightLocal * def) Line 2038    C++
     TheDarkModx64_debug.exe!R_CreateLightRefs(idRenderLightLocal * light) Line 738    C++
     TheDarkModx64_debug.exe!idRenderWorldLocal::UpdateLightDef(int lightHandle, const renderLight_s * rlight) Line 423    C++
     TheDarkModx64_debug.exe!idLight::PresentLightDefChange() Line 1076    C++
     TheDarkModx64_debug.exe!idLight::Present() Line 1132    C++
     TheDarkModx64_debug.exe!idEntity::Think() Line 2318    C++
     TheDarkModx64_debug.exe!idLight::Think() Line 1274    C++
     TheDarkModx64_debug.exe!idGameLocal::RunFrame(const usercmd_t * clientCmds, int timestepMs, bool minorTic) Line 3366    C++
     TheDarkModx64_debug.exe!idSessionLocal::RunGameTic(int timestepMs, bool minorTic) Line 3063    C++
     TheDarkModx64_debug.exe!idSessionLocal::RunGameTics() Line 3109    C++
     TheDarkModx64_debug.exe!idSessionLocal::FrontendThreadFunction() Line 3159    C++
     TheDarkModx64_debug.exe!idSessionLocal::StartFrontendThread::__l2::<lambda>(void * x) Line 3235    C++
     TheDarkModx64_debug.exe!unsigned int <lambda>(void *)::<lambda_invoker_cdecl>(void * x) Line 3236    C++
     [External Code]    

The code in question:

ID_INLINE int idList<type>::AddGrow( type obj ) {
    if ( num == size ) {
        int newsize;

        if ( granularity == 0 ) {    // this is a hack to fix our memset classes
            granularity = 16;
        }
        newsize = (size * 3) >> 1;            // + 50% size
        newsize += granularity;                // round up to granularity
        newsize -= newsize % granularity;    //
        Resize( newsize );
    }

    list[ num ] = obj;
    num++;

    return num - 1;
}

size = 968651120, so newsize is calculated as  -694506944, which causes the assertion.

Obviously something is running away and the code ends up trying to resize the list to a size that is so big that it wraps around to negative. Is my mod doing something wrong that is leads to this? How do I track this down? I can supply the FM package if that will help. Or, I can set some breakpoints if there's something I can look at here.

Thanks.

Edit: Here's the FM: https://www.dropbox.com/scl/fi/3fg2x3w4owmku1l4gifi7/altham_crash.pk4?rlkey=z0q1gaflz5b7mifeuayoqmlug&dl=1

To reproduce the problem, start the mission (Easy difficulty), then just stand there. The guard will walk close to you but he shouldn't see you (you can always set the "notarget" console command to ensure he doesn't see you). After a few minutes, the game freezes. A few minutes after that, it crashes. Sometimes it happens in just a minute or two; other times it takes several minutes. But so far, it always happens eventually.

Edited October 18, 2023 by joebarnin
provide test case

[stgatilov]

Wow!
This is something interesting.

You have areas 177 and 179 (you can detect them using teleportArea command) with 8 visportals between them, all in same plane.
I hope this is as intended, and you really have many separate visportals?
This should be perfectly valid of course.

FloodLightThroughPortals searches for all visportal sequences which might get some light rays through.
For some reason, it manages to go back and forth between areas 179 and 177 something like eight times in the same sequence.

It is not forbidden for the traversal to go through the same area many times, but it is forbidden to go through oriented visportal twice. However, there are too many possible sequences of going through 8 two-sided visportals, and the code obviously tries to enumerate and save them all. At some point it bumps into 32-bit integer limits and everything crashes. By this moment it has wasted 32 GB of RAM so I don't think it would live long even if it did not crash here.

You don't have the issue in 2.11 because:

1. This is a dynamic light from some lantern bot, TDM 2.11 only runs this function for static lights.
2. TDM 2.11 does not try to save all sequences, it would only enumerate them in horrendously slow time.

I think some lantern bot crosses the visportal between areas 179 and 177 at this moment, causing the issue.
Also, you can most likely reproduce it in 2.11 by putting a large enough light at the same location (12565.0205, 411.300110, 522.249939).

[stgatilov]

I guess I'll try to reproduce it in a testmap.

Like there are two rooms with a planar wall between them.
There are 25 windows in this wall (e.g. arranged as 5 x 5 grid), each window has a visportal.
There is a moveable light source which covers the whole wall.
If player tried to move it between rooms (e.g. through a door), then everything hangs up and crashes, hopefully

[joebarnin]

Thanks for the explanation. The building where this happens is an "open plan" warehouse, 4 stories, with a wall at one end but 2 doorways in the wall on each side, so 8 total:

Each doorway is a visportal. They are lined up on the X-plane.

First question, are all of these portals unnecessary? They do reduce the tris somewhat, but maybe not worth it? Second question, I recall occasional problems when visportals are lined up exactly. Maybe I'll move each of them a slightly different amount in the x direction, see if that helps.

I'll experiment and let you know what I find.

[joebarnin]

So I adjusted each portal slightly in the x direction. The portals were 8 x 112 x 128 (x,y,z), and I moved the x-plane of each portal by a different amount (1 unit, 2 units, 3 units, etc), so that none of them are in the same plane. And now it no longer crashes! So I have a workaround, at least.

Thanks for identifying the root of the problem! Let me know if there's anything else I can do; for now, I'm going to proceed with my workaround.

[stgatilov]

Actually, this is a duplicate of 3815.

The portals traversal code is the same for view and for light.
But for view it was hardened against this issue, while for light the issue was very unlikely to happen.

[stgatilov]

Fixed the engine in svn rev 10464.

Also I noticed something really weird (at 13227 1608.52 1110.76)

I suppose dmap caused this?...

[joebarnin]

I see that too (in 2.12, not in 2.11). I've seen issues like this before, occasionally. I usually just fiddle with the brushes until the problem goes away.

[joebarnin]

1 hour ago, stgatilov said:

Fixed the engine in svn rev 10464.

 

Great news, thanks! Do you know when this be available in a dev build?

[stgatilov]

20 hours ago, joebarnin said:

Great news, thanks! Do you know when this be available in a dev build?

Mmm... whenever a new dev build would be published?...

The schedule for dev builds is rather unstable these days.

[joebarnin]

No worries, I’ve got a workaround in my current FM so testing it on 2.12 can continue.