Jump to content
The Dark Mod Forums

Event limits


joebarnin

Recommended Posts

Another data point. I grabbed the save game from The Painter's Wife, and I was able to recreate the error ("Game Error. Event Overflow. Possible infinite loop in script"). I then ran my custom version of the DM exe that doubles the event limits. The error still happens. That indicates that this problem really is an infinite loop, and supports stgatilov's contention that upping the limit won't do any good.

On the other hand, the crash in Hidden Hands goes away when I double MAX_EVENTS.

Maybe these are two separate issues. Both errors happen in the event code, but in different places.

  • Like 1
Link to comment
Share on other sites

I'm trying to reproduce using this data in Hidden Lands, did not succeed from the first try, but succeeded on the second.

The culprit is idAI::_reEvaluateArea on two elementals: elemental_l1 and elemental_l2. They spam all the 10K events except for 60 normal events.

These events are sorted by their time, which increases from 10313870 (current time) to 10431966 (two minutes in the future). All for same area 307.

Investigating further.

  • Like 1
  • Thanks 2
Link to comment
Share on other sites

The code for flying AIs has a hack to remember closed doors as closed to avoid bumping into them infinitely. It schedules an event for itself 120 seconds into the future to recheck the area in case it has become available. The problem is that this event is scheduled again and again every frame. So it manages to push 7200 events per one elemental.

I guess the intent was to not do anything for an area if it is already "forbidden".
I would simply make AddForbiddenArea return a bool (true if the area is new and was added), and avoid posting event unless it is true.

Maybe @grayman would say something?
Here is the relevant code:

Spoiler

	// grayman #4412 - testing shows that dmap doesn't properly mark all portals for
	// flying AI. So the AI won't see a closed door as a solid wall. To solve this,
	// see if there's a nearby door and mark its AAS area unreachable if the AI
	// can't fly through the door. This allows him to seek an alternate route.

	if ( GetMoveType() == MOVETYPE_FLY )
	{
		idBounds bounds = physicsObj.GetAbsBounds();
		bounds[0].x -= 16;
		bounds[0].y -= 16;
		bounds[0].x += 16;
		bounds[0].y += 16;
		idEntity* ents[MAX_GENTITIES];
		int num = gameLocal.clip.EntitiesTouchingBounds( bounds, CONTENTS_SOLID, ents, MAX_GENTITIES );
		if (num > 0)
		{
			for ( int i = 0; i < num; i++ )
			{
				// check if there's a door
				idEntity *e = ents[i];

				if ( e == NULL )
				{
					continue;
				}

				if ( e->IsType(CFrobDoor::Type) )
				{
					CFrobDoor* frobDoor = static_cast<CFrobDoor*>(e);
					bool foundImpassableDoor = false;

					if ( frobDoor->IsOpen() )
					{
						if ( !FitsThrough(frobDoor) )
						{
							foundImpassableDoor = true; // can't fit through the open door
						}
					}
					else // can't go through the closed door
					{
						foundImpassableDoor = true; // can't go through the closed door
					}

					int areaNum = frobDoor->GetAASArea(aas);
					if ( foundImpassableDoor )
					{
						// add AAS area number of the door to forbidden areas
						gameLocal.m_AreaManager.AddForbiddenArea(areaNum, this);
						PostEventMS(&AI_ReEvaluateArea, doorRetryTime, areaNum);
						frobDoor->RegisterAI(this); // grayman #1145 - this AI is interested in this door
					}
					else
					{
						// door is passable, so remove its area number from forbidden areas
						gameLocal.m_AreaManager.RemoveForbiddenArea(areaNum, this);
					}
					break;
				}
			}
		}
	}

 

 

Link to comment
Share on other sites

Speaking of the second case with an elevator in Painter's Wife, it looks like an error in the custom script.

The script code for onMultiStateMoverLeave was:

    if (door.IsOpen())
    {
        door.Close();
        tdm_elevator elev;
        elev=mover;
        sys.waitFrame();
        elev.stopMoving();
        //mover.disable();
        while(door.GetFractionalPosition())
        {
            sys.waitFrame();
        }
        //elev.lastTriggeredBy.trigger(elev);
        elev.activate(elev.lastTriggeredBy);

So when you call and elevator, which is at some station with door open, it starts calling activate, which in turn calls onMultiStateMoverLeave again, creating an infinite loop.

  • Thanks 1
Link to comment
Share on other sites

I have implemented some code to simplify diagnostics of events overflow without a pocket coder 😉

 

When number of alive events exceeds 2K (controlled by g_eventAliveSoftLimit), a warning is posted to console every 5 seconds. Along with this warning, 10 currently alive events (controlled by g_eventNumberPrintedOnLimit) are printed to console. Half of them are chosen randomly, the other half are the very last in the queue. When something bubbles the event queue, all of the printed events will most likely be related to the problem. For instance, here is what I see for Hidden Hands:

WARNING:Soft limit of alive events exceeded (2048)! Some events printed below:
Event idAI::_reEvaluateArea on elemental_l2 at T=10351854
Event idAI::_reEvaluateArea on elemental_l1 at T=10353214
Event idAI::_reEvaluateArea on elemental_l2 at T=10354606
Event idAI::_reEvaluateArea on elemental_l1 at T=10359374
Event idAI::_reEvaluateArea on elemental_l1 at T=10363582
Event idAI::_reEvaluateArea on elemental_l2 at T=10368174
Event idAI::_reEvaluateArea on elemental_l2 at T=10368190
Event idAI::_reEvaluateArea on elemental_l1 at T=10368190
Event idAI::_reEvaluateArea on elemental_l1 at T=10368206
Event idAI::_reEvaluateArea on elemental_l2 at T=10368206
Total: 2048/10240 events alive
WARNING:Soft limit of alive events exceeded (2662)! Some events printed below:
Event idAI::_reEvaluateArea on elemental_l2 at T=10353470
Event idAI::_reEvaluateArea on elemental_l1 at T=10354414
Event idAI::_reEvaluateArea on elemental_l1 at T=10363566
Event idAI::_reEvaluateArea on elemental_l2 at T=10363630
Event idAI::_reEvaluateArea on elemental_l1 at T=10367598
Event idAI::_reEvaluateArea on elemental_l1 at T=10373182
Event idAI::_reEvaluateArea on elemental_l1 at T=10373198
Event idAI::_reEvaluateArea on elemental_l2 at T=10373198
Event idAI::_reEvaluateArea on elemental_l2 at T=10373214
Event idAI::_reEvaluateArea on elemental_l1 at T=10373214
Total: 2662/10240 events alive
...

Aside from that, the new command listEvents prints all the currently alive events to console.

 

When number of events processed in single frame exceeds 5K (controller by g_eventPerFrameSoftLimit), warning is posted to console. In fact, two warnings are posted: the first when it hits the limit, and the second when processing ends (if it ends properly, of course). First 10 events (controlled by g_eventNumberPrintedOnLimit) after exceeding the limit are printed to console between the two warnings. Here is how it looks in Painter's Wife:

WARNING:Soft limit of 5120 events per frame exceeded! Some events printed below:
Event idThread::<immediateremove> on multistateposition::onMultiStateMoverArrive at T=20318312
Event idThread::<execute> on tdm_elevator::storeTrigger at T=20318312
Event idThread::<execute> on multistateposition::onMultiStateMoverLeave at T=20318312
Event CMultiStateMover::propagateSound on marlow_elevator at T=20318312
Event CFrobDoor::propagateSound on scissorgate_1 at T=20318312
Event idThread::<immediateremove> on multistateposition::onMultiStateMoverLeave at T=20318312
Event idThread::<immediateremove> on tdm_elevator::storeTrigger at T=20318312
Event idThread::<execute> on multistateposition::onMultiStateMoverArrive at T=20318312
Event CFrobDoor::propagateSound on scissorgate_1 at T=20318312
Event idThread::<immediateremove> on multistateposition::onMultiStateMoverArrive at T=20318312

This spam gets posted every frame when the limit is exceeded, without any speed limit. I think if someone manages to consistently process 5K+ events per frame, the game will slow down dramatically, and playing it won't be nice anyway.

Interestingly, there are several moments when I see 1100 or 2700 events processed in one frame during game start. These events are usually something like calling postspawn function for most of the entities. If we see a problem with it in the future, I suggest increasing the limit x10 times during the very first frame after game start of game load.

 

UPDATE: Created issue 5270 for improving events overflow diagnostics --- that's worth being tracked.

  • Like 1
  • Thanks 1
Link to comment
Share on other sites

Committed all the changes to trunk:

8761-8764: new diagnostics for event overflow error.

8765-8766: new cvars: crash on error (allows saving crashdump), promote warnings to errors.

8767: fixed the main problem with flying AIs which caused event spam in Hidden Hands.

  • Thanks 4
Link to comment
Share on other sites

@stgatilov thanks very much for looking into this, without neither a "pocket coder" nor some kind of event diagnostics itd have been tough to figure out what's going on behind the scenes.

Regarding the elevator script not respecting wait commands, I'll open a thread in the dev forum.

7 hours ago, stgatilov said:

Interestingly, there are several moments when I see 1100 or 2700 events processed in one frame during game start. These events are usually something like calling postspawn function for most of the entities. If we see a problem with it in the future, I suggest increasing the limit x10 times during the very first frame after game start of game load.

If the first frame regularly exceeds the soft limit of 2k, would it not make sense to right now raise the limit or ignore the first frame, instead of waiting for problems to occur in the future?

Link to comment
Share on other sites

On 6/7/2020 at 7:44 PM, stgatilov said:

Committed all the changes to trunk:

8761-8764: new diagnostics for event overflow error.

8765-8766: new cvars: crash on error (allows saving crashdump), promote warnings to errors.

8767: fixed the main problem with flying AIs which caused event spam in Hidden Hands.

 

Thank you very much for fixing these issues!  

After beta testing and integration of the briefing video, I will wait for 2.08 to be released before publishing the mission.

Edited by JackFarmer
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recent Status Updates

    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 0 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
    • OrbWeaver

      I like the new frob highlight but it would nice if it was less "flickery" while moving over objects (especially barred metal doors).
      · 4 replies
    • nbohr1more

      Please vote in the 15th Anniversary Contest Theme Poll
       
      · 0 replies
×
×
  • Create New...