Large Maps Do Not Display Properly [closed]

[OrbWeaver]

As discussed in the "Itches and glitches" thread, large maps such as mansion_alpha display incorrectly, with almost everything missing the XY window. Windows-only bug, reported as unpredictable (sometimes a reload fixes it).

 

Under investigation. Initial testing reveals that a mansion_alpha with all entities apart from worldspawn removed (in the .map file) loads OK, so not purely a size problem.

[OrbWeaver]

I tried to narrow it down to a particular entity, but is is too unpredictable for that. I can get the error to occur in a map of around 75 entities (700 - 775 of mansion_alpha.map), so it not a size issue.

 

The fact that the green "target" lines display, but sometimes go off to infinity suggests that there is some problem cause by these lines. I will try to isolate the code that draws them and see if this changes anything.

[OrbWeaver]

Well it's not the target lines -- I can disable them by commenting out RenderableConnectionLines::attach() in entity/targetable.h and the only change is the NOTHING shows up (rather than just the target lines) in affected maps.

 

So much for that idea; it must be something else in the entity definitions causing the problem.

 

EDIT: Perhaps it is the "rotation" keys on the entities, some of them are using very small exponents. TODO: remove all rotation keys with grep and see if problem still occurs.

 

EDIT2: Nope, discarding rotation keys does not solve the problem. Interestingly, saving the map to a new file seems to make it re-appear.

[OrbWeaver]

I have now been able to create this error on Linux, loading askave.map. This is not reliable, and I cannot now recreate it since the initial two instances.

 

EDIT: Now recreated a few times on Linux with askave.map, but never at all with mansion_alpha.map

[greebo]

Just an idea: Could it have something to do with the dummy brushes? (As re-saving seems to facilitate this error)

[OrbWeaver]

No, I meant just saving causes the map to re-appear, not saving then reloading. The dummy brushes won't exist at this point because they are inserted into the token stream during saving, never the scenegraph itself.

 

I think it is going to be necessary to produce some more helpful debugging tools. How easy do you think it would be to add another XML file to the registry - "debug.xml" - which would only be loaded and use in Debug builds, and would allow the mapdoom3 plugin to load or discard particular ranges of entities, particular keys and particular types of entity (so I could say "load everything except the func_statics")?

 

EDIT: Actually this should be very easy indeed. I will have a look at doing this.

[greebo]

Yes, this would be very easy, just add the appropriate statement into mainframe.cpp.

 

I would suggest adding a " key into the main user.xml file, that triggers the loading of this debug.xml. Or to be even more flexible with naming the file. This way you wouldn't even have to recompile.

[OrbWeaver]

I was planning to put the load statement between "#ifdef _DEBUG" guards, so that the code wouldn't even exist in Release builds -- but now I think about it, this won't be sufficient because many Windows problems ONLY occur in Release builds, so I may well do what you suggest.

[greebo]

How's it going in this matter, Orbweaver? Could you find out more about this?

 

Earlier today I thought about this and I believe all these large maps like Askave and Bonehoard are using emitter entities or at least entity connections. Maybe there is something amiss with drawing the arrows or the entity connections messing up the OpenGL state?

[OrbWeaver]

The debug.xml handling is in, and I have started work on adding "discard entity types" debug code in the mapdoom3 plugin. This in turn will require refactoring the parsing code since it is somewhat convoluted, which is where I am now.

 

Once I have done this, I will add "only load entities from a certain numeric range" debug functionality as well. With these tools it should be much easier to narrow down the problem to a specific entity or entity type.

[greebo]

Would this also facilitate the display of which entity is being loaded during the map loading process? (Currently, only the entity number is displayed.)

 

This is of course just a nice-to-have feature, nothing of importance

[OrbWeaver]

Would this also facilitate the display of which entity is being loaded during the map loading process? (Currently, only the entity number is displayed.) This is of course just a nice-to-have feature, nothing of importance

 

It may well be possible to display the classname, although I think it would change too quickly to be of any diagnostic value.

 

Ideally I would like to have a proper progress bar rather than a "unknown length" pulsing bar, but this would require knowing either the position in the stream or the total number of entities, neither of which are possible with the current implementation (unless we added some DarkRadiant-specific comments at the beginning of the file, such as "entity count").

[greebo]

Hm, it would not be worth the hassle, I guess. If displaying the classname is easy to accomplish, this would be enough. At least there is something to look at during the (long) loading pauses.

[OrbWeaver]

Class-based entity discarding is now in. Next to come are loading entities from a numeric range, and discarding a particular keytype.

 

So far I have been able to load askave.map without problems while discarding func_static and worldspawn entities, but on Linux the error is very unpredictable anyway so this doesn't prove much. I will need to test this on Windows.

[greebo]

Another thought, as the light rendering code has been changed during the last weeks: Can you already filter light entities? Maybe it's special spawnargs for projected lights or lights using the "light_center" key with the accompanying render code?

 

I also managed to produce this error yesterday with aitest2.map and aitest3.map, just to mention it here.

[OrbWeaver]

Both class-based and range-based entity filtering a now committed, so feel free to experiment (the example debug.xml contains both keys, but you will need to set the "debug" key to 1 in user.xml to activate them).

[greebo]

Ok, but I will first try to commit my changes before I start experimenting in this matter, perhaps I will be able to accomplish something in the evening.

[OrbWeaver]

There does not seem to be any logic behind this at all. One minute I can reproduce the bug by loading entities 120-165 of askave.map, the next minute it is loading those entities fine, and even expanding back to 100-175 doesn't reproduce the error even though it did previously.

 

Removing ALL light-based classes seems to stop the error, removing all except light_elemental displays the error, but removing light_elemental alone does not solve anything. Removing all light based entities except light_elemental but ALSO removing path_corner DOES solve the error. It almost seems like it is dependent on the number of entities in total, except there is no logic to this either - it can load 859 entities and display them OK, then fail to display the first 500, and then start displaying the first 500 again.

[greebo]

Is it possible that there is an error in the GL commands? Or are the graphics cards (or their drivers resp.) foolproof enough to handle such (stupid) situations. As you may notice I don't have a thorough understanding of OpenGL yet, but I could imagine that some rendering routines leave the GL state somewhat "open" or "undefined" and subsequent rendering routines are blarged up thereafter.

[OrbWeaver]

I don't think we have made any changes to OpenGL code, so I doubt it would be a raw OpenGL problem, although it could very well be a sorting/bounding box/culling problem (front-end rendering).

 

Do you know when this problem first appeared? You mentioned that you and angua had seen it before -- is this in 0.7.1 or an SVN build?

[greebo]

I can't say exactly, but I guess it appeared the first time somewhere in the beginning of December. The bonehoard map wasn't that large before December, so it may well be that we just didn't notice the error.

 

Some testing with a previous release (0.6.0 or anything like that) should make checking easy. I will ask angua to check it if she finds some time this afternoon.

[OrbWeaver]

When did you do the refactoring of code which enabled the multiple XY-window thing? I have no idea whether this is relevant, but it is the only change to the XY view that I can remember. I made some changes to the md5model and shaders plugins in December, but I can't see how that would effect rendering of entities in the XY window.

 

It may be that I have to get into the render code and actually see what happens when an entity is/is not rendered, since the "exclude entities on load" tests don't seem to reveal anything useful.

[greebo]

I refactored this about two weeks ago, but we observed this problem definitely before the XYView update, so this can be excluded.

[angua]

It seems to be at least partially related to the world origin. In xz or yz view, if I can see the map at first and the move the origin out of the sceen at the upper edge of the ortho-view, everything is gone. If I move it back, everything's there again. This doesn't work in the xy-top-view, and I'm sure that this is not the only problem, as I often don't see anything even if the origin is on the screen.

 

Edit: no, this was wrong. In bonehoard, everything disappears when the height of 512 is moved out at the upper edge while the origin is well inside the visible part of the map (and it's the only thing still showing up...).

 

Edit2: and even this is not reproducable

 

More edit: but it always helps in Askave to delete all the lights

[OrbWeaver]

That is very interesting, it does look like there is a problem with culling, possibly involving uninitialised values (hence the unpredicatability, and the tendency of the error to stop happening after a few attempts).

 

The lights thing is also interesting -- greebo, how does the light_center interact with the worldAABB of the LightInstance? Does it increase it, or just fall outside of it? I think you suggested that there could be something going on with light centers outside the bounding box, which may be worthy of investigation.

 

I am going to start splitting/refactoring/examining the front-end render classes, so I can see what is going on (and perhaps remove the THREE templated wrapper classes involved in the process - CullingWalker, ForEachVisible and RenderHighlighted).