dmap and LOD implementation

[motorsep]

So we implemented LOD last night. It works, but there is an issue with map's geometry (and inlined entities). Apparently dmap merges surfaces with identical materials into one large surfaces (and that could cause longer map compiling times too). For LOD to work properly, each surface has to be written as separate surface.

 

Is it possible to make dmap write each surface as separte one, regardless if materials are identical ? Where in the code dmap does the sorting and merging ?

 

Thanks.

[OrbWeaver]

I don't know about the dmap code (which would be needed if you wanted to disable this globally), but it can be disabled for a specific material using the "discrete" global keyword.

[nbohr1more]

If I read Fabien's code review correctly, this happens at "PutPrimitivesInAreas".

 

I will take a closer look myself and see if I can grok anything...

[SteveL]

Have a look at the list of keywords in dmap.cpp function "Dmap". I think there's one called noOptimize or something like that and I suspect it's the optimize function that does the stitching (which as nbohr1more said is either part of or runs just after PutPrimitivesInAreas).

 

Optmize was what slowed down my test of inlining 2000 touching barrels at the weekend. The further down the list of barrels it got, the longer each one took.

[motorsep]

I recall noOptimize for dmap is to not cut surfaces on the light boundaries. I'll try "discreet" flag in the material and report back.

[nbohr1more]

OK, I couldn't fathom it elsewhere but Orbweaver's suggestion is akin to my work with the spectrum keyword hack.

 

usurface.cpp

 

   // set merge groups if needed, to prevent multiple sides from being
// merged into a single surface in the case of gui shaders, mirrors, and autosprites
if ( s->material->IsDiscrete() ) {
for ( tri = triList ; tri ; tri = tri->next ) {
tri->mergeGroup = (void *)s;

 

 

basically, add another branch which polls your LOD attribute here and that would do the trick.

Edited November 25, 2014 by nbohr1more

[OrbWeaver]

I recall noOptimize for dmap is to not cut surfaces on the light boundaries

 

That never happens by default; you need to specify it explicitly with the "lightCarve" option.

 

(Of course that's not to say that "noOptimize" doesn't also disable this functionality, but it would be rather pointless since it's not enabled in the first place.)

[SteveL]

I recall noOptimize for dmap is to not cut surfaces on the light boundaries. I'll try "discreet" flag in the material and report back.

 

You might well be right, although like Orbweaver I thought that was lightcarve. I don't have it in front of me.

 

But this discussion makes me think my test of "inline" (which made me reject fixing it at the weekend) might have been a bad one. I'll try out dmapping those 2000 inlined barrels again with them not touching and see if the problem goes away.

 

It won't help the map bloat of course, but if it gets rid of the shadow errors and the hours-long dmap time, that would make it usable.

[OrbWeaver]

I'll try out dmapping those 2000 inlined barrels again with them not touching and see if the problem goes away. It won't help the map bloat of course, but if it gets rid of the shadow errors and the hours-long dmap time, that would make it usable.

 

Admittedly I'm not all that familiar with what you're trying to achieve, but my gut feeling is that trying to inline this much geometry is a workaround for a problem that would be better fixed at source, e.g. by increasing entity limits, improving the efficiency of rendering large numbers of duplicated entities or whatever.

[motorsep]

OK, I couldn't fathom it elsewhere but Orbweaver's suggestion is akin to my work with the spectrum keyword hack.

 

usurface.cpp

 

// set merge groups if needed, to prevent multiple sides from being
// merged into a single surface in the case of gui shaders, mirrors, and autosprites
if ( s->material->IsDiscrete() ) {
for ( tri = triList ; tri ; tri = tri->next ) {
tri->mergeGroup = (void *)s;

 

 

basically, add another branch which polls your LOD attribute here and that would do the trick.

 

Thanks. I think adding 'discrete' flag should solve it. I'll test it and if it doesn't work I'll add another condition to check for lod surfaces and not merge them. But I have a gut feeling 'discrete' is all I need

[motorsep]

It won't help the map bloat of course, but if it gets rid of the shadow errors and the hours-long dmap time, that would make it usable.

 

Map bloat isn't a problem. That's how maps are in other engines. Older engines were even worse, having lightmaps embedded and models inlined. Some levels in Steel Storm are 80+Mb and that doesn't have negative effect.

 

So the only issue I see is longer compile times (if you get a rid of shadow issues, which I haven't experienced yet).

 

Although I would prefer having entities that don't count as entities

[SteveL]

Admittedly I'm not all that familiar with what you're trying to achieve, but my gut feeling is that trying to inline this much geometry is a workaround for a problem that would be better fixed at source, e.g. by increasing entity limits, improving the efficiency of rendering large numbers of duplicated entities or whatever.

Map bloat isn't a problem. That's how maps are in other engines. Older engines were even worse, having lightmaps embedded and models inlined. Some levels in Steel Storm are 80+Mb and that doesn't have negative effect. So the only issue I see is longer compile times (if you get a rid of shadow issues, which I haven't experienced yet). Although I would prefer having entities that don't count as entities

Absolutely agreed both. Inlining is a step backwards when what we really want is better instancing by whatever method rather than duplicated geometry. Combining surfaces from all the "inert" models into one entity is one thread of last weekend's discussion, instancing for repeated models another. But the context of the inline testing was a map that Biker has *right now* that's exceeding the limit, and inlining would a 3-line fix in the code which could happen right now. All that's needed is to stop the inlined entity spawning, all the rest is already in dmap.

[motorsep]

Absolutely agreed both. Inlining is a step backwards when what we really want is better instancing by whatever method rather than duplicated geometry. Combining surfaces from all the "inert" models into one entity is one thread of last weekend's discussion, instancing for repeated models another. But the context of the inline testing was a map that Biker has *right now* that's exceeding the limit, and inlining would a 3-line fix in the code which could happen right now. All that's needed is to stop the inlined entity spawning, all the rest is already in dmap.

 

How is it a step backward ? You'd really only want to use inlining for stuff you never need to reference to, like rocks, grass, foliage, architecture. Even though the surfaces aren't merged in .proc, they will be batched in runtime, just like the models are. Dmap just saves one step by pre-merging surfaces in .proc.

 

Now that I think about it more, there is one inconvenience - gotta recompile map every time you move inlined entities. Might as well just not inline them until final build :/

 

The way we fixed inlining was not preventing entity from spawning. The fix was removing inlined entity after spawning, and retaining collision model instead of discarding it.

[SteveL]

How is it a step backward ?

Maybe it's not, for the reasons you point out. I'm used to thinking in terms of wanting to reduce reliance on memory (stored state), and the vertex data we send to the gpu, but the latter only really applies to animated meshes where we send skinned vertex data every frame. Static surface vertices get sent just once. And a few dozen megabytes is nothing by modern standards, whether you're talking RAM or downloads.

The way we fixed inlining was not preventing entity from spawning. The fix was removing inlined entity after spawning, and retaining collision model instead of discarding it.

That sounds like a better approach, but we'd have to do some jiggery pokery to make sure the map didn't exceed our limit before they got removed and we'd probably run into a problem with the even tighter clipmodel limit. It's a good way to handle it but it's not something we'd be able to implement and test for 2.03 so we might be stuck with using inline in combination with clip brushes if we do "fix" it in 2.03.

[motorsep]

Hmm. Makes sense to not let them spawn on map load. I wonder if entities can be removed before entity count kicks in.

[SteveL]

No, unfortunately. If they spawn, they are registered and sit in the fixed-size array until removed. You'd have to change a lot of code to alter that, for example which physics object owns the clipmodel (though you already did that bit by the sound of it).

 

You could try loading just the collision models without registering the entity. Or you could write the CMs for the inlined entities into the .cm file as part of the world cm. but that would mean writing new code, though you could harvest from nearby code. But you are still likely to run into a clipmodel limit.

 

Requiring mappers to put clip around inlined entities would have performance benefits: complex collision models generated from the visible model would be exchanged for very simple ones, and lots of inlined entities (think grass clumps or books on shelves) don't need their own CMs.

[OrbWeaver]

Maybe it's not, for the reasons you point out. I'm used to thinking in terms of wanting to reduce reliance on memory (stored state), and the vertex data we send to the gpu, but the latter only really applies to animated meshes where we send skinned vertex data every frame. Static surface vertices get sent just once. And a few dozen megabytes is nothing by modern standards, whether you're talking RAM or downloads.

 

Available memory may be increasing rapidly, but so are the expectations of mappers and players. If you want 2000 objects today, someone is going to want 3000 tomorrow, and in ten years maybe 50,000 or 100,000. With inlining the memory usage scales linearly with the increasing model count, and — since increasing a number in a script is much easier than manually adding detail to models and maps — this usage is likely to increase much faster than the usage by regular (non-cloned) architecture. Imagine if 90% of the GPU memory was consumed by your redundantly-inlined 100,000 identical grass or tree models. Reducing this redundant data storage seems like a very good idea to me.

 

Now that I think about it more, there is one inconvenience - gotta recompile map every time you move inlined entities. Might as well just not inline them until final build :/

 

Not just an inconvenience, but it completely excludes the possibility of creating the clones dynamically based on user-selectable levels of detail, which is a pretty common technique in modern games.

[motorsep]

OK, I couldn't fathom it elsewhere but Orbweaver's suggestion is akin to my work with the spectrum keyword hack.

 

usurface.cpp

 

// set merge groups if needed, to prevent multiple sides from being
// merged into a single surface in the case of gui shaders, mirrors, and autosprites
if ( s->material->IsDiscrete() ) {
for ( tri = triList ; tri ; tri = tri->next ) {
tri->mergeGroup = (void *)s;

 

 

basically, add another branch which polls your LOD attribute here and that would do the trick.

 

Got some info from Brian Harris. He said that discrete surfaces will never be combined. I wonder if there is some exception for IsDiscrete in the render code too, besides dmap code. If so, then I'd have to naturally add another check into that if() in dmap so that while surfaces are separate in .proc, they still get merged on render time.

[motorsep]

Not just an inconvenience, but it completely excludes the possibility of creating the clones dynamically based on user-selectable levels of detail, which is a pretty common technique in modern games.

 

I would think for indies using idTech 4 / BFG engine, having ability to shove in more entities on the level without increasing upper limit would be a good thing I also doubt that average indie will manage to make scenes with millions of world clutter objects.

 

In a long run proper instancing of static and dynamic (skeletal) models would be a good thing. However when I spoke to one of the renderer programmers I hired to work on shadow mapping, I was told that it would be a major rewrite of the portions of the current renderer. And that still doesn't solve the issue with entity limit, unless we can simply increase it to a million and have no negative effect (someone mentioned that increasing size of that fixed array decreases size of the other two dynamic arrays, which will make the engine run out of dynamic entity slots and crash during the gameplay)

 

And when it comes to major rewrites of idTech 4 / BFG, it makes me think going with UE4 / Unity would be a smarter decision (unless introducing instancing isn't that big of a deal).

[motorsep]

Oook, "discrete" doesn't do anything with .proc - it seems that it only works for rendering.

 

So, no matter what I do (and I bet I do it wrong), dmap gets all the surfaces and saved them as one.

Edited November 26, 2014 by motorsep

[motorsep]

So, modifying how dmap sorts out surfaces and saves them per "area" is how it should be done. I did that manually and it worked like a charm.

 

Did some performance testing last night with LOD. 24k x 24k (or something like that) flat open level with models totaling 13M tris. Only tested with LOD meshes so far and got pretty descent performance (55 fps w/ 12ms on GPU with distance of 1024 to the first high poly LOD; shorter distance results in 60 fps and ~8ms on GPU).

 

Next step is to replace all LOD entities with regular func_statics containing mesh with LOD1 polycount. The goal is to see whether performance is comparable or if LOD approach still fares better. The question is if the engine still batches meshes based on material on runtime, and even if it doesn't (as I was told it merges surfaces on map load time) I gotta see whether LOD is still a better approach than bruteforce rendering of high poly meshes.

[OrbWeaver]

I would think for indies using idTech 4 / BFG engine, having ability to shove in more entities on the level without increasing upper limit would be a good thing I also doubt that average indie will manage to make scenes with millions of world clutter objects.

 

True, inlining is probably fine for the levels of detail required by current mappers, and maybe vast numbers of auto-spawned detail objects aren't relevant for a claustrophobic stealth game like TDM (as they would be in an open-world MMO, for example).

 

Oook, "discrete" doesn't do anything with .proc - it seems that it only works for rendering.

 

Odd, I wonder how that works. Does it merge all the surfaces in the .proc but then reload just the discrete surfaces from the brushes in the original .map file, so it can render them without reference to the merged .proc geometry? Seems a bit of a bizarre way of doing things, but maybe it's done for performance reasons.

[motorsep]

Odd, I wonder how that works. Does it merge all the surfaces in the .proc but then reload just the discrete surfaces from the brushes in the original .map file, so it can render them without reference to the merged .proc geometry? Seems a bit of a bizarre way of doing things, but maybe it's done for performance reasons.

 

Apparently it works for GUI surfaces. I have 2 of func_statics made with GUI surface and regular brush for each, and dmap writes them as separate "models" into .proc file (2 "models", each contains brush surface and gui surface), while the rest is shoved into another "model" (even though there are discrete surfaces there).

 

I am thinking maybe our dmap port for BFG engine is lacking something. I would have to test it with stock idTech 4 dmap to confirm.

[motorsep]

Ok, apparently there is some tricky hardcoded total polycount limit in BFG engine. Instead of going all crazy with 13M tris, I reduced polycount per entity (highest LOD iteration) to ~2900 (since it's just suppose to be rocks and foliage) and now I have no issues. Having LOD is still twice faster than bruteforce rendering, so I assume engine still batches surfaces in runtime based on matching materials.

 

The sad truth that idTech 4 / BFG is definitely not an outdoor engines. I tested my massive outdoor map with LOD entities and issue with corrupted surfaces popped up again. Maybe splitting terrain into smaller enclosed sections, connected with tunnels and having vizportals should do the trick.

[Tels]

I would think for indies using idTech 4 / BFG engine, having ability to shove in more entities on the level without increasing upper limit would be a good thing I also doubt that average indie will manage to make scenes with millions of world clutter objects.

 

Well, I already tried it:

 

http://forums.thedarkmod.com/topic/12107-announcement-seed-system/page__view__findpost__p__243243

 

 

The models are cloned dynamically, then combined into larger models automatically. On LOD changes, the model is recompiled.