Announcement: SEED system

[nbohr1more]

Since ASE models have their vertex normals visible in the text file would it help debug this issue by comparing the native ASE vertex normal values to wherever you are acquiring them for your construction calls... or are the data formats way too different (too much stuff happens in-between)?

 

[Tels]

Leaving the wrong-normals-or-lighting bug alone for now I have taken care of issue #2599 - make spawnargs "spawn_xyz" more versatile.

 

To recap, there are two methods of specifing what entities the SEED should generate:

 

* add entities in DR, then link them from the SEED

* specify them via spawnargs on the SEED

 

There is also a third way, target an existing entity and tell the SEED to watch over all entities of that class. This is what we talked about in the last few posts, but is not what we talk about here now

 

Back to generating random entities in random places: The first is the normal way mapper might use, because it provides you with visual template. However, this technique has a few drawbacks:

 

* if you use the same template many times you end up with a lot of superflous template entities that are first spawned and all their assets loaded and cached!) and then throw away at map start.

* even worse, if you f.i. use 20 templates (to get a lot of diversity) and then specify a max_entities = 5 (so to get never more than 5) you can end up loading entities that are never used in the map elswhere, wasting a lot of resources to load their assets.

* and last but not least it is not possible to setup pre-made entities with templates so the mapper can use them right-away (except maybe with prefabs, but that is somewhat ugly)

 

So I added last Summer the ability to specify spawnargs, but that had a few problems:

 

* spawn_class was easily confused with "spawnclass"

* it only worked for entity classes, but not for plain static models

* it did only allow to specify the class, count and skin, but no other see spawnargs

 

I fixed all that, renamed the spawnargs to "template_...", made it work with model names, too, and relay all seed spawnargs to them.

 

Documented all that, too: http://wiki.thedarkmod.com/index.php?title=SEED_-_Templates_via_Spawnargs

 

The only thing left is optimizing this routine as currently it actually does spawn a temp. entity,because the routine which is supposed to create the template needs an entitDef, not just an idDict.

 

But since this will not change the working, just make it faster, this will probably have to wait for v1.05.

 

The following spawnargs thus lead to two mvoeable and one combined static entity with 50 models:

 

"combine" "1"
"max_entities" "50"
"template_count" "200"
"template1_count" "2"
"template" "models/darkmod/decorative/games/cards_stack02.lwo"
"template1" "atdm:moveable_cup_marble_rough"
"template_map" "splat"

 

 

While I was at it, I also fixed one small bug related to the max_entity count of classes (wich was ignore by accident).

 

Anyway,back to work for me now

[Tels]

Have now fixed our existing random entities (well, the only we had) to use the new "template" spawnargs. Also added a few more (random fruit, random grass in normal, sparse and dense variants, random rubble, random game pieces etc. to come soon) and started a wiki page to document them:

 

http://wiki.thedarkmod.com/index.php?title=SEED_-_Premade_entities

 

While doing so, I noticed and fixed a bug (crash due to removing the SEED while multi-statics where present). Also noticed another two things:

 

* the entity count is used literally, and it is not possible to leave it at 0 (and get dynamic entity counts). This means the grass patch has always the same number of entities regardless of menu setting. Oops, need to fix that. This is fine if you want to place X entities on a table, or X bottles in a shelf, but not for X00 plant entities where we need variety on the numbers, too.

* with the high amount of grass entities and the SEED always using the same orientation and image map, all the grass patches look alike, even tho they are clearly different. I guess this means we need "random image map selection", a few more random-looking image maps as well maybe a random "angle" spawnarg for the SEED. Otherwise placing 5 of these grass entities in a map will result in 5 identical-looking grass-patches.

 

Heh, and I though't I'd run out of things to do by tomorrow

[Tels]

Got a bit of work on the random generator entites done, but it is clear that they need a couple of more features before they are useful like:

 

* new image maps

* random image maps

* random angle on the SEED

* automatic offset calculation for moveables (pain in the bottom to enter them all manual)

* collision between entities for moveables (or they stick into each other)

 

and so on. *sigh*

 

Anyway, good news is I now know why the super-bright ivy happens.

 

The reason is that the ivy surface is twosided, and my modelgenerator doubles the texture twice. It came to light (no pun when I looked at r_showLightCount (given a hint by stgatilov that the ivy surface is rendered twice, already r_logfile proved to be invaluable!) and then disabling things in the material shader until it looked right.

 

Here are the stats with not-twosided:

 

Allocating for surface 0: textures/darkmod/decals/vegetation/ivy_mixed_pieces
numVerts 864 numIndexes 2592
megamodel
Static model.
bounds: (-201.483963 -55.809967 -38.887402) to (132.844864 113.125008 332.000397)
   verts  tris material
0:   864   864 textures/darkmod/decals/vegetation/ivy_mixed_pieces

 

 

and here again with two-sided:

 

Allocating for surface 0: textures/darkmod/decals/vegetation/ivy_mixed_pieces
numVerts 1728 numIndexes 5184
megamodel
Static model.
bounds: (-201.483963 -55.809967 -38.887402) to (132.844864 113.125008 332.000397)
   verts  tris material
0:  1728  1728 textures/darkmod/decals/vegetation/ivy_mixed_pieces
1:  1728  1728 textures/darkmod/decals/vegetation/ivy_mixed_pieces

 

Note that 1728 / 2 = 864.

 

So it seems that my Modelgenerator already sees a duplicated vert data, and then when feeding this to FinishSurfaces() everyting gets duplicated again.

 

Uhoh, I have an idea: when the surfaces are:

 

A
B

 

and then doom3 duplicates them, you end up with:

 

A
B
A1
B1

 

but the number of base surfaces is 2. That lead me to believe you can just skip A1 and B1 and be done. That works, but if you have only one, it might we well and up with

 

A1

 

where A1 simply is a set of duplicated vertices. That means I need to detect the second case somehow and fix it.

Oh well...

[rebb]

Cool, so maybe the normals were never rotated the wrong way .

Can you detect if there is a twosided material on a surface and switch according to that ?

[Tels]

Cool, so maybe the normals were never rotated the wrong way .

 

Yes,that's what I think. And stgatilov looks at scaling, so even this case should be correct then.

 

Can you detect if there is a twosided material on a surface and switch according to that ?

 

Actually, I already have written, tested and so on code for that, but the thing is, I disabled the code because there seemed no point in it when the duplication of surface data just added another surface. However, this case is different, the surface data is just duplicated on the same surface.

 

Maybe D3 implements a trick that when you only have one surface, it duplicates the data, but if you have more than one, it duplicates the surfaces?

 

I am sure this can be found out by making testmodels with different surface combinations, and I know that the ASE exporter for DR is just the right tool

 

Edit: Actually, after thinking about it can't be that simple. When the ivy model is loaded, the surface is duplicated by appending the data to surface 0. When I hand the FinishSurfaces() one surface, the data is duplicated by creating a new surface. So there must be somethign else going on. I have an idea how I can test which case is which, but I hope this can be done stable.

[Baddcog]

Well, I doubt I can solve this for you but it seems like this.

 

Doom sees a single surface with twoSided.

 

It duplicates that, so now it's 2 surfaces with twoSided (but it doesn't use the twoSided now, it's done with that during load).

 

Now Seed combines these models and recreates them as 1 model, so it reloads the material, it sees 2 surfaces with twoSided so it ends up making it 4 surfaces.

 

You need to make it ignore the twoSided property because Doom already took care of that.

 

or, we need to make the models have 2 sides to begin with and get rid of the twoSided materials. (which probably should've been done initially, but for a long time I was under the impression that twoSided materials [this engine and others] weren't doubling tris).

Also bad because of the typical issues:

1-lots of models to fix

2-lots of materials to fix

3- a lot of those materials are probably used on patches in existing missions.

 

or we make copies of materials to apply to models...

 

I think easier to make Seed ignore... at least for everyone except Tels

[nbohr1more]

Oh, I don't know...

 

It looks like Tels is doing everything and the kitchen sink for the mapper.

 

From boiling down spawnargs to pre-packaged entities to creating an image map utility.

 

Pretty soon he'll have a microphone input where you'll say "Make a bunch of shrubs next to the big gray house" and SEED will just say "Acknowledged" and create them (while blinking it's cold red LED at you HAL 9000 style).

 

 

I think he'll probably even cook-up stuff to make Baddcog's job easier at some point?

 

( Maybe the idea of a 'Model Generator" is intimidating to a modeler? )

[Serpentine]

I am sure this can be found out by making testmodels with different surface combinations, and I know that the ASE exporter for DR is just the right tool

 

Just be careful, the DR exporter actually produces ASE's which do not conform to the rather loose conversions for ASE, tho doom3 does seem to deal with them, you might run into some trouble that you shouldn't with correctly formatted models.

 

I'm not advocating anything, just "trust but verify" before you chase red herrings

[Tels]

Well, I doubt I can solve this for you but it seems like this.

 

Doom sees a single surface with twoSided.

 

It duplicates that, so now it's 2 surfaces with twoSided (but it doesn't use the twoSided now, it's done with that during load).

 

Now Seed combines these models and recreates them as 1 model, so it reloads the material, it sees 2 surfaces with twoSided so it ends up making it 4 surfaces.

 

You need to make it ignore the twoSided property because Doom already took care of that.

 

Well, yes and no You are almost right, except it is more complicated:

 

The model generator takes an existing model (with the two-sided surfaces already doubled at load time, this is what the engine does), and then can either duplicate it, or add other models to it.

 

There is, however, a snag. We cannot just modify the data, and the end of the routine, we need sometimes to call finishSurfaces(). This is an internal routine that calculates shadow hulls. You only need to call finishSurfaces() if you have light interactions on your generated data.

 

So we need to call it when:

 

* there is at least one shadow casting surface

* or we have twosided shaders but only one copy of the data and need it duplicated

 

However, since finishSurfaces() *always* duplicates twosided surfaces - and one can only prevent this by using a one-sided material shader instead, but the snag is, we ain't got not all of the twosided shaders as a one-sided varient, so we cannot just switch the shader fooling the routine into "oh I don't need to duplicated".

 

So the concept becomes actually this:

 

* if we have one single shadow casting surface, then "dedouble all data", then call finishSurfaces() at the end

* if we have one twosided surface, then "dedouble all data", then call finishSurfaces() at the end

* if we have no shadow casting surfaces, but only onesided surfaces, then just leave the data as it is, and do not call finishSurfaces()

 

The reason why we avoid calling finishSurfaces() like the plague is that it always takes approx. 10 times as long as my code that copies the data together...

 

The entire thing did however not work in the specific case, beause instead of two surfaces I ended up with only one and my code not recognizing this case (nor could it deal with it as "dedouble the data" always consists only of "leave of a specific surface at the end").

 

So the easy solution is to detcet the case, and correct for it.

 

The more complicated but faster solution would go like this:

 

* if we have one single shadow casting surface, then "dedouble all data", then call finishSurfaces() at the end

* if we have no shadow casting surfaces, but only one- or twosided surfaces, then just leave the data as it is, and do not call finishSurfaces()

* if we have no shadow casting surfaces, but only onesided surfaces, then just leave the data as it is, and do not call finishSurfaces()

 

Basically we would speed up the second case. However, the code still needs to:

 

* A: deal with both cases (e.g. dedouble, or not)

* B: detect both cases reliable (e.g. in one case at least it doesn't)

* C: know how to dedouble a two-sided surface with the data in th surface doubled

 

Anyway, let's see what I can do about it and whats need to be postponed to v1.05.

 

I think easier to make Seed ignore... at least for everyone except Tels

 

It already does but we found a case where it goes wrong so I need to fix it, anyway

[stgatilov]

However, since finishSurfaces() *always* duplicates twosided surfaces

I regret that TDM's engine is both cross-platform and closed source. No chance to hack it .

[stgatilov]

* A: deal with both cases (e.g. dedouble, or not)

* B: detect both cases reliable (e.g. in one case at least it doesn't)

* C: know how to dedouble a two-sided surface with the data in th surface doubled

 

Do I understand you correct that you have a big array of geometry and you need to know whether it is several copies of the same array? And find the original array? Are the copies of arrays shuffled or just two equal arrays concatenated?

 

 

By the way, a funny trick to determine doubled array comes to my mind: XOR everything and determine if the result is zero Far from reliable though.

 

 

[Tels]

Do I understand you correct that you have a big array of geometry and you need to know whether it is several copies of the same array? And find the original array? Are the copies of arrays shuffled or just two equal arrays concatenated?

 

I have no idea yet

 

So far I see two cases:

 

* case 1: surface A and it's flipped copy A1 (I can detect this case because index of A is < baseNumSurfaces, and index of A1 > baseNumSurfaces

* case 2: surface A and it contains the data twice. Might be detectable because there is either only 1 surface at all, or there is no other surface with the same shader of A in the surface list. However, I have not yet looked how the data is duplicated, my guess is with the normal/tangents flipped, and then the indices get duplicated with ABC => CBA (vertix indexes) swapped.

 

I was wrong, the second case never happens, instead we find:

 

models/darkmod/nature/ivy_part02.lwo
Static model.
bounds: (-20.292803 -7.486900 -38.887402) to (20.292803 4.125100 0.000000)
   verts  tris material
0:    12    12 textures/darkmod/decals/vegetation/ivy_mixed_pieces
1:    12    12 textures/darkmod/decals/vegetation/ivy_mixed_pieces

 

Basically, the number of base surfaces is two, and each surface holds one copy, and one of them is flipped.

 

The difference of this case to the ones I already handle is that the duplicated surface is inside the base surfaces number, and not an extra surface at the end. Arg. That changes a lot of assumptions I made...

 

(about numBaseSurfaces - each model has a list of surfaces and that list consists of two types of surfaces, first the base surfaces (presumable what was in the model file) and additional surfaces, like decals (that get tacked at the end and can be removed). Also, backsided surfaces get sometimes tacked at the end - except, as we found out, in the special case 2 above.

 

By the way, a funny trick to determine doubled array comes to my mind: XOR everything and determine if the result is zero Far from reliable though.

 

I think if we check that:

 

* number of vertices is even AND

* vertex[0] is a flipped version of vertex[ number / 2 ] AND

* index 0,1,2 are the same as index[ number / 2 + 2], index[ number / 2 + 1], index[ number / 2 + 0]

 

However, before we even do this, the complicated thing is that:

 

The model generator gets a list of LOD models (aka just a list of base models). This might have shadow casting surfaces, or not. Or twosided surfaces, or not.

 

It also gets a list of offsets, where it should put each model, which one from the list of LOD models and with or without shadow.

 

It then combines everything into a big model. However, we do not want all surfaces to be duplicated (e.g. if two offsets use two models with the same surface, the data should go into one combined surface).

 

That makes it nec. to make it at least three passes:

 

* run through all used models and note if: they are shadowcasting, they have actually shadow casting surfaces, and which surfaces we can skip (pure shadow caster surfaces must be eliminated when the model has noshadows, while shadow+diffuse must survive because we need the diffuse stage). Also note that if any of the surfaces casts a shadow, we need to call FinishSurfaces() to calculate the shadow hull.

* run through all the actually used models and collect which surface of their surface maps to which target surface of the combined model. We need to do this, because we need to know beforehand how many vertices/indices we need before we allocate their space.

* And finally, run through all offsets, run through all models, then run throuhg all their surfaces, and copy the data. or not.

 

That thing is quite complicated and my head spins just from the description of it How I wish we had instancing and could the graphic card let handle this..

 

Edit: Started this:

 

http://wiki.thedarkmod.com/index.php?title=FinishSurfaces

 

Edit: Corrected assumption and descriptions.

[rebb]

However, since finishSurfaces() *always* duplicates twosided surfaces - and one can only prevent this by using a one-sided material shader instead, but the snag is, we ain't got not all of the twosided shaders as a one-sided varient, so we cannot just switch the shader fooling the routine into "oh I don't need to duplicated".

 

Could setting the idMaterial::shouldCreateBackSides of the currently being processed surfaces to "false" work ?

At least until FinishSurfaces() is done ?

[Tels]

Could setting the idMaterial::shouldCreateBackSides of the currently being processed surfaces to "false" work ?

At least until FinishSurfaces() is done ?

 

Unfortunately the the material is returned as "const idMaterial *shader", so I cannot modify it. But nice idea

 

Anyway, I am currently in the midst of rewriting the entire beast...

[Tels]

It's done! Interestingly enough, I think Johannes proved accidentily that programmers are not good testers and relying solely on the programmer to test and document his work is bound to fail

 

Apart from the model generator there were three other bugs, 2605 (the first watched entity always lost his rotation), watching only a single entity made that entity invisible, and image maps need to be 1.0f in case we don't use one...

 

Anyway, these are all now fixed. And I completely rewrote the entire strategy of the model generator. It now works in 5 stages, accounts all two-sided and pure-shadow-casting surfaces, correctly identifies the surfaces used for ivy, and the lily plant (my other hard testcase), and also avoids calling FinishSurfaces() much more often:

 

1. In case a shadow is needed (and actually one of the stages can cast a shadow!), it removes twosided faces and calls FinishSurfaces().
   
2. in case no shadow is wanted or can be cast, it keeps two-sided surfaces and just copies everything, then does not call FinishSurfaces(). This is *way* faster (about a factor 8..12).
   

 

This means instead of 0.38 ms (copy data twice) + 31ms (FinishSurfaces()) it now needs only 0.28 ms (copy data only once, plus some overhead for figuring things out) + 0 ms. So the entire combined model is build in about 0.3 milliseconds

 

Here are screenshots, as usual, ignore the FPS as they are bogus:

 

 

 

As you can see, the screenshots show it looks identical, except there are about nearly 70 less entities I'm totally exhausted now... and haven't had any chance to look at the bugs I actually wanted to

 

Btw, I would like to thank stgatilov for fixing the issue with the scaling, rebb and stgatilov for the brainstorming discussions and last but not least Johannes for the beautifull test map

[Johannes Burock]

Ok Tels, you are my personal hero^^

 

When and how can i use it...???

[nbohr1more]

WAHOO!!! :D

 

(Somebody has to thank Tels too ...and God... ...)

Edited February 2, 2011 by nbohr1more

[Aprilsister]

I'll thank Tels...

 

Thanks, Tels (and all those you mentioned) for what will surely be a long-living powerful enhancement to FMAs, and by extension the FMs, and by extension my enjoyment! And it's been fun just watching the brainstorming too.

 

...but as for this "God" character I don't recall him/her participating at all and so to thank him/her would be to slight the actual producers.

Edited February 2, 2011 by Aprilsister

[Springheel]

As you can see, the screenshots show it looks identical, except there are about nearly 70 less entities

 

So how much of an FPS gain is that?

[nbohr1more]

Rebb needs to record a timedemo of a SEED map verses a non-SEED map to get the statistical variation...

 

Or a lucky low-ender needs to get a crack at this

[Tels]

So how much of an FPS gain is that?

 

It varies way too much on my system, but as soon as someone compiled a new DLL for windows, people can try it for themselves.

 

maps/test/seed_combine_models_2.map and maps/test/seed_combine_model.map are the two variants.

 

At least the drawcalls are reduced quite a bit, even tho we got a few more tris - not sure where they exactly come from, tho. Probably from out-of-screen-but-still-rendered parts of the entity, which were before culled. But let me assure you 2K more tris, but 350 less drawcalls will still make it faster.

 

drawcalls    tris  shadow
-----------------------------
Before:
     553   36244    9760

After:
     198   38348    9760

[Baddcog]

Awesome Tels.

 

I bet the FPS also varies greatly system to system due to the cpu usage. Obviously Johannes has a great system and will probably see little effect, where as NH who is on an older system (if I remember correctly) will probably see a substantial effect.

[Tels]

Awesome Tels.

 

 

I bet the FPS also varies greatly system to system due to the cpu usage. Obviously Johannes has a great system and will probably see little effect, where as NH who is on an older system (if I remember correctly) will probably see a substantial effect.

 

Just for the record, the effect of "watching over entities" here means that the combined entity is static, e.g. it is never rebuild. That is because we watch over func_statics and they have no LOD spawnargs at all. So the system just sees that the entity is always visible, and thus never changes.

 

So it builds it once at map start, removes all the "old" pieces (and then could remove itself, which does not work yet due to some data dependecies I need to resolve). So you end up exactly as if you had build all this from little patches in DR and combined the manually into one giant func_static. Except it is automatic and works with models, too.

 

One of these days I will hopefully find time to do some explaining graphics and put them on the wiki..

[stgatilov]

Wow, the code has become much more readable now! So the rewriting the ModelGenerator is a very good change from all sides.

 

Regarding scaling it is not complete yet. Only entities without clipmodels can be scaled properly - we still don't know how to put scaled clipmodels into the engine.

 

By the way, how often would SEED use non-shadowcasting objects? Seems that shadowcasting objects would take a lot of preprocessing time anyway. Right now I'm trying to see what exactly is so slow in FinishSurface. I still have a hope that it could be ignored if source was available