Announcement: SEED system

[nbohr1more]

BTW... Does tracker 2593 fix tracker 2297 ?

[Tels]

BTW... Does tracker 2593 fix tracker 2297 ?

 

No, not really. First, the ModelGenerator contains two routines, a simple one to simply duplicate one existing model and a more complicated one to "combine" arbitrary models (aka stages) arbitrarily often.

 

* The first routine needs a bit of love, it has not yet been rewritten as the second one.

* even if it is, it needs to be hooked into SetSkin() so that every time a skin changes the "sidedness" of a model, it clones the model (and omits/adds the nec. surfaces).

* However, then we also need a way to manage these shared models,because we A: don't want them to be done more than once, and B: need to free them later. (and we cannot just let the D3 model manager manage them, as I think it can't be given new models to manage, or queried about existing new ones).

 

This is all quite some work to fix a bug that nobody cared for the past years, anyway

[nbohr1more]

Yeah, I have a feeling that 2297 was opened because of your concerns about the dilemma posed by 2593... Now that you've got past that concern 2297 is just a nifty little augmentation...?

[Tels]

Yeah, I have a feeling that 2297 was opened because of your concerns about the dilemma posed by 2593... Now that you've got past that concern 2297 is just a nifty little augmentation...?

 

No, #2297 was opened because we have had one-sided bush materials, and the models reskinned with two-sided materials remained one-sided. Likewise, after I switched the material to two-sided, you no can no longer make a one-sided bush (because the model gets the two-sided surfaces appended at map load).

 

And it is not an "nifty little augmentation", it is actually quite a lot tricky work as the model manager in D3 is closed source, so you have to work around it. But not before v1.04 and possible not even then There are more important things.

 

As:

 

* now fixed the DuplicateModels() routine in the ModelGenerator to use the same strategy and thus work more robust. That routine is used exclusively to copy render models of func_statics compromised of map geometry, e.g. if you place a brush in DR, and except the SEED to duplicate it. In this case there is no model file, so loading the model later is not going to work, so we save a copy of it.

* also improved error message if an image map could not be loaded (thanx stgatilov)

[nbohr1more]

I wasn't saying it was easy.

 

(Though... you have "raised the bar" so easy is hard to define... )

 

I just wasn't sure of 2297's remaining importance...

 

Things keep looking better and better

 

Thanks for all the updates :wub:

[Tels]

More work done:

 

* updated heath model (26 vs 18 polies, but looks much better when seen from slightly or direct above),and made it nonsolid

* added new image maps:

 

"vegetation" (good if your SEED covers an entire area and you want to make the f.i. grass look not so uniform)

 

 

"patterns" (the same)

 

 

"spots" (for these "plant some grass spots here" situations)

 

 

"two_spots" (for the "I need a table with stuff spawned for two persons", e.g. do not put the game pieces all over the table)

 

 

* stgatilov also did some profiling and found out that the ImageMap library always reloaded the entire PNG file for each entity - which resulted in a massive slowdown. And then fixed it (Amazingly enough, even with loading a PNG file for each entity, the system was not so slow that it was unusable, merely slow

 

 

Today I'll look into the debug asserts triggering, there seem to be some things going wonky

[nbohr1more]

Looking forward to seeing a result shot for the "two spots" image.

 

Has your original swamp map seen any improved performance after the rebuild or will that have to await different models still, etc?

[Tels]

Looking forward to seeing a result shot for the "two spots" image.

 

In editor (the spawnarg "angle" "90" is missing here, but otherwise it is this and nothing more):

 

 

In game:

 

 

The table will be available as prefab. Currently the moveable game pieces stick into the table, but this a bug that hopefully can be fixed even before v1.04

 

Has your original swamp map seen any improved performance after the rebuild or will that have to await different models still, etc?

 

Only looked at it yesterday and there is was too slow - but that was with the PNG bug still present. But it isn't comparable, anway, because I changed some models and stuff around and it still needs more models and so on. Might revisit it after Saturday

[Tels]

Thought that before I add more half-finished features, I'd see if the Restore() might not be complicated to fix. Turned out it was complicated and took way longer than anticipated.

 

But it works no, no longer doesn't just not crash, but correctly restores the entities.

 

Here is a difference picture between before/after save, you can see that apart from a few pixels around the light and the FPS, nothing changed:

 

(Warning 1920x1200 pixel image!)

 

 

So another bug bites the dust.

[7upMan]

Okay, so what exactly do we see (or don't) in this last screenshot?

[Tels]

Okay, so what exactly do we see (or don't) in this last screenshot?

 

It is a screenshot taken before Save(), and from that subtracted a screenshot after Restore(). It shows that everything (positions, textures etc.) were correctly restored, because when you subtract them, you end up with a lot of black. If there was something different between before and after, it would show up as colored pixels.

 

You can see a few where the FPS counter is and around lights, which flicker.

 

It's basically the only way I can "prove" you that Restore() now works

 

(And stgatilov found and fixed another restore bug so thinks are shaping up quite nicely and stable now

[Baddcog]

OK, dining room table (I want the plates and silver ware, especially salad forks in correct spots). lol.

 

Nah, that game piece thing is really cool.

[Tels]

OK, dining room table (I want the plates and silver ware, especially salad forks in correct spots). lol.

 

You will laugh, but this is exactly what is missing: Have the SEED spawn entities in definite offsets from itself. Basically like "def_attach", but where the attached pices *can* be attached, or not. And they are not really attached, but just spawned.

 

For instance think of "plate/plate2/platechina plus fork/knife/spoon plus tankard/othertankard". You want one of each, but each piece should be randomly choosen.

 

Right now you would need three SEED entities just to achive this, or one SEED with three different image maps which dictate the placement. It would be much more cool if you could just say "I want this entity spawned at this offset, with a random angle and a very small random offset added". So the tankard is always sort of top-right from the plate, and the knife/spoon whatever is always to the right. And yet it looks a little bit random.

 

The current way "everything can be everywhere" is a little bit too random for a lot of things.

 

Nah, that game piece thing is really cool.

 

 

Added a wiki page with visual examples for the image maps we have:

 

http://wiki.thedarkmod.com/index.php?title=SEED_-_Image_maps

 

Also a first stab at fixing #2608 - if you set

 

"map" "a,b,c"

 

then the image map on the SEED will be choosen randomly between "a","b" or "c".

 

It is also possible to set "random_angle", then the angle will be choosen randomly between 0 and 360°.

[Tels]

Looked at the issue with the moveables sticking halfway into the floor, and there were two bugs:

 

* template_XYZ spawnargs were not properly relayed ending up as "seed__XYZ" instead of "seed_XYZ". This made manually added "target_offset"'s not work.

* moveables have their origin at the center to make physics work, but the SEED assumes he origin is at the bottom. Correcting all moveables manually is a lot of error-prone, not-fun-only-you-like-to-poke-yourself-in-the-eye work.

 

So I fixed the first bug and automatically calculate the offsets for moveables if it isn't set.

 

Thus you can now spawn moveables with the only two problems being:

 

* moveables sometimes stick into each other, as the collision between things created by a SEED is not proper yet

* they can float in the air because the physics is not activated

 

 

And last but not least but a bit unrelated: The code that creates the entities does first create X entities from template X, then X2 from template Y and so on. However, if given "3+2+2" entities and an overall limit of 5, it should not always create "3+" entities, but other combinations like "1+2+2" or even "1+1+1" etc. E.g. right now it prefers one class, creates all max possible entities from it, and then might run out of entities to create. So the amount of entities from each class should be balanced more.

 

However, even with all this, here are two screenshots of a testlevel with 3 tables and 3 SEEDs spawning random bottles, random cards and random game pieces. The shots were taken by simply loading the map twice in a row:

 

 

Here is how it looks in the editor:

 

[Aprilsister]

Seedboxes...

 

So cool. But now we have another conflict in tech-speak!

[Tels]

Seedboxes...

 

So cool. But now we have another conflict in tech-speak!

 

Never heard of a seedbox before... (oh, duh, rent a server, put a torrent tracker and seeder on it. And that's news?)

 

Anyway, I fixed now 2579. Sounds tricky, but turned out not that hard to fix

 

Basically, the normal SEED code notes what entities it must spawn where, and then later spawns them from a "class template". This works fine if all your entities have only a classname, model, skin and rotation, like static models, but as soon as they need additional spawnargs (that are not in the entity def, but set by the mapper in DR), then it breaks down, because these spawnargs were missing.

 

A classical example is a smoke emitter, which doesn't have a model, instead the particle it uses is stored in the "smoke" spawnarg. Not having this set when you spawn the entity results just in.. well, nothing

 

So I now parse the map file (when nec.) and note all the spawnargs that were set in DR minus the ones we don't need (like "classname") or can't use (like "targetX"). When we then spawn the entity, these are cloned in.

 

Here is a silly example, showing 163 randomly placed smoke emitters:

 

 

Now spawning random smoke might not sound useful, but apart from that this fixes a pile of other cases (lights with color, doors with rotation, entities with sounds defined etc), particle emitters can also be almost anything. Think leaves on trees, or dust falling from the ceiling, or bubbles bubbling up under-water, or steam from rock cracks, or indeed, smoke from many chimneys on the city rooftop mission

[nbohr1more]

Can the particle emitters be adjusted via LOD so that closer ones emit more particles (etc)?

[Tels]

Can the particle emitters be adjusted via LOD so that closer ones emit more particles (etc)?

 

No, not really. LOD needs still to be build into the emitters. (I had the idea that at least the emitter switches off if it is a certain distance away. Less particles might be tricky as the emitter is usually all-or-nothing because it just "plays" its particle). I think rain works like this, but maybe not smoke ("func_smoke" is different from "func_emitter").

 

I am not sure, but I do think that emitters do not amit particles if they are outside the PVS anyway. Plus all particles and all smokes are rendered in one go. So it is not that a big deal.

 

However, think "huge valley with a path with torches leading through it". Then you'd see all torches and all particles, so it might be something to test. Hm, should track this so I don't forget it.

[nbohr1more]

I was thinking of the Particle leaves on the trees. Not just the draw calls but the amount of Alpha-Sort on the GPU... Of course, there are probably other nifty things that distance based particle effects can do but I also never looked into whether this is already something that can be controlled via an FX definition...

[Tels]

Found time today to revisit my pond map. After fixing a few entity problems and changing their distribution, here are the preliminary results - the far landscape is not finished, and need more LOD models and so on. So this is far from final, and very far from a playable map. But it serves as a performance benchmark:

 

Object detail at "Lowest":

 

 

8212 models combined into 301 entities, 506700 tris, 677 draws, ~25 FPS.

 

Object detail at "Normal":

 

 

11466 models combined into 314 entities, 1180981 tris, 1846 draws, ~19 FPS.

 

Object detail at "Very high":

 

 

16608 models combined into 323 entities, 2299223 tris, 2131 draws, ~15 FPS.

 

 

There are quite some stutters (in all quality levels) when models are updated, which means the ModelGenerator needs to be speed up.

Here is the timer info for the "Very High" case:

 

ModelGenerator: combines 1840, total time 5775.05 ms (for each 3.14 ms), copy data 723.65 ms (for each 0.39 ms), finish surfaces 4835.43 ms (for each 2.63 ms)
ModelGenerator: dup verts total time 557.11 ms (for each 0.30 ms), dup indexes 84.67 ms (for each 0.05 ms)

 

That means the code spent 5.7 seconds alone just updating models, and the bulk (4.8s) for FinishSurface, while only 0.7s for my code that copies the data. And from these 0.7, we took 0.56s for the copy of the vertices, and 0.084s for the indexes.

 

With 3.14 ms per update, one couldn't update more than one or two models per frame (which is for 60 FPS, only 16.6ms long). Without FinishSurfaces() it would be 8x faster, so you have a lot more time.

 

So, eliminating FinishSurfaces() is the highest priority, then the vertex copy, then the index data.

 

 

In addition, the combining of entities could possible be improved (less drawcalls). And last but not least lower-poly models for the far-away places to fill the screen need to be done. Also the bush model we have has a lot of polies, but no LOD model whatsoever.

 

So quite a bit of work remains

[nbohr1more]

One thing that would be cool to see is a map of just one entity type that has LOD levels and compare:

 

1) All high detail models (No LOD, No SEED) at a barely playable density (15fps ?)

2) Just LOD

3) Just SEED

4) LOD + SEED

 

 

 

Also, to clarify does the "watch_brethren" feature let you specify how close things are combined or does it always combine all in the selected region except for culled items?

 

Is there a way to make the "combine" feature combine smaller numbers of high-poly items that are close to the player then "combine" big swaths of low-poly models in the distance? (Does it do that automatically currently? )

[New Horizon]

Nice work.

[Tels]

For tonight I had to fix two bugs, and both came to light (no pun in testing NHAT 3/3 - the forest mission. So I thought this mission would make a nice usage case for SEED optimization and answer the question "And what does it give us?"

 

So basically what I did was the following:

 

* take a copy of forest.map and move the player into the wood, looking across the lake

* copy this map again

* add a few SEED entities, and let each of them "watch" over some entities. One over the cattails, one over the flower patch, another over the other flowers, and one big SEED over some trees, the leavy plants and so on.

 

Basically the same idea as with Johannes' map, except that instead of 70 ivy models, this now combined f.i. 97 flower models, several trees and so on.

 

Here are some examples:

 

SEED atdm_seed_white_flowers_1: Combined 97 entities into 1 entities, took 0 ms.
SEED atdm_seed_big_plants: Combined 873 entities into 49 entities, took 3 ms.
SEED atdm_seed_ivy: Combined 55 entities into 1 entities, took 0 ms.

 

I also looked into other improvements, but Mortem and GoldChoco did a hell of a job here, everything is optimized already! The only exception I found was the white gargoyles and columns in the city, they did not have hide_distance on them. This means they are visible from the forest, even tho they are either hidden behind the wall, the trees, or you are way to far away to see them, anyway. However, for my testscene below they did not make much of a difference because they are out-of-the-screen, anyway. But if the map is ever optimized again, this would be an easy thing to do.

 

Aynway, apart from reducing the entity count during map run (e.g. improved render performance), the SEED combiner also reduces the size of the savegame (because a lot less entities exist):

 

6739962 2011-02-12 22:33 seed.save
8281755 2011-02-12 22:35 nonseed.save

 

Note that the savegame sizes above are done with v1.04 and thus quite reduced from what you'd see with v1.03, because all the needless "editor_" spawnargs are no longer existing and thus not saved (Thanx stgatilov! . However, with v1.03 the difference would be even more drastic, on the order of 70 vs 90 Mbyte or so

 

So what about rendering performance. Here is the testscene:

 

 

The scene on the left is actually done with the "Object detail" set to "Lower". This reduces the view distance, and improves performance. That is one of the v1.04 features. For the actual comparisation I of course used "Normal",as the r_tris scene on the right shows. But I forgot to take another screenshot

 

First a word about FPS. Normally, on my system, they fluctate wildly. Not so with this mission, they are very stable, with one exception. In this scene, the FPS stay for 10..15 seconds at the same level, then suddenly they raise for about 3 seconds much higher, then fall back. I report both numbers here, as I do no know why. The lower number is the one relevant for playing the game, of course.

 

Stats from r_Primitives and FPS:

 

Original:

 

3623 drawcalls, 584000 tris, 6 .. 18 FPS (12..25 FPS after killMonsters)

 

With SEED:

 

1270 drawcalls, 541000 tris, 8-9 .. 22 FPS (17..32 FPS after killMonsters)

 

As you can see, SEED improves the rendering performance, without taking away from the visuals. And if the performance is still to low, you could of course use the new menu setting to lower the quality. However, at least on my system biggest performance hit is the AI.

 

What does not work yet?

 

The flowers/reed use func_movers to "sway" in the wind. After the modelgenerator creates a combined model, this no longer works. To make this work one would need to dynamically update the combined model all the time, and I think this would be too slow - or at least it is atm because of a bugs: "noshadows" "1" on a targetted entity is not yet propagated properly to the combiner, so it uses FinishSurfaces() needlessly. That costs a lot of time.

 

One might think that one could do some tricks with combining the 97 flowers into 5 different entities, then make each of them "sway" on its own, but still, this would look odd.

 

What could also be done:

 

The trees could use LOD models. Likewise, the reed/cattails could use LOD models (that did not exist when NHAT 3 was made). Also, the SEED system might provide the toolspopulating the forest with a lot more models, like grass, mushrooms and so on. Right now the performance of the map is borderline, adding more models would only make it unplayable. The SEED system can add these models at very low costs, so the grass is only ever visible where the player is.

 

Work continues

[Aprilsister]

As you can see, SEED improves the rendering performance, without taking away from the visuals....

 

And this is above and beyond the genius of the underlying functionality to begin with.

 

A huge success.

[Sotha]

Nice to see performance gains with the new SEED system.

 

It just occured to me and I'd like to ask the following:

 

Could SEED be somehow used to improve performance for some particle effects like rain.

 

For a scene with two light rain patches (with rain splash patches) vs the same scene without rain r_showprimitives 1 gives the following:

Rain:

views: 7, draws: 85, tris: 9195

Without rain:

views: 7, draws: 57, tris 2578

 

As one can see, the rain is very expensive thing. It would be cool of the performance drag could be alleviated with some magic. Could there be a way?