Announcement: SEED system

[nbohr1more]

I will be presumptuous (as always) and conjecture an answer:

 

The SEED system is very much like a very large scale particle system. In as much as all the particles are grouped into a small number of draw calls.

 

The only way to improve the performance of particles with the SDK would be to use LOD so that farther particles emitters produce less particles or simpler particles.

 

The dream solution would be to use Instancing.

 

This Demo has various known Instancing methods compared to "one draw per object"... There may be a way to implement at least one of them but I am not sure where Doom 3's native particle management compares to any of them:

 

http://www.humus.name/index.php?page=3D&ID=52

[Tels]

Nice to see performance gains with the new SEED system.

 

It just occured to me and I'd like to ask the following:

 

Could SEED be somehow used to improve performance for some particle effects like rain.

 

For a scene with two light rain patches (with rain splash patches) vs the same scene without rain r_showprimitives 1 gives the following:

Rain:

views: 7, draws: 85, tris: 9195

Without rain:

views: 7, draws: 57, tris 2578

 

As one can see, the rain is very expensive thing. It would be cool of the performance drag could be alleviated with some magic. Could there be a way?

 

Funny that you should ask, I already thought about "combining" particle emitters into "multiemitters". However, I am not sure what the performance impact really would be.

 

First: As far as I know, all particles in a scene are rendered in one batch, anyway. So you would not save any rendering time. The only thing you would save are multiple entities vs. only one entity, which is really a tiny amount of CPU and only a few bits of memory.

 

The idea I had about "multiemitters" would basically only make sense in scenarios where you have hundreds of emitters. Think "100 trees with particle leaves" or "100 smoking debris heaps on a battle field" or "100 bubble emitters in a voclano lake". In these casses having 100 entities doing the same work would waste CPU time and memory, count against the entity limit and increase save/load times and space and also take time to spawn the entities.

 

Second: All the performance drain comes from the fact that all particles are created on the CPU (and not even in a sep. thread which would at least give the second CPU core something todo). I am not sure, but I think the particle engine isn't open source. Even if it is, with the rest of the renderer closed, it is not yet possible to push this to the GPU.

 

The real performance saver would be to generate the particles right on the GPU, render and then throw them away. But we can't do this yet.

 

However, all hope is not lost for the rain, because I think our current "rain particles" are the wrong way to do it. The "rain splashes" need to stay (they need to be 3D), however, the actual falling rain can be just anoverlay on the screen, that doesn't need to be 3D and could be just rendered to a texture. Unfortunately, I don't know much about shaders, but I think it must be possible to write a shader that just draws the rain on the screen as overlay by rendering only a quad. (Whether the engine allows such shader is another thing, tho).

 

@nbohr1more: You are right that the only way to increase performance would be to reduce the particle numbers. However, it is unclear how much that would actually save, plus, there is the problem that the code would first need to find out which particles it could leave of. You either need two/more sep. particle models (e.g. like LOD, you just switch to a different particle, for torches f.i. it has only 10 (more dense) instead of 50 (lighter) smoke particles.

 

The other way would to programatically reduce the number of spawned particles, but that might just make it look wrong.

 

One "cheap" way would be, of course, for the first step to add a "hide_distance": I am not sure if particle emitters can even be switched of with distance yet. Edit: Of course they can, rain patches are emitters. But I tried it with func_smoke, and they don't support hide_distance yet.

[Tels]

Here is a bit of insight from smokeparticles.h:

 

       Smoke systems are for particles that are emitted off of things that are
       constantly changing position and orientation, like muzzle smoke coming
       from a bone on a weapon, blood spurting from a wound, or particles
       trailing from a monster limb.

       The smoke particles are always evaluated and rendered each tic, so there
       is a performance cost with using them for continuous effects. The general
       particle systems are completely parametric, and have no performance
       overhead when not in view.

       All smoke systems share the same shaderparms, so any coloration must be
       done in the particle definition.

       Each particle model has its own shaderparms, which can be used by the
       particle materials.

 

We need a wiki article that collects that info.

 

Anyway, looking at the code from smokeparticles, this shows f.i. that the entire model is rebuild every frame, so this is costly. Particles like rain should be cheaper, but I haven't actually understood how our rain works, according to the wiki it is emitted from a texture...

[nbohr1more]

One "cheap" way would be, of course, for the first step to add a "hide_distance": I am not sure if particle emitters can even be switched of with distance yet.

 

 

Sotha had that trick working in 1.02 but I understand that LOD has become a bit more complex now... It would be a good thing to verify

[Tels]

Uhm, http://www.iddevnet.com/doom3/ gives only 404 for everything? Is it gone?

 

Luckily cached:

 

http://webcache.googleusercontent.com/search?q=cache:guqJJOxVHMYJ:www.iddevnet.com/doom3/materials.php+doom+3+texture+deform&cd=3&hl=en&ct=clnk&source=www.google.com

[Baddcog]

Just figured I'd post this here since the 1.04 thread is for lock down issues.

 

As far as lod distances, player control. I think Tels is right, the player should have the control. The only reason it is implemented (at least the main reason) is to squeeze more performance for those with slower machines.

So if the mapper is the only one that can decide, all maps would be only for high end machines. I would certainly not want models popping in/out.

It's ugly and distracting, but if it helps someone play the game then I guess they can deal with it.

 

I propose that the 'standard' setting is highest possible. That way the player WILL know they adjusted it down for better performance, not a bug or weird issue with a map.

 

These days it's not an uncommon feature, Oblivion had it for foliage and a lot of Thief players have played that.

 

But there's also no reason the mapper shouldn't have some level of control. Not really to force the player, but to optimize. If a model fades at 80 units but the mapper knows it can't be seen past 50 no reason for them not to be able to set it at 51.

[Tels]

Just figured I'd post this here since the 1.04 thread is for lock down issues.

 

As far as lod distances, player control. I think Tels is right, the player should have the control. The only reason it is implemented (at least the main reason) is to squeeze more performance for those with slower machines.

So if the mapper is the only one that can decide, all maps would be only for high end machines. I would certainly not want models popping in/out.

It's ugly and distracting, but if it helps someone play the game then I guess they can deal with it.

 

I propose that the 'standard' setting is highest possible. That way the player WILL know they adjusted it down for better performance, not a bug or weird issue with a map.

 

These days it's not an uncommon feature, Oblivion had it for foliage and a lot of Thief players have played that.

 

But there's also no reason the mapper shouldn't have some level of control. Not really to force the player, but to optimize. If a model fades at 80 units but the mapper knows it can't be seen past 50 no reason for them not to be able to set it at 51.

 

Basically agree with you. Except of where to set "the highest level": The "Normal" setting is supposed to be the balance between performance and visual quality. E.g. at normal, you should NOT see models popping, nor switches of LOD quality, unless you either:

 

* have a really high resolution,

* and have keen eyesight,

* and use the spyglass

* your system can't keep up and the quality is automatically lowered (not implemented yet)

 

If such issues do occur, then the system isn't tuned right.

 

If you really need better quality, (or want even more vegetation), then the "Better", "High" and "Higher" levels exist for this. It's f.i. important for people with a resolution over 3000 pixels, like these eyefinity setups

 

Anyway, the issue that sometimes things "pop" or "vanish" while you look can be solved with technical means - it just needs implemting a few basic rules, I already laid the foundation for this. Except a better code of this in v1.05.

 

But of course, the mapper should be able to override it, because in some cases an automatic system will make the wrong decision.

 

As a side note, a lot of the "pop" situations only arise because our LOD model selection is so tiny. You can see it on the Aphrodite statue, on the "Normal" level the switches are almost invisible (unless you look close and use the spyglass) and the statue still looks good in all situations. It is certainly better than before, where it looked brilliant in all situations, but used a lot of tris just to achive that However, a lot of other models simply lack LOD stages (notable our trees), or have only 1 or 2, or even have non-matching stages (like our pine models...).

 

If I just was as good with programming as with making models...

 

Unfortunately, the LWO vs ASE situation doesn't help, either, because we cannot replace an LWO model with an ASE one :-/

[Sotha]

One "cheap" way would be, of course, for the first step to add a "hide_distance": I am not sure if particle emitters can even be switched of with distance yet. Edit: Of course they can, rain patches are emitters. But I tried it with func_smoke, and they don't support hide_distance yet.

 

hide_distance was broken in v1.03, I understood it will be back in v1.04. I hope it will be.

http://modetwo.net/d...post__p__241764

 

My maps performance was okay when the rain patches autohid when the player was far enough. The map is relying a lot on hide_distance weather optimization.

 

Now each rain drop is an individual set of two tris, right? If one rain particle contained three drops, it would cut down tris count while making the rain still look "dense." Don't know what that would look like, though..I suspect someone would have tested this already..

[Tels]

hide_distance was broken in v1.03, I understood it will be back in v1.04. I hope it will be.

http://modetwo.net/d...post__p__241764

 

Yes, it will be and it is working nice in the testmap

 

My maps performance was okay when the rain patches autohid when the player was far enough. The map is relying a lot on hide_distance weather optimization.

 

That will work again.

 

Now each rain drop is an individual set of two tris, right? If one rain particle contained three drops, it would cut down tris count while making the rain still look "dense." Don't know what that would look like, though..I suspect someone would have tested this already..

 

Yeah, I had similiar ideas. However, this is complicated because the code that actually emits the particles from the surface is closed source, so it is a bit hard to see how it works.

 

As for "doing it otherwise", replicating the exact effect (the entire surface emits particles with a certain density) is difficult, because you would need either special code (which we don't have yet) or some sort of multi-emitters (which we have neither).

 

I think for rain/snow it might be possible to come up with special code (e.g. something that replicates the closed source code) but whether that would be actually faster?

 

As I said, an overlay where the rain is simply painted on and maybe moves very fast downward via texture shift might be a lot faster. (Or not, as the overlay would paint the entire screen and not just the few rain splashes). Mabye we could try to bind a particle effect (with an offset) to the player's head?

 

(It would work unless the player looks out of a window and it is supposed to rain outside).

[Baddcog]

Haven't touched rain, but I assume the parameters are adjustable right?

 

ie: number of drops/second and particle texture.

 

Eh, just looked at A-Z on the wiki and the texture just emits rain, doesn't say anything about adjusting rain other than making more then one patch.

Did they really hard code rain amount/particles? Maybe it didn't get used much in Doom3 but that is odd.

 

Has anyone just tried making rain particles? just like tree leaf particles? don't see why it can't be done. Then it could have fewer large particles with several drops per quad. It could be adjusted for amount/speed/time, etc...

[Tels]

Haven't touched rain, but I assume the parameters are adjustable right?

 

ie: number of drops/second and particle texture.

 

Eh, just looked at A-Z on the wiki and the texture just emits rain, doesn't say anything about adjusting rain other than making more then one patch.

Did they really hard code rain amount/particles? Maybe it didn't get used much in Doom3 but that is odd.

 

It uses a special "deform texture" command, and there is almost no documentation, and I am not sure it was ever used in D3, and we only seem to use it for rain/snow.

 

Also, the "density" is apparently adjusted by using another particle, but I am not sure how that really works. Something to find out on a long winter night...

 

Has anyone just tried making rain particles? just like tree leaf particles? don't see why it can't be done. Then it could have fewer large particles with several drops per quad. It could be adjusted for amount/speed/time, etc...

 

Don't think anyone has ever done this. The tricky part is to have the rain in a definite area, the "deform texture" makes that easy, just drag the patch out over the area you want to have rain. With particles, you would need to place multiples of them to over an area.

 

That is why I think a special "rain entity" could work much better, it would generate the raindrops as surfaces on a prebuild model and change them every frame, and that *might* be faster. Surely not something done in 5min, tho.

[Tels]

We should probably move the rain discussion to an extra thread.

 

Back to the topic at hand, in this case, LOD!

 

I noticed and fixed two bugs (2634 and 2635).

 

The first (#2635) was that if you had an entity with some LOD stages set via spawnarg, and then set a "hide_distance" smaller than these stages, this was ignored. E.g. an entity which shows until 2000 units away stage 1, until 4000 units away stage 2 and then gets hidden at 4000 units would ignore "hide_distance" "2500". Instead it really should get hidden from "2500" units away, regardless on what the LOD stages say. This way the mapper can easily override the hide threshold.

 

The second (#2634) was that if the user set the quality to "Low" or "Lower", this could lead to entities getting hidden on ridiculous low distances (as the user can tone down the quality all the way to 0.01 if we he wants). So I implemented support for a spawnarg "lod_normal_distance", which defaults to 500 units. This spawnarg sets the distance from where the lower quality is used. If the entity is closer than this distance, we pretent the user has at least set "Normal", so the entity stays visible. The testmap test_hide_distance has two rocks that show the difference. Here are screenshots:

 

There are two rocks, the left one has "lod_normal_distance" "500", the right one has "lod_normal_distance" "200". Both have a "hide_distance" off 500. As you can see in this screenshot (Object Detail set to "Lowest", distance roughly 450 units), the left rock is visible (since 450 < 500), the right not (since 450 > 200 units, and the "Lowest" Object Detail treats the rock as it was twice as far (900 units) and it is invisible at this distance:

 

 

Here is the same scene with "Normal" detail:

 

 

(You can see the different detail levels on the aphrodite statue on the right)

 

The right rock only appears if you walk closer than 200 units, here the two detail levels again:

 

 

Since the setting defaults to 500 units, that should at least prevent the most grave errors on hiding entities. It is documented in the def file and at the wiki.

 

The default needs to be modified depending on entity size (can be less for grass, and should be higher for statues/houses), but this is a sep. bug and I leave that for another day

 

Both fixes are probably going to be part of v1.05, as v1.04 is in freeze already. But it will be greebo's final decision if they are important enough to be included right away.

[Baddcog]

One bonus I just thought of is treasure items in chests. Treasure items can be high poly so hiding them at short distances like 100 in a chest should work really well (enough space in case someone opens and walks away)

 

Anyway, in some instances like that rock things can be hidden quite well in shadows too and even if they pop in it's not really noticeable. So distance isn't every thing. In my city mission wip I have done this with some pipes in corners. It's dark enough that you really can't see them even pop in or out.

[Tels]

One bonus I just thought of is treasure items in chests. Treasure items can be high poly so hiding them at short distances like 100 in a chest should work really well (enough space in case someone opens and walks away)

 

Anyway, in some instances like that rock things can be hidden quite well in shadows too and even if they pop in it's not really noticeable. So distance isn't every thing. In my city mission wip I have done this with some pipes in corners. It's dark enough that you really can't see them even pop in or out.

 

Yep, but I think I wasn't quite clear in my bug description. If you set a very low "hide_distance", than this is no problem under "Normal" or "Better" levels. The item hides either exactly where you want it, or later. And "later" is no problem, except for performance, but then, players expect a higher quality to go with lower performance.

 

The problem is the "Lower" levels which could cause the entity to vanish even lower than what the mapper intended and what is sensible. For instance if you put "hide_distance" 100 on a treasure item in a chest, then "Low" means they player could potentially never see it, as the visible distance gets shortened in the worst case to 50 units!

 

The new default of "minimum 500 units or the quality is not lowered under "Normal"" will fix this case, and the mapper can still override this.

[Tels]

Done some more work, esp. on particle emitters:

 

idEmitters are (under the hood) idStaticEntities, so they automatically think about LOD. That's cool, so you can switch f.i. smoke to a simpler emitter, or hide it. Unfortunately, when they switch models, the switch is immidiate. That means that if you have f.i. raising smoke, you get some discontinuities. But heh, at least it works

 

However, while working on this, I noticed two problems:

 

* SaveLOD()/RestoreLOD() was called for idStaticEntities and idEntity. That needlessly saved/restored the data twice (and I think it might also be a (small) memory leak on every Restore()). Fixed that. Savegames will be smaller by a few dozend bytes for each LOD entity now.

* idSmoke are not idEmitters, instead they are their own class. So they did not call idEntity::Think(), so they never thought about their LOD stage. This is fixed now, idSmoke can now have different particle models as everyting else. (idSmoke are not as efficient as idEmitters and should only be used for smoke that you bind to some moving entity, f.i.. If you want a chimney smoke, use an idEmitter). So whether them having LOD stages helps is a bit unclear, but it's just one small test in idSmoke::Think() so it won't hurt.

 

Open problem: It seems LOD stage 1 is never considered, the emitters switch between stage 2 and 0 only. This needs still fixing. Scratch that, I was testing with "Lowest detail" and the switch distance was "500", so it never got considered. Works as designed.

[Serpentine]

Uhm, http://www.iddevnet.com/doom3/ gives only 404 for everything? Is it gone?

 

Should be back up now

 

Also:

particle, <particleDecl>, Emit particles from the surface instead of drawing it.

particle2, <particleDecl>, Like particle but ignores the surface area so small surfaces emit the same number of particles as large surfaces.

 

The first will stretch the particle bounds to the edges of the face while the second will in effect tile the surface in proportion. Density itself is managed by the rain particle effect.

 

In regards to using rain sheets, you'll most likely find that they become very visible and also that you might lose performance if you're using specular or something to light them as the overdraw will increase quite quickly. Using simple blends will be ok I guess. But avoiding overdraw from particles is quite important.

[nbohr1more]

Holy...

 

Awesome Tels!

[Tels]

Holy...

 

Awesome Tels!

 

Erm, how come you praise me before I posted the stuff below?

 

Anyway, it occured to me that apart from "combining" func_statics, there must be a way to do the same with particle models. However, you can't just "copy" the particles together, as they are emitted and need to be updated all the time. Plus, the part that emits the particles is in the closed source portion

 

So I browsed through the source and noticed idMultiModelAF (in game/afentity.cpp), which presents multiple models to the renderer. Woa, perfect!

 

As a first stab I implemented a very similiar scheme into func_emitter. After the first success I was a bit worried that all the particles would be in sync, but it turns out you can do:

 

       renderEntity.origin = GetPhysics()->GetOrigin() + m_modelOffsets[i];
       renderEntity.axis = GetPhysics()->GetAxis();
       renderEntity.hModel = m_modelHandles[i];
       renderEntity.bodyId = i + 1;
       // make each of them have a unique timeoffset, so they do not appear to be in sync
       renderEntity.shaderParms[ SHADERPARM_TIMEOFFSET ] = -MS2SEC( gameLocal.time - gameLocal.random.RandomInt( 32767 ) );

       // add to refresh list
       if ( m_modelDefHandles[i] == -1 ) {
           m_modelDefHandles[i] = gameRenderWorld->AddEntityDef( &renderEntity );
       } else {
           gameRenderWorld->UpdateEntityDef( m_modelDefHandles[i], &renderEntity );
       }

 

And the internal renderer of Doom3 will copy the randomly choosen SHADERPARM_TIMEOFFSET for each model!

 

Here is proof that it works, first in the editor:

 

 

And here in game, as you can see, the 3 extra particle appear where they should and are indeed not synced (they look slightly different). The second picture is the proof that there is actually only one entity here:

 

 

Remarks:

 

* Usage cases are: multiple fog particles strewn over a wide area, multiple chimney smokes in a big city, or multiple particle leaves on trees.

* The render performance is likely unchanged, as the renderer still does all the same work. The only thing you are saving here is multiple entities, e.g. memory consumption, spawn time, entity thinking time, as well as having a smaller savegame.

* It is not very easy to use for the mapper, as the offsets and additional models need to be added manually without a visual help. In this case I planted little glasscube markers to see see which offsets I'd need to choose, then used brushes to measure them. However, the main appication will be less the case for mappers, but more for automated methods that collect multiple emitters and combine them into one model.

* Only the "main" particle model switches LOD. This needs more work, because either we restrict the mapper to have all the same particle in the different places (they can then share an LOD struct), or we need a way for the mapper to specify different LOD data via spawnargs. I think a combination of both will happen: either have multiple models with a common shared "hide_distance", or have all the same model with a common shared LOD data with stages.

 

Now I need to tie this into the SEED system.

[nbohr1more]

Would idMultiModelAF help with "bound children" issues as well (presuming this is referring to "bind")?

 

(Or am I lost in the woods again? )

 

 

Edit:

 

Yep, I'm in the woods...

 

I think the concept is:

 

"How do you propagate the bind relationship to all the spawned items that have been combined into one model?"

 

 

I did think, however, that idMultiModelAF could be used to issue two or more combined SEED models in one shot. So that would meet the criteria of combining more low-poly LOD stuff when it is farther away. Kinda like combining combined SEED's....

Edited February 20, 2011 by nbohr1more

[Tels]

Would idMultiModelAF help with "bound children" issues as well (presuming this is referring to "bind")?

 

I have no idea, the comments in the code say it is "for debugging only".

[Flanders]

This seed thing is quite ingenious .

 

Not sure, is there already some swaying grass/plants in the mod, that would be really cool with this. something like this:

 

 

textures/Flanders/gras_bruin

{

surftype15

description "foliage"

twoSided

nonsolid

noshadows

 

qer_editorimage textures/Flanders/gras_bruin

 

{

blend diffusemap

map textures/Flanders/gras_bruin

translate 0, 0.5

shear 0.075 * sintable [.1 * time + .005*parm9 + .005*parm10], 0

translate 0, -0.5

alphatest 0.25

}

}

 

 

or something that shears with sound of the wind:

 

 

 

textures/Flanders/gras_bruin2

{

surftype15

description "foliage"

twoSided

nonsolid

noshadows

zeroclamp

 

qer_editorimage textures/Flanders/gras_bruin

 

{

blend diffusemap

map textures/Flanders/gras_bruin

translate 0, 0.5

shear 0.2 * sound, 0

translate 0, -0.5

alphatest 0.25

}

 

}

 

[Tels]

This seed thing is quite ingenious .

 

 

Not sure, is there already some swaying grass/plants in the mod, that would be really cool with this. something like this:

 

At the moment I think the model generator is way to slow to rebuild the entire model everytime just to have "sway". In any event, it is not easily added, *unless* you add it via "texture translation", that sounds certainly like a workable idea.

 

I have to experiment with that, that sounds very useful for grass! Thanx!

[Sotha]

In L4D2 the mission replay value is increased due to items appearing in random places.

 

A theoretical question regarding SEED:

Could it be possible, using SEED, to have mission critical objects to appear in different location each mission restart? Could it be possible to spawn special clues pointing to the objects?

 

Example: magic talisman part a is spawned in a library and part b in a hidden room. Other options, where they could appear would be study, crypt, loft and storage.

If the part a was spawned in the library, then spawn in the clue readables pointing to the library in several locations.

 

Making this kind of randomizing could increase fm replay value and make each play time unique.

[Baddcog]

Well, in L4D2 the random spawns aren't game required items. It's things like pills, adrenalin, some guns... And when you learn the maps you know where the spots are, you just don't know whether or not stuff will be there.

) Then again I rarely play campaign as I don't find it that replayable. Even vrs is fairly predictable, but the players aren't and that's what really makes it so replayble imo.)

 

They only aid in game play. Or hurt game play because going in that closet to check could be the end of the round (especially in vrs).

 

So I can see using it for arrows, health, etc... possibly even small loot pieces.

 

But with Thief usually the main objective items or large loot pieces are placed in certain areas for certain reasons. They are hinted at in readables. Other players can help you finish a mission because they know where stuff is. Get too random and it would confuse everyone imo.

I guess if an author planned it carefully they could have references to 3 spots, and the piece would be in one of 3. Not too random, but not too predictable either.

I just don't see it as a good idea for objective items.

 

 

What I would like to see is possibly a script that could control how much random stuff is placed.

ie: small loot. Say you require there to be 450 worth of small loot pieces in 8 zones. then you make 12 seed zones.

And on load 450 loot is randomly created and spread amongst those 8 of those 12 zones. So small loot items (or health packs) could be in predetermined locations, but may or may not be there, and it may or may not ever be the same loot pieces. But the author could count on the loot goal not being broken due to not enough loot being placed.

 

(Or I guess authors could just place the goal required amounts in the map. Then let seed take care of random pieces of scattered loot just to mix it up some).

[nbohr1more]

It's funny that you mention L4D2, RadiatorYang mentioned that TDM would really benefit from an AI director.

 

I think that an "AI Director" might be antithetical to the Thief AI approach as having every AI be very smart and self-sufficient as possible seems to be a point of pride for Thief and TDM.