Revelator's TDM Branch

[revelator]

Hmm that was my first try unfortunatly while it compiles it crashes the engine on map load.

[SteveL]

Is fog lamp being blendlight as good as normal fog lamp, but with less performance overhead?

 

Fog lights are blendlights... a special kind of blendlight that creates extra tris round its light volume and draws on them as if there were patches there. That's so a thick fog can be seen in a wide open area, where there's no solid surface for it to hit.

 

@revelator: yes I think ambients create interactions, though I think they have a special shader too so they might skip some interaction stages. Ambients are the one drawing stage I haven't lokoed at at all for any of my own projects yet. I didn't get what the problem was just by looking through the commits posted above... Are you putting a material stage keyword in the light shader or the material shader? If you point me to the right commit in your repo I'll have a play with it too tomorrow when I'm less sleepy!

[revelator]

Yep used the test material supplied by nbohr1more, still crashes if i remove the interaction code in draw_arb2.cpp it works again though.

so i suspect we need some code to check if the material is active or else fallback to the normal, strangely enough the IsCustomLight check allways returns true even if theres no material

[SteveL]

I had a quick look at your "removed" commit but I'm still struggling to catch up on the conversation. One point that confused me is the code touches the "shaderpasses" parts of the renderer whereas I'd expect we'd want to use the light-interacting drawing phases.

 

Correct me I'm wrong on any of this... First and foremost, we want to add a customLight keyword for new opaque surface shaders, as opposed to alpha blend materials or light shaders.

 

Only opaque surface materials are light-interacting and are locked down to a specific interaction shader right now. For other types, we can already define custom ARB programs. Those are the "newStages" being mentioned above: shaders that get drawn after the lighting passes and that provide their own shader programs.

 

There would be nothing to stop a mapper creating a special material right now, adding an appropriate ARB program, then passing it a light position and color using shaderparms and drawing a beautiful fur in TDM. The mapper would get problems though for a few reasons:

1. The shader would be classed as a newstage because it had its own ARB program, so it would be drawn after the light interactions, during the "shader passes". It would be ignored during the early depth pass and not write to the depth buffer.
   
2. The surface would be drawn only once, so the number of lights whose details could be passed to it would be limited by the number of shaderparms available. It would take 6 parms to pass the origin and colour of a light, so the limit would probably be one light.
   
3. The surface wouldn't receive shadows. By the time the shader passes get drawn, it's too late to rely on the stencil buffer.
   

I'm assuming we want to create opaque, fully light-interacting surfaces that get lit by however many lights strike them. So we don't want the renderer treating them as newstages, we want it to draw them during the lighting pass where it cycles through each light in turn and builds up colour for the surfaces hit by each light in the color buffer. So we'd want the front-end to create one interaction for the surface for each light as normal, but we'd want to swap in a new ARB program at the right moment.

 

Questions...

 

Is that roughly right, or am I on the wrong page completely?

 

Like you noted above, the engine sets a load of hard-coded environmental variables for the light interaction shader to use, and it does it once before it draws all the surfaces for a light. It would simplify things a lot if we required that any new shaders use that same set of variables, referenced by the same numbers. Would that be a problem for the plan?

[revelator]

Exactly and thats why if we want this to be fully able to control all light via custom shaders that i ment it would have to be early on in the shader process, it will be one hell of a coding nightmare but it should be doable. And as you pointed out the engine can allready use custom shaders and theres nothing keeping them from defining a lightsource but there will be drawbacks.

Im pretty much 100% that if we want this in we wont be able to do it with a few lines of code i suspect something in the line of several hundreds of lines of code hehe.

[nbohr1more]

Hmm. I initialized customLight = false; in material.cpp. I will double check for a typo there.

 

Edit: nope... no typo ... there's a weird parse portion where it looks for !imcp token and then sets the value to true

but it does this for ambientLight, etc and all those work as expected. Unless there some weird boolean flipping elsewhere

this shouldn't matter...

 

OK, that pretty much settles it. We probably can't use the existing newstage variable to store this because it is triggering too much other architecture.

Time to make a new storage attribute that the engine wont notice till the bind branch.

[revelator]

Was starting to wonder myself why it got flipped on, does vanilla allready use the customLight token somewhere ?.

But yeah better to put it off for now untill we can make some headway into doing this correctly.

[SteveL]

i suspect something in the line of several hundreds of lines of code hehe.

I'm more optimistic! Or rather, I agree, but I'm not sure we're talking about the same code I think we can do it with just a few lines of engine code provided that we're willing to rewrite unlimited amounts of ARB code. Basically, the engine changes can be very small if and only if we are happy to adapt any shader program that we want to use to the current engine setup. From my point of view, I think that that's ok, because I'm focussed on TDM and whatever system we adopt, if I bring a shader program from anywhere into TDM, I'd optimise it for TDM anyway. But if you want this branch to work with other projects, I'm happy to help design something more flexible.

 

My thought is, the engine changes can be light if we can apply some heavy constraints to any new shader programs: they use the existing idTech4 environmental variables to get their data. If there are shader progs we want to use that need other stuff, we add it. TDM passes only a dozen or so variables to shader progs right now, so we have plenty of free slots. But let's make sure that if we add variables, we do it in a way that's conducive to any shader-program-variable-management system that we might have to add later if we run out of slots.

[revelator]

If we go full shader hum yes if we still want support for older cards then no hell no, but atleast your optimistic .

Take a rather convinient engine as an example, if you ever heard of an engine called tenebrae ? it was the first quake engine out there with support for PPL bumpmapping specular pretty much the works.

It had one downfall though at the time OpenGL really didnt have much in the shader support department but it did have some ARB support, the shader part took up about 100 lines of code, the support code

took up 5000+ lines of code and it was very very buggy and extremely slow. Darkplaces took some of the ideas behind that engine and refined them cooking down the support code to a few hundred lines instead but darkplaces also support GLSL besides ARB, and you need to have a card that supports at least ARB2 for it to work to its strengths.

 

So it will be quite a lot of code in case we want this on cards with suboptimal shader support, but it can be cooked down considerably with shader capable cards, still it will take some work

 

Btw the guy who made the tenebrae engine back then, should know what hees talking about since hees the lead dev on enemy territory quake wars

imagine my shock when i found out that was the reason he let the tenebrae engine die back then.

[SteveL]

I admit I don't know anything about hardware or gl support, but I've been blithely assuming that if we stick to techniques that were available in the original idtech4 engine, those that hit the top end of its range in 2004 like ARB2 shaders, then every card in use nowadays will support those techniques, and therefore we should be able to remove the pre-ARB2 dross for pre-2004 cards. Am I wrong?

 

Whether we can remove support for old specs is always a hot topic. There's no objective right answer to which is best: the game progressing vs the game being playable on 10-year old hardware. Sometimes a compromise is possible, with both being supported in some way, but that's often a huge amount of work -- so big that it completely rules out the possibility of the game progressing.

[nbohr1more]

Yeah, no DX8 hardware could ever realistically run TDM. TDM's minimum spec at launch was Radeon 9600 \ FX5200 and even that was pushing it.

You really needed to have a Geforce 6600 or higher to play anything beyond the smallest missions with any reasonable comfort.

I simply don't see the point of supporting pre-DX9 hardware other than masochism or to impress some FOSS warriors... Sticking with ARB is already

in that vein too

[SteveL]

Tbh I thought you guys had already removed the decrepit non-arb2 stuff from the engine. That's one reason why I want to try out gpu skinning in your branch rather than in the trunk -- it won't be practical to implement it or work around it for older GPUs, so any benefits would be ruled out. And I think TDM effectively removed support for non-ARB2-capable hardware years ago -- no-one without an arb2 card has been able to see water for the last several years, for example -- although yes we'll need to have a discussion and make a good case to the community before removing the code from the engine forever. I've only been around TDM for a year but I've seen a few discussions on this kind of question. We'll have a much better chance of persuading the community at large that advancements are worth the cost of dropping support for elderly machines if we have a few carrots to hold out So (personal opinion only) I think we should go on and try implementing some goodies that require hardware from the last decade

[revelator]

Most of it left in backends for the nv20 and the r200 as they do have some shader capability and still work on modern cards, but you're right we should just scrap those to, its not worth the effort to

keep them alive anymore and they add to the complexity rather a lot. BTW the ARB backend had no shader support at all, it was just ment as a fallback in case all else fails but it did have one unintended

side effect -> you could actually play doom3 on a card as old as a voodoo2 . Sadly the r200 and nv20 backends are from an era where gfx card manufacturers could not agree on a commmon interface

so they used ATI specific and nvidia specific opengl pointers for the funny stuff, if you'd try running the r200 backend on a nvidia the engine would crash and vice versa. ARB2 solves that as it uses the standardized versions that where added to the spec later. The reason i belive it will take a bit more than a few lines of code is that vanilla allready uses a rather large heap of the code to support per pixel lighting because that part is old and at the time when it was build it still needed to support the older gfx cards as well, in fact the portion of the shaders directly involved in light are not much bigger than shadow.vfp which consists of about 4 lines of shader code :S the rest is handled by the support code which is much much larger -> (tr_light.cpp tr_lightrun.cpp interaction.cpp tr_stencilshadow.cpp etc.)

and its not really all that programmable, but enough to be used for most things by using material code (used for specifying color / reflectiveness etc).

 

 

As i see it we could do this by digging out the parts for lighting from interaction.vfp and load customized versions via material scripts, but that would break backwards compatibility, or we could try to come up with a way to override the defaults in interaction.vfp with the shaders we specify in materials using a keyword, like if _customLight skip reading from interaction.vfp and read from where i tell you.

This might also work for other stuff like Parallax occlusion mapping which does not behave nicely with all kind of surfaces.

[nbohr1more]

OK, I think I should've looked back sooner but it seems that sikk just made a idstr parameter and then populated it via Find ARB program in Material.cpp

then simply:

 

qglBindProgramARB( GL_VERTEX_PROGRAM_ARB, surf->material->GetInteractionVP_Amb() );

 

I think I'm a few more experiments away, I can taste it...

[SteveL]

Good luck!

[Tels]

Sorry for the long delay, RL interfered.

 

The latest git pull does still not compile on my system, it misses a compatible libglew. However, I haven't found a package for my old kubuntu yet and had not yet the time to compile it from source.

[Tels]

Tbh I thought you guys had already removed the decrepit non-arb2 stuff from the engine. That's one reason why I want to try out gpu skinning in your branch rather than in the trunk -- it won't be practical to implement it or work around it for older GPUs, so any benefits would be ruled out. And I think TDM effectively removed support for non-ARB2-capable hardware years ago -- no-one without an arb2 card has been able to see water for the last several years, for example -- although yes we'll need to have a discussion and make a good case to the community before removing the code from the engine forever. I've only been around TDM for a year but I've seen a few discussions on this kind of question. We'll have a much better chance of persuading the community at large that advancements are worth the cost of dropping support for elderly machines if we have a few carrots to hold out So (personal opinion only) I think we should go on and try implementing some goodies that require hardware from the last decade

 

I'd just like to throw out some food for thought here: We shouldn't give too much weight to people who only theoretically argue for "but the people with the old hardware can't play TDM!". The "can't see water" is proving this. If the support is already broken, and nobody bloody noticed it in the last few years (!), then we don't need to keep it around, regardless on what armchair engine users theorize about

 

The TDM engine is already way too complex and underdocumented, keeping old, but broken features might be more hassle then ripping them out, f.i. when someone wants to figure out something and then has all the broken code to confuse them. There is a cost in maintainance and we are lacking in personal resources everywhere.

[revelator]

Ok then removal of all backends beside ARB2 in next commit

Need to investigate a crash after i reverted nbor1more's test code, seems i overlooked something after i fiddled with it.

[Tels]

Ok then removal of all backends beside ARB2 in next commit

Need to investigate a crash after i reverted nbor1more's test code, seems i overlooked something after i fiddled with it.

 

And don't forget to take out the never-used glasswarp

 

Btw, I wonder if this is the problem with GLEW on my system:

 

# greebo: Use custom compiled GLEW devIL and libjpeg in Linux

 

Your branch added GLEW in the above comment (compared to stock TDM), but it's only a comment. Maybe it is missing some actual scons magic?

 

Edit. There is a file SConscript.gl, but it is never referenced.

Edited December 6, 2014 by Tels

[nbohr1more]

OK now I'm getting a little further out of the forest.

 

I think I need to make whatever idstr value I grab a subset of lightShader...

 

but I found a comment that is giving me some pause:

 

tr_local.h

 

// back end should NOT reference the lightDef, because it can change when running SMP

 

which means both my work and Rebb's ambient workaround break the possibility of repairing SMP

frontend vs backend processing as is. I guess vLight could be excluded from that SMP work?

 

Eh, what's done is done... time to return to studies... so close...

[SteveL]

It's the render light that you have to avoid referencing. It's ok to use the viewLight. That's what it's for, to keep the back end clear of the front end's modifications.

From tr_local.h:

// viewLights are allocated on the frame temporary stack memory
// a viewLight contains everything that the back end needs out of an idRenderLightLocal,
// which the front end may be modifying simultaniously if running in SMP mode.

 

Did you guys ever get round to creating an svn branch by the way?

If not what's the status of the git repo? Does it compile and run ok right now?

[revelator]

Not yet though ill start work on it by the weekend.

trunk compiles but crashes atm after reverting the light commits by nbhor1more, think i forgot something but have not had the time to look into it.

I hope to get it back in working order again by the weekend.

[Tels]

It's the render light that you have to avoid referencing. It's ok to use the viewLight. That's what it's for, to keep the back end clear of the front end's modifications.

From tr_local.h:

// viewLights are allocated on the frame temporary stack memory
// a viewLight contains everything that the back end needs out of an idRenderLightLocal,
// which the front end may be modifying simultaniously if running in SMP mode.

 

Did you guys ever get round to creating an svn branch by the way?

If not what's the status of the git repo? Does it compile and run ok right now?

 

It doesn't link on my system, because libGLEW is too old (or incompatible or I don't have the 32bit variant. However, I just remembered getlibs:

 

~/src/tdm_experimental/The-Darkmod-Experimental/src$ getlibs -p libglew1.5 libglew1.5-dev
The following i386 packages will be installed: libglew1.5 libglew1.5-dev
Continue [Y/n]? Y
Downloading ...
Installing libraries ...

 

Not sure if that already fixes it.

[revelator]

unless libglew version 1.5 is to old (it was not on my machine) it should be enough i removed the glew include dir from the scons project so that it picks up the system provided one instead.

Also changed the include path for glxew.h to <GL/glxew.h>

[Tels]

Hm, and how do I update? "git pull" says "already uptodate" even tho I haven't updated for a week or longer.