TDM Engine Development Page

[motorsep]

Hmm found this while searching but im not sure its for windows.

 

http://forum.unity3d...-editor.191595/

 

I do have a GLSL editor though, no not rendermonkey ill see if i can dig it up on my overfilled harddrives.

 

It's for Unity and it does't save into Cg / GLSL.

[SteveL]

Hmm and as for exporting the depthmap textures, might have to write some code for that, maybe take hints from the SaveTga function ? maybe we can use it without even writing a new function, ill see what i can do.

I've been trying that, using code nicked from other bits of the engine, but no joy so far. I've only tried to copy the image from the gfx card to a buffer then passed that to R_WriteTGA

 

byte * junk = (byte *)Mem_Alloc( potWidth * potHeight * 4 );
memset( junk, 0, potWidth * potHeight * 4 );
qglGetTexImage( GL_TEXTURE_2D, 0, GL_RGB8, GL_UNSIGNED_BYTE, junk );
R_WriteTGA( "SteveL_depth.tga", junk, potWidth, potHeight );
Mem_Free( junk );

 

Mostly I come out with plain black squares. I need to write some test code to see if there's anything but zeroes in the texture. It could be something silly like the values coming back are clamped to 0..1 and are being floored or rounded to 0 or 1 out of 255... that kind of thing. I can test that later, but any hints you can offer would be very welcome

 

Right now I'm trying to work out why using any custom shader prog in my diffusemap stage kills the alpha blend and my ignoreDepth keyword. Hoping I don't have to re-implement those in my shaders. This is all first-time-round learning for me so everthing's taking 8 times as long as it should!

[revelator]

Had a look at draw_exp.cpp and the funtion for making depthbuffer images for the shadows seems to be something to take into account.

 

Found the editor but might get a newer version from here https://www.opengl.org/sdk/tools/ShaderDesigner/ mines over a year old.

Also supports exporting to 3dsmax

[revelator]

static void R_CreateShadowBufferImage( idImage *image ) {
byte *data = (byte *)Mem_Alloc( lightBufferSize*lightBufferSize );
memset( data, 0, lightBufferSize*lightBufferSize );
image->GenerateImage( (byte *)data, 4, 4,
 TF_LINEAR, false, TR_CLAMP_TO_BORDER, TD_HIGH_QUALITY );
// now reset it to a shadow depth image
GL_CheckErrors();
image->uploadWidth = image->uploadHeight = lightBufferSize;
qglTexImage2D( GL_TEXTURE_2D, 0, GL_DEPTH_COMPONENT24_ARB, lightBufferSize, lightBufferSize,
 0, GL_DEPTH_COMPONENT, GL_UNSIGNED_BYTE, data );
qglTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_COMPARE_MODE_ARB, GL_COMPARE_R_TO_TEXTURE );
// qglTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_COMPARE_MODE_ARB, GL_NONE );
qglTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_COMPARE_FUNC_ARB, GL_LEQUAL );
// explicit zero depth border
float color[4];
color[0] = color[1] = color[2] = color[3] = 0;
qglTexParameterfv( GL_TEXTURE_2D, GL_TEXTURE_BORDER_COLOR, color );
GL_CheckErrors();
Mem_Free( data );
}

 

lightBufferSize is const int 1024.

 

In retrospect this function uses dynamic allocation on a const so could probably get away with using an array instead on data, eg data[1024][1024]; and skip the Mem_Alloc / Mem_Free completely.

Should not even be nessesary to image->uploadWidth = image->uploadHeight = lightBufferSize; as we are not using the function to render a texture but rather write it out.

So an R_WriteTGA at the end instead hmm ???.

[SteveL]

Had a look at draw_exp.cpp and the funtion for making depthbuffer images for the shadows seems to be something to take into account.

 

TDM doesn't have a draw_exp.cpp but there are dozens of copies on github. I'll have a read.

 

EDIT: crossed with your post. I'll try to digest that...

 

EDIT2: I'm not worried about performance. This is purely for diagnostics, unless there's something I haven't thought of

Edited October 7, 2014 by SteveL

[revelator]

draw_exp was an experimental renderer it was disabled in vanilla but theres some interresting stuff in there like shadow maps instead of shadow volumes, bloom, and other stuff.

the shadowmap code was rather nasty in some places (used the non portable windows gl wgl) and is probably the reason why they ditched it.

Looking around i found evidence that most use FBO's for depthbuffer operations but id prefer not to go down that path, since it would mean rewriting large parts of the image libraries.

It might be worth it though but id rather not be the one to do it :S i have tons of work ahead allready.

[motorsep]

So I take it there is no interest porting TDM to BFG derivative ?

 

P.s. Yes, I am aware there is no publicly available stable, production ready fork yet

Edited October 7, 2014 by motorsep

[nbohr1more]

I think there's "some interest". The main problem (I can see) would be porting all the custom GUI code which is derivative of vanilla's

native GUI code and not compatible with Flash. Though the game dll model would also be pretty awkward to move from

and we've grown accustomed to using it (warts and all).

 

I see that the OpenIdTech branch of RB-DOOM3BFG is switching to yet another GUI system which is a shame since the GUI code in

vanilla is already GPL... why not extend and improve it rather than replace it?

 

I'd love to see someone show a proof-of-concept with an older TDM build (1.07) running under BFG though.

[motorsep]

I think there's "some interest". The main problem (I can see) would be porting all the custom GUI code which is derivative of vanilla's

native GUI code and not compatible with Flash.

 

That may be. Although you can do a way more with Flash, especially animation wise. Not to mention higher res images, atlasing and all that (automatic atlasing).

 

Though the game dll model would also be pretty awkward to move from

and we've grown accustomed to using it (warts and all).

 

And how is that? It's the same exact code. Just instead of DLL and EXE you get only EXE. There is no voodoo about it. Afaik there are no 3rd party mods for TDM, so why even care for that DLL ? Debugging is easier with monolith build, and memory management is better.

 

I see that the OpenIdTech branch of RB-DOOM3BFG is switching to yet another GUI system which is a shame since the GUI code in

vanilla is already GPL... why not extend and improve it rather than replace it?

 

Because old fullscreen GUI was a major CPU hog? That was one of the reason to replace it with Flash. Or do you mean they are putting in something entirely different, some 3rd party lib ? If so, that's nuts. You might want to scratch off that OpenIdTech off the list of useful forks.

 

Currently we are implementing proper shadow mapping, using RBDoom 3's one as base. Initially we thought it was pretty good, except shader is a bloated horrible mess. Turned out even C++ code has pitfalls. So instead of small project, now it turned into more invasive port.

 

So whether it is idTech 4 or BFG, a team needs a savvy engine programmer, savvy shader programmer, and savvy gameplay programmer. These engines are not to be taken lightly, even though the code is readable and clean.

 

I'd love to see someone show a proof-of-concept with an older TDM build (1.07) running under BFG though.

 

Why waste time? It's a fact that BFG engine is faster than idTech 4.

[revelator]

Hmm wouldnt call the old gui a cpu hog its an xml parser they are by design pretty light on resources.

There are other more deserving targets for that claim like old OpenGL 1.4 calls (there are quite a lot in vanilla) or lack of a multithreaded renderer .

 

I can respect that you find vanilla archaic and would rather force others to use the newer engine but take into account the work these devs have put into darkmod,

it might be hard for them to totally scrap everything they created so far and starting over.

 

On the positive account we might end up with some nice improvements to vanilla even if it takes some time.

Its also a good learning experience .

[SteveL]

Agreed, there's plenty we can do incrementally to update the engine without needing to start an unfinishable migration.

 

Its also a good learning experience .

 

Speaking of which...

 

why using any custom shader prog in my diffusemap stage kills the alpha blend and my ignoreDepth keyword. Hoping I don't have to re-implement those in my shaders. This is all first-time-round learning for me so everthing's taking 8 times as long as it should!

 

I can withdraw this bit of my request for help. I now have an assembly shader playing nicely with my new keyword and with the particle system's use of vertex colours to control alpha, and that lets me fade out parts of the particle selectively based on coords. It's not a PP effect either. It'll draw at the same time particles do now, with the rest of the alpha blended materials.

 

Two problems down, just the depth sampling to work on now before I can put it all together.

[revelator]

Nice work steve yeah im also still beating my head around the depthbuffer, i wish sikkpin was around as he has more experience with that part.

My depthbuffer code might also be of use to him and he could probably iron out any bugs i might be responsible for heh.

 

Im going to have a look at FBO's and see if i can find a not to messy way of working them into vanillas code, should help a bit with the above.

[SteveL]

Cheers. Yes I've read that FBOs are the 'modern' way to do it, but not studied them yet. I'm learning what I need to to get my head around the existing set up before I think about any changes. I'll have another crack at the depth capture tonight. For all I know it might be working already. Monday's tests didn't settle it one way or the other, but I'll try to get an answer to that tonight.

[revelator]

void idImage::CopyDepthbuffer(int x, int y, int imageWidth, int imageHeight, bool useOversizedBuffer) {
this->Bind();
// if the size isn't a power of 2, the image must be increased in size
int potWidth, potHeight;
IMAGE_ROUND_POWER2(imageWidth, potWidth);
IMAGE_ROUND_POWER2(imageHeight, potHeight);
GetDownsize(imageWidth, imageHeight);
GetDownsize(potWidth, potHeight);
// Instruct openGL that we won't bind a color texture (depth images are greyscale).
glReadBuffer(GL_NONE);
// only resize if the current dimensions can't hold it at all,
// otherwise subview renderings could thrash this
if ((useOversizedBuffer && (uploadWidth < potWidth || uploadHeight < potHeight)) || (!useOversizedBuffer && (uploadWidth != potWidth || uploadHeight != potHeight))) {
 uploadWidth = potWidth;
 uploadHeight = potHeight;
 if (potWidth == imageWidth && potHeight == imageHeight) {
  glCopyTexImage2D(GL_TEXTURE_2D, 0, GL_DEPTH_COMPONENT, x, y, imageWidth, imageHeight, NULL);
 }
 else {
  glTexImage2D(GL_TEXTURE_2D, 0, GL_DEPTH_COMPONENT24_ARB, potWidth, potHeight, 0, GL_DEPTH_COMPONENT, GL_UNSIGNED_BYTE, NULL);
  glCopyTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, x, y, imageWidth, imageHeight);
 }
}
else {
 // otherwise, just subimage upload it so that drivers can tell we are going to be changing
 // it and don't try and do a texture compression or some other silliness, still need to create it as depth buffer image though.
 glTexImage2D(GL_TEXTURE_2D, 0, GL_DEPTH_COMPONENT24_ARB, potWidth, potHeight, 0, GL_DEPTH_COMPONENT, GL_UNSIGNED_BYTE, NULL);
 glCopyTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, x, y, imageWidth, imageHeight);
}
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
backEnd.c_copyDepthBuffer++;
}

 

try this one out, i removed the fix for bilerp seams (not needed and might have broken it) and changed the depth component to GL_DEPTH_COMPONENT24_ARB (better).

 

the extra glTexImage2D(GL_TEXTURE_2D, 0, GL_DEPTH_COMPONENT24_ARB, potWidth, potHeight, 0, GL_DEPTH_COMPONENT, GL_UNSIGNED_BYTE, NULL);

is a test cause im not sure just copying the depthbuffer data is enough (or does the driver know its a depthimage when not specified ?).

[SteveL]

Thanks, will do.

 

I've had trouble working out what the different calls are for. It doesn't help that the different paths in the vanilla function use three different setups, apparently to do the same thing, and the function was unused and so might never have worked. My latest hypothesis is that the call to glTexImage2D just enlarges the texture before it gets written to. Also, capturing rgb might be enough if the early depth pass writes a color gradient instead of just the depth buffer. It does bind _white iirc.

 

I have a hope of getting to the bottom of this now that I've got asm shaders working as a diagnostic tool. I can take my test map with its one shaded patch and use it to write color to the screen to find out what's going on. I've already used it to figure out which variables are in the 0..1 range and which are in the screen coords range for example (because there's no readable documentation for asm--Just an indigestible spec from the ARB).

[revelator]

http://glampert.blogspot.dk/2014/01/visualizing-depth-buffer.html

 

found some usefull info here, looks like we have to read the backbuffer ? which left me with a wtf moment, cause someone told me not to read from that.

also it seems really simple compared to my massive function. Also a test shader on that site sadly its in GLSL so not directly useable with vanilla.

 

btw this part

 

// otherwise, just subimage upload it so that drivers can tell we are going to be changing

// it and don't try and do a texture compression or some other silliness, still need to create it as depth buffer image though.

glTexImage2D(GL_TEXTURE_2D, 0, GL_DEPTH_COMPONENT24_ARB, potWidth, potHeight, 0, GL_DEPTH_COMPONENT, GL_UNSIGNED_BYTE, NULL);

 

remove it again, it causes the engine to run to a crawl.

 

And same here, newer found a lot of good info about ASM shaders :S

[motorsep]

Hmm wouldnt call the old gui a cpu hog its an xml parser they are by design pretty light on resources.

 

That isn't a hypothesis. It's what I was told by ID Software on the subject "why did you drop old fullscreen guis".

 

I can respect that you find vanilla archaic and would rather force others to use the newer engine but take into account the work these devs have put into darkmod, it might be hard for them to totally scrap everything they created so far and starting over.

 

I failing to understand where this assumption comes from. As if you or TMD programmers never bothered to check the game code in BFG against stock Doom 3 / RoE.

 

On the positive account we might end up with some nice improvements to vanilla even if it takes some time.

Its also a good learning experience .

 

Time isn't your friend A few years back, there was some info on ARB shaders circulating around. It's not the case today. A few years from now, you might be in a position where ARB shaders are no longer supported in drivers, and the engine still has no significant performance gains. Whether porting it to BFG would end up having max performance you can get our of idTech 4+ architecture, plus having modern code that is easy to maintain. The way I see it from the perspective of finding programmers for engine vs gamecode, it's easier to find someone to work on the game code (helping porting it, maintaining it, etc.) vs finding someone to turn idtech 4 into BFG.

 

Again, it seems like some folks here looking at this as personal gig, just something to poke at and learn new things, and some looking at it as actual end-goal project. Go figure :/

[SteveL]

Again, it seems like some folks here looking at this as personal gig, just something to poke at and learn new things, and some looking at it as actual end-goal project. Go figure :/

 

Now that's unfair. We can always do more of course, but TDM is an award-winning project* that has delivered its goals, and deserves recognition as such.

 

*before I had anything to do with it so I'm not blowing my own trumpet here.

[Tels]

That isn't a hypothesis. It's what I was told by ID Software on the subject "why did you drop old fullscreen guis".

 

There are surely quite a few reasons why one would want a different GUI:

 

* Unicode support and I18N (no longer 8-bit bitmap fonts)

* more effects like soft fonts, glow, or blur

 

But performance is really the least concern, at least for TDM. Esp. because most of the GUI (menu) has no performance impact on the game at all and the in-game GUI rarely has any (the only thing I can think of is the underwater GUI and surely we could replace this if it is really that important. But it seems it wasn't so far).

 

However, changing the GUI to Flash? Flash from Adobe? You are kidding, right? Apart from the fact that you lose Linux (Flash support has ended on Linux several years ago), there is also this: http://yro.slashdot.org/story/14/10/07/1725232/adobe-spies-on-users-ebook-libraries

 

I'd consider Flash a dead-end - who knows if Adobe still supports in in 3 years? Maybe then they moved everything to the cloud, or whatever.

 

Even if we let the actual technology aside, changing to a new GUI is also a lot of work. TDM has already never enough developer capacity, so redoing the entire GUI "just because" isn't going to fly.

 

Time isn't your friend A few years back, there was some info on ARB shaders circulating around. It's not the case today. A few years from now, you might be in a position where ARB shaders are no longer supported in drivers, and the engine still has no significant performance gains. Whether porting it to BFG would end up having max performance you can get our of idTech 4+ architecture, plus having modern code that is easy to maintain. The way I see it from the perspective of finding programmers for engine vs gamecode, it's easier to find someone to work on the game code (helping porting it, maintaining it, etc.) vs finding someone to turn idtech 4 into BFG.

 

I do think you underestimate the changes TDM has already done to the idTech engine - an engine switch is not that simple as it sounds. Especially not if the new engine needs some kind of new art assets,TDM has simply not enough people to work on these areas. We are happy if we can fix the most obvious bugs and add a few features from now and then.

 

Of course, having the engine decoupled and getting worked on by other people would be cool, but I can totally see why this isn't happening.

[Springheel]

Again, it seems like some folks here looking at this as personal gig, just something to poke at and learn new things, and some looking at it as actual end-goal project. Go figure :/

 

Yeah, starting with Doom 3 and making a new, entirely standalone game that wins PC Gamer's Mod of the Year award is just us "poking around".

 

If you're so convinced that it's easy and beneficial to port TDM to BFG, then put together a proof of concept and demonstrate it. Trying to badger other people into doing work that only you seem to see the value in is not going to accomplish anything.

[AluminumHaste]

And also, ALL of this is a voluntary.

If people want to donate their time to this project it's welcome. Whether it's just to poke around and learn and maybe make some beneficial changes, or because they want to stick with it to the end, it's all welcome.

There's lots of people who've come and gone in a flurry of activity, and there's some who've stuck around for years and spend all their free time with his project.

And there's also every one in between. It's all welcome.

[Tels]

Addedndum: I guess what I tried to express with the above post was that TDM lacks the ressources for large changes (like changing the GUI, adding a differen physics engine, or even port to a new engine).

 

Now, a few of these might be beneficial (because you can then delegate work to other teams and just use their resources). However, without the initial "push" the change isn't going to happen.

 

Also, what SteveL said: TDM has more then reached it's goals. Everything else is a bonus right now. So while I can see the justification for some changes from a ressource-management POV, I don't think big changes will happen. Unless somebody goes ahead and just does it - then we can see how far it brings us

[Tels]

Again, it seems like some folks here looking at this as personal gig, just something to poke at and learn new things, and some looking at it as actual end-goal project. Go figure :/

Yeah, starting with Doom 3 and making a new, entirely standalone game that wins PC Gamer's Mod of the Year award is just us "poking around".

 

To be fair, I read this line the other way round: He was trying to say we see the BFG port/engine work as only a toy port and poke to learn, but don't want to use it.

 

Which is a completely different meaning.

[motorsep]

But performance is really the least concern, at least for TDM. Esp. because most of the GUI (menu) has no performance impact on the game at all and the in-game GUI rarely has any (the only thing I can think of is the underwater GUI and surely we could replace this if it is really that important. But it seems it wasn't so far).

 

HUD is fullscreen GUI script. That might what they were talking about.

 

However, changing the GUI to Flash? Flash from Adobe? You are kidding, right? Apart from the fact that you lose Linux (Flash support has ended on Linux several years ago), there is also this: http://yro.slashdot....ebook-libraries

 

Yeah, and believe or not, there are no dependencies for Flash in BFG engine. And it runs just fine on Linux and Mac.

 

Even if we let the actual technology aside, changing to a new GUI is also a lot of work. TDM has already never enough developer capacity, so redoing the entire GUI "just because" isn't going to fly.

 

I understand that. That's why I suggested porting old Doom 3 fullscreen GUI system to BFG.

 

 

I do think you underestimate the changes TDM has already done to the idTech engine - an engine switch is not that simple as it sounds. Especially not if the new engine needs some kind of new art assets,TDM has simply not enough people to work on these areas. We are happy if we can fix the most obvious bugs and add a few features from now and then.

 

Can't really compare game code programming to engine programming (rendering, etc.; lower level tasks). Game code is just more work, but naturally simpler by nature. So it would really just come down to spending quality time with WinMerge Unless TDM engine rendering part is vastly different than Doom 3.

 

If old GUI is ported back, assets would be all the same by the way. The only things that is different are shaders and fonts, but there is a font tool that comes with RAGE SDK that generates fonts for BFG engine.

[motorsep]

Yeah, starting with Doom 3 and making a new, entirely standalone game that wins PC Gamer's Mod of the Year award is just us "poking around".

 

I think you misunderstood me. Poking around references to recent attempts to modify renderer and such, not the game code of TMD.