Storm Engine 2 (Doom 3 BFG fork) video

[motorsep]

Finally, put together a video showing progress on the game I have been working on.

 

The video particularly shows progress made on the engine:

 

[nbohr1more]

Gotta admit, there is some good work in there. Good luck with this project.

[motorsep]

Thanks I am going to release Storm Engine 1 soon (dhewm3 + tools + few rendering enhancements), but I wonder if anyone even care.

[nbohr1more]

We'd probably "care more" if there weren't so much antagonism around big structural changes. As it stands, with our

minimal coding crew the modus operandi is "don't upset the apple cart" which is understandable but sort of a shame.

 

I hope someday we end-up with a Git repo where others can easily pilot new stuff and tinker.

 

That said, if your game doesn't have a stealth element it might not have as much appeal here as it would

in another gaming spaces. There must be a forum where Oni and Cell-Shaded shooter game fans would congregate.

Just like Taleworld forums and RPGCodex are pretty keen on TDM.

 

I understand that this forum is sorta like the last embers of Doom3world for that modding community so it's cool to

see you and other D3W regulars around nonetheless.

[New Horizon]

We'd probably "care more" if there weren't so much antagonism around big structural changes. As it stands, with our

minimal coding crew the modus operandi is "don't upset the apple cart" which is understandable but sort of a shame.

 

I don't think it's antagonism about structural changes...it's simply being practical. People would drop in all the time over the years and say, "hey, you should port this to Unreal..idtech4 sucks...or port it over to the half life 2 engine"....this is no different. The problem with that is that you never really get anywhere...Duke Nukem forever is the commercial equivalent of that.

 

Switching to another engine is a serious investment, not only in coding hours but also in fixing maps and testing the hell out of everything. I have no doubt that a major shift to an engine like BFG would kill any and all remaining motivation in the key members of the team. It's not worth it.

 

If the existing engine can make use of some updates and optimizations from other strains of idtech4 without breaking everything, that is definitely the most logical way to go.

[nbohr1more]

I understand this. But it is surely a path to stagnation for an open source project to place a damper on tinkering. Sure, the core members should not be forced to endure any major upheavals in asset changes (etc) but coders outside the project should be able to offer technical demonstrations or optimization. There needs to be an outlet for growth and experimentation without so much implied FUD about breaking stuff and bugs. TDM needs its code in a public space where tinkerers can work their magic.

[motorsep]

I understand moving stuff to another engine. But dhewm3/RBDoom3/etc. (any idTech 4 fork) is the same engine that runs TDM. Game code is 99.9% the same as Doom 3, all assets are exactly the same, etc.

 

One thing is guaranteed - you can't bring BFG engine coolness into TDM as easy (or perhaps it's not even going to happen) as you could port TDM code into BFG codebase.

[New Horizon]

I understand this. But it is surely a path to stagnation for an open source project to place a damper on tinkering.

 

Stagnation in what sense? You can't put a Mod out and expect FM authors to keep using it if major engine changes come along and render all their previous work useless. Would existing missions only be playable in some previous form of TDM, would someone have to code support for legacy mode, or would FM authors or the core team have to carefully rebuild / fix/ update all previously existing missions to work with the new engines? As time goes on, it become less feasible to think of porting to a base that brings in too many changes and far more practical to try and implement the least invasive but most beneficial changes into the existing framework.

 

I don't think anyone here has any intent to be working on TDM forever, so it's likely best to work within realistic and attainable goals.

[nbohr1more]

Alright, to clarify... There are tons of things that could be done to optimize Id Tech 4 that would not break maps. Such changes could potentially be risky but that is the price you pay for improvements. There would be no onus whatsoever on existing team members to implement these changes because we would instead see them in the wild by outside contributors on github. If any of this outside work is deemed beneficial, then it could be merged back into the real TDM project. The risk of not continuing development is that we are already at a point where CPU single threaded performance is stagnant. As more games come out which take better use of GPU or multi core TDM's limitations will grow less and less appealing to mappers. Further, New Dark continues to gain steam as it has taken steps to modernize. The risk is obsolescence and exodus. The potential reward is larger views and better performance.

[New Horizon]

Alright, to clarify... There are tons of things that could be done to optimize Id Tech 4 that would not break maps. Such changes could potentially be risky but that is the price you pay for improvements. There would be no onus whatsoever on existing team members to implement these changes because we would instead see them in the wild by outside contributors on github. If any of this outside work is deemed beneficial, then it could be merged back into the real TDM project. The risk of not continuing development is that we are already at a point where CPU single threaded performance is stagnant. As more games come out which take better use of GPU or multi core TDM's limitations will grow less and less appealing to mappers. Further, New Dark continues to gain steam as it has taken steps to modernize. The risk is obsolescence and exodus. The potential reward is larger views and better performance.

 

I don't think anyone would take issue with those types of changes, but what was initially being said felt a bit vague so I wasn't sure what people meant. So yes, I too would love for TDM to be able to take full advantage of all that my CPU has to offer. How do we move beyond the hypothetical though? Is there some way for us to get a ballpark number of the optimizations that may benefit TDM and whether or not we could integrate them with relative ease?

 

I'm hopeful openmp could still offer us something.

[motorsep]

How can you break maps with new engine?

 

OpenMP is not a solution as it uses physical cores. So you have 2 cores, rendering runs on one, sounds runs on another. That's it, no more cores, no more performance increase.

[New Horizon]

How can you break maps with new engine?

 

If the engine is different enough, how wouldn't the maps be broken? I guess it's more accurate to say that if the engine was different enough that the maps would require changes, repackaging, etc, in order to work on something that was vastly different. Doom 3 BFG, as far as I know, does not support the original Doom 3 map format. So the maps would have to be re-released in the new binary format or the legacy format ported into D3 BFG...which has likely already been done?

 

Renderer changes could potentially affect how the maps play....our light gem is based on a render shot of a diamond model. If the lighting changes slightly ingame, brighter or darker, then the AI will behave differently than the mapper intended.

 

OpenMP is not a solution as it uses physical cores. So you have 2 cores, rendering runs on one, sounds runs on another. That's it, no more cores, no more performance increase.

 

It's better than nothing at the moment though. Currently Doom 3 is limited to a single core. If I had the option of being able to have TDM run on two cores, that would be an improvement over what we have now.

[motorsep]

Doom 3 BFG, as far as I know, does not support the original Doom 3 map format. So the maps would have to be re-released in the new binary format or the legacy format ported into D3 BFG...which has likely already been done?

 

BFG uses exactly same maps and all same assets as Doom 3. It automatically binarizes/compresses them into binary formats. So you won't even have to mess with maps to make them work with BFG engine.

 

Renderer changes could potentially affect how the maps play....our light gem is based on a render shot of a diamond model. If the lighting changes slightly ingame, brighter or darker, then the AI will behave differently than the mapper intended.

 

I don't see how it is an issue. I don't know who LightGem works exactly, but I assume it works based on information in .proc file, which contains precompiled shadow volumes. For testing purposes, compile a map without shadow volumes and see how AI reacts to it. It it works as expected, then it's something else. Either way, shadows in Doom 3 BFG are as dark as in Doom 3. The only thing that makes BFG brighter is perhaps light scale cvar and primitive bloom.

 

It's better than nothing at the moment though. Currently Doom 3 is limited to a single core. If I had the option of being able to have TDM run on two cores, that would be an improvement over what we have now.

 

Afaik, Doom 3 runs rendering on one core, and the rest on second core, if available (particularly sound engine). You already having 2 cores occupied. So people with dual-core CPUs won't feel any performance boost with OpenMP.

[New Horizon]

BFG uses exactly same maps and all same assets as Doom 3. It automatically binarizes/compresses them into binary formats. So you won't even have to mess with maps to make them work with BFG engine.

That's good to know.

 

I don't see how it is an issue. I don't know who LightGem works exactly, but I assume it works based on information in .proc file, which contains precompiled shadow volumes. For testing purposes, compile a map without shadow volumes and see how AI reacts to it. It it works as expected, then it's something else. Either way, shadows in Doom 3 BFG are as dark as in Doom 3. The only thing that makes BFG brighter is perhaps light scale cvar and primitive bloom.

 

The lightgem doesn't work that way. To obtain our lightgem readings, there is a diamond shaped model in place of the players body. A camera then takes a snapshot of the top and then one of the bottom of the diamond. The snapshots alternate every frame. The two shots are then averaged, giving the final lightgem reading you see on screen. Our lightgem is 'what you see is what you get'. So if there are any changes in how the lights finally appear on screen, brighter or darker, that will affect the final output of the gem and in turn alter the gameplay as intended by the mapper. That means either the lights in all the maps would need to be adjusted and tested, or the renderer altered to behave as it did in D3...the latter being more preferable I think.

 

Afaik, Doom 3 runs rendering on one core, and the rest on second core, if available (particularly sound engine). You already having 2 cores occupied. So people with dual-core CPUs won't feel any performance boost with OpenMP.

 

I'm pretty sure it only uses a single core for everything. That's what I've always heard over the years anyway, but maybe those people were wrong. Not sure.

[nbohr1more]

@motorstep: As far as BFG is concerned, I understand that you have some motivation to get us to convert to it

so that you can benefit from shared development. Which is an understandable motivation.

 

I don't think debating the merits will gain you much traction here.

 

If your coding team demonstrated a TDM port (say v1.07) running on BFG it would probably open the dialog for this direction.

Sorry if that sounds too steep of a demand, it's just a realistic appraisal of the current stances with regard to these types of changes.

 

(ranting over )

Edited June 17, 2014 by nbohr1more

[motorsep]

@motorstep: As far as BFG is concerned, I understand that you have some motivation to get us to convert to it

so that you can benefit from shared development. Which is an understandable motivation.

 

I don't see how I am going to benefit, if we already got the engine to the release state with almost all features we need for the game. I am almost certain what's really needed won't be fixed, since it hasn't been fixed to the date by anyone in any idTech 4 / BFG fork. Sorry for being blunt.

 

I don't think debating the merits will gain you much traction here.

If your coding team demonstrated a TDM port (say v1.07) running on BFG it would probably open the dialog for this direction. Sorry that if that sounds too steep of a demand, it's just a realistic appraisal of the current stances with regard to these types of changes.

 

I don't really care if TDM goes with BFG. I am just pointing out the reality of things when it comes to improving idTech 4 vs going with BFG (or even Storm Engine 2 when it's out to the public). Just think about it - all this time various devs have been trying to make idtech 4 perform better, and failed. At the same time it took us less time to fix BFG, make it moddable and improve rendering.

[New Horizon]

I apologize for any misconceptions and ignorance on my part. I'm just going by things I've read on the net. If using BFG is that easy and could prove a better platform than the dhewm3 fork, then it may be worth discussing internally. I'm not sure if Serpentine would be up to the task of looking into it should he return.

[motorsep]

I apologize for any misconceptions and ignorance on my part. I'm just going by things I've read on the net. If using BFG is that easy and could prove a better platform than the dhewm3 fork, then it may be worth discussing internally. I'm not sure if Serpentine would be up to the task of looking into it should he return.

 

Not a problem. What I am trying to get at is that finding gameplay programmer who can port game code from one codebase to another, especially when there is virtually no difference between stock codebases, is infinitely easier than finding a deep engine programmer, who can take old engine arch circa 2004, and turn it into modern code base, with performance matching leading engines (or even just getting noticeably better).

 

I still can not find available Cg/HLSL shader programmer, paid. Not to mention OpenGL / physics / particles programmers (paid also). Literally, there are not enough qualified workforce out there, especially when AAA companies are willing to pay big dollars for good programmers.

[nbohr1more]

I admit, I was originally dismayed by BFG and gave into the FUD about it too. The breaking of mod support and the (initial) reduction in dynamic

shadows in the game gave me the feeling that it's hybrid engine veered too far towards Id Tech 5's static light style. Then, the fact that it still

wasn't using a deferred rendering approach left me completely puzzled about the whole endeavor.

 

It wasn't 'till I read Fabian Sangarland's blog about it that I understood what happened.

 

It definitely would be great if we had TDM running on an engine where the CPU does not have to perform skinning and pass triangles

(rather than vertex or control points) to the GPU. Ever since it was discussed on Beyond3D back in 2004, I have wanted to know

what Doom 3 would be like when shadow code was less reliant on the CPU. It seems that BFG demonstrates what this would be like

but it does so in a very unusual way compared traditional solutions. I presume the BFG approach was done to make the the use of

GPU mesh deform a more cohesive part of the engine.

[motorsep]

Currently, BFG engine is as fast as it can get.

 

If rendering of transparent surfaces, physics and collisions are threaded. Then you can squeeze out even more performance boost. However, knowing what's involved into job queues threading, I don't think it's practical to do so.

 

Forward+ rendering and fast shadow mapping (not what tr3b has in RBDoom 3 BFG, which is _slow_) would bring really nice performance boost, but afaik forward+ requires OpenGL 4.x.

 

In other words, at this time, so sake of compatibility with older hardware, I think BFG engine is really solid engine to use, without any fancy rendering enhancements.

[motorsep]

Btw, we just restored Script Debugger, that wasn't working in stock Doom 3. Now there IDE to debug scripts in similar to C++ manner.

 

Screenshot is here http://www.kot-in-action.com/blog_new.html

Edited June 18, 2014 by motorsep

[HMart]

Motorsep without trying to start a fight with you or something i find your claims about BFG a little misleading, i don't know what you did to make it all work on your end but my tests with the RBDoom 3 BFG was not a walk in the park, none of my lwo models works on the BFG engine, i only see black boxes ingame and a simple map with a simple script that works fine in vanilla idtech 4 doesn't work at all, then an even more simple map with only worldspwn geometry didn't worked fine as well, i converted some of the geometry to func_static (just to test) and add all of them turn invisible ingame, but collusion was still working so i was colliding against invisible walls.

 

So even tho i would love to see TDM evolve to a much modern engine i can see how New Horizon is cautions on this and imo with good reason, from my small tests porting TDM to BFG would be a big undertaking because i'm sure many things would not work out of the box. This is the same thing that happened to the chinese room guys developers of the Dear Esther game, they ported their episode 2 source version of the game to the Portal 2 version and add many problems.

 

See this blog

 

http://www.littlelostpoly.co.uk/porting/

[motorsep]

@HMart: Well, either dig deeper (cos I actually got some mods from Doom 3, the ones without any C++ code involved, running with RBDoom3 and stock Doom 3 BFG), or wait until Phaeton alpha is out to get a hold of the engine. While we did take early RBDoom 3 BFG as base code, we did a lot of modifications to the engine to have it game dev/modding friendly.

 

Also, Storm Engine 2 != BFG engine. I don't know why you say I am misleading people. On that video, I show Storm Engine 2 running with scratch content, not BFG engine, not with stock Doom 3 BFG content.

[wesp5]

Just for a player's side of things, what would be the benefit of the BFG engine? If it's only about optimization, Doom 3 is old enough that most PCs run it maxed out nowadays anyway!

[chakkman]

Some TDM maps run choppy even on nowadays high end machines...

Edited June 18, 2014 by chk772