Roadmap for TDM in 2020

[nbohr1more]

I hate to join the chorus of negativity but I honestly feel like Vulkan itself is a bad move.

 

It pretends to be a modernized "clean slate" OpenGL but it really just encourages vendor lock-in since it is so closely tied to the hardware.

 

It is mostly big industry players getting together and going "why do we need a flexible open standard when we are the only players left?"

then collaborating on a standard that will only work for big industry players.

 

The only thing I can say for it's defence is that OpenGL has been tainted by Nvidia's overwhelming ARB voting influence which

has caused AMD to always be crippled in OpenGL performance.

 

With Vulkan, AMD can equal or best Nvidia because they can force developers to write to their hardware directly.

AMD is mostly behind this Vulkan push. They started things with their "Close to Metal" GPGPU initiative then developed Mantle

then co-developed DX12.

 

All these moves are addressing two issues at AMD.

The aforementioned Nvidia control of the OpenGL standard and the fact that their driver teams are short staffed and less capable than Nvidia's.

Why hire more OpenGL driver devs when you can get engine developers to write their own drivers via the Vukan or DX12 API?

I can't fault it as a "strategy" but I'm not a fan of it as a standard. I guess PowerVR is pretty glad to see it too though. They also were

living in the shadow of Nvidia's voting power in the OpenGL ARB and were happy to have a way to bypass that too (and even happier to offload driver

development to engine devs).

[Jedibeeftrix]

Vulkan is the only future.

OpenGL drivers are going to get less and less attention from Intel, AMD and nVidia going forward.

So unless the hope is to run TDM over an OpenGL > Vulkan wrapper then it gets increasingly hard to see how any Intel Skylake and higher, AMD GCN 3rd and higher, Nvidia Kepler and higher get to play TDM (which is perhaps a perfectly valid strategy).

And if this is the strategy, then it's far enough in the future before OpenGL driver updates stagnate that it doesn't need to be worried about yet.

Edited April 3, 2020 by Jedibeeftrix

[stgatilov]

46 minutes ago, Jedibeeftrix said:

Vulkan is the only future.

OpenGL drivers are going to get less and less attention from Intel, AMD and nVidia going forward.

Hmm... you cannot imagine how much software works on OpenGL.
As for now, all those vendors would more likely drop support for Vulkan, calling it a "failure", than drop support for OpenGL

[HMart]

IMO Opengl will never go away (unless you are Apple), there millions of software in the wild that don't need the close to the metal performance of Vulkan, many indie games for example will be OpenGL for years to come, hardware vendors will be unhappy but they will have to still support OpenGL essentially forever, 20 years from now you will still be able to play OpenGL games of today, like you can still play directdraw games from the 90's, at the same time, they will be happy that OpenGL will stagnate at some point in the future and they will just need to maintain that version, instead of constantly having to adapt to new OpenGL.

nbohr1more imo Nvidia still has a huge influence over Vulkan, is not only the new Vulkan raytracing based on RTX but also Kronos president is from Nvidia.

Edited April 3, 2020 by HMart

[Betaversion]

@SNOWY

On 3/8/2020 at 11:32 AM, snowy said:

I think that instead of requesting more features and better tech, a better move forward would be simplification. Simplification of code, coding workflow and more accessible tools to create new content, especially for new content creators.

Yes!... THAT's the way to give this game a prosperous long life. I'm a (very rusty) amateur map-maker (Enemy Territory and Urban Terror) from years ago, and am just-now beginning to re-learn those skills... I want to be able to contribute material to the community. The KISS principle should always be kept in mind as a guiding rule for development.

Also, changing the subject... It would be so nice to have, as an option for map-making, multiplayer team CTF or other objective-based team-vs-team challenges. I think it would be hilarious (in a good way, both funny and fun) to be able to use the resources and 'feel' of TDM for massed team-vs-team combat/challenges.

Of course, that's for the (not too distant I hope) future of TDM development... But just think of it: Shadowy Steam Punk Goes to War!... I imagine there'd be a sizeable influx of players migrating from ET, UT and other Quake-based games.

Just a suggestion. Cheers everyone.

[madtaffer]

I don't understand engines as much so I apologize on rookie-speak in advance.

Could you update TDM engine to Quake-Spike (http://quakespasm.sourceforge.net/download.htm) ?

Quake has very small editor and maps are extreme small size. As an example all of Arcane dimensions maps weight about 330MB.  Performance and map loading is incredibly fast and smooth.

Big maps like The Painter’s Wife would have much smaller package and high performance in QSpike.

Edited January 31, 2021 by madtaffer

[HMart]

1 hour ago, madtaffer said:

I don't understand engines as much so I apologize on rookie-speak in advance.

Could you update TDM engine to Quake-Spike (http://quakespasm.sourceforge.net/download.htm) ?

Quake has very small editor and maps are extreme small size. As an example all of Arcane dimensions maps weight about 330MB.  Performance and map loading is incredibly fast and smooth.

Big maps like The Painter’s Wife would have much smaller package and high performance in QSpike.

For someone that claims not knowing much about engines, you seem to be very sure that QS is a silver bullet.

Just because you played something, on a engine that was faster than TDM engine that doesn't always mean said engine, is better or TDM engine is to blame for the slowness of the mission or even the slow mission, would run faster or load faster on QS, giving the same parameters and effects. The performance of a game/map/mission has many many factors beyond just the quality of the engine.

Btw the performance of a map has nothing to do with the size of the map file in MB. I can make a 5MB map run very slow, all depends on what you put on it.

Now about changing engines, I'm not from the TDM team but this has been asked before for UE4 and the answer was no, at this time is VERY improbable that the current TDM team, are willing to ditch decades of work and start from scratch in many stuff, just to change engines, is not a plug and play thing. 

Also TDM engine, is being constantly updated by the TDM team and imo is faster to improve a engine that you know deeply than to change the game to another one and have to learn a new engine internals from scratch.

Edited January 31, 2021 by HMart

[nbohr1more]

54 minutes ago, madtaffer said:

I don't understand engines as much so I apologize on rookie-speak in advance.

Could you update TDM engine to Quake-Spike (http://quakespasm.sourceforge.net/download.htm) ?

Quake has very small editor and maps are extreme small size. As an example all of Arcane dimensions maps weight about 330MB.  Performance and map loading is incredibly fast and smooth.

Big maps like The Painter’s Wife would have much smaller package and high performance in QSpike.

Old quake based engines perform well because they have baked lighting and quasi-2D path-finding and scripted physics.

Doom 3 / iDTech4 have real-time moving lights, real-time physics, and full volumetric path-finding.

There is currently some investigation regarding the addition of "Light Probes":

https://bugs.thedarkmod.com/view.php?id=5239

https://docs.unity3d.com/Manual/LightProbes.html

which would offer similar performance benefits as light-mapping for static lighting.

[madtaffer]

23 hours ago, HMart said:

Now about changing engines, I'm not from the TDM team but this has been asked before for UE4 and the answer was no, at this time is VERY improbable that the current TDM team, are willing to ditch decades of work and start from scratch in many stuff, just to change engines, is not a plug and play thing. Also TDM engine, is being constantly updated by the TDM team and imo is faster to improve a engine that you know deeply than to change the game to another one and have to learn a new engine internals from scratch.

I've miswritten. I didn't mean to change to new engine but adapt from QuakeSpazm  performance and low size maps  to the TDM.

Edited February 1, 2021 by madtaffer

[cabalistic]

13 minutes ago, nbohr1more said:

There is currently some investigation regarding the addition of "Light Probes":

https://bugs.thedarkmod.com/view.php?id=5239

https://docs.unity3d.com/Manual/LightProbes.html

which would offer similar performance benefits as light-mapping for static lighting.

Wait, no. That ticket is about reflection probes, which are meant to greatly simplify the setup of (parallax-corrected) cubemap reflections in the scene. They are not about lighting and have absolutely nothing to do with lightmaps. They also won't improve performance and would in fact cost a little if enabled, as it's a new render feature.

[nbohr1more]

5 minutes ago, cabalistic said:

Wait, no. That ticket is about reflection probes, which are meant to greatly simplify the setup of (parallax-corrected) cubemap reflections in the scene. They are not about lighting and have absolutely nothing to do with lightmaps. They also won't improve performance and would in fact cost a little if enabled, as it's a new render feature.

Oh drat. I was hoping we would extend the ambientCubicLight feature to use probes. Oh well.

[STiFU]

2 hours ago, madtaffer said:

I've miswritten. I didn't mean to change to new engine but adapt from QS  performance and low size maps  to the TDM.

The size of the Painter's Wife does not just come from the map format, but from the massive amount of custom assets and there is no way to decrease their size without sacrificing quality.

[Dragofer]

I think what drives most of the large FMs to be as voluminous as they are is that they contain custom assets which don't get used. Mappers might try out various ambients before choosing one, download texture packs and only use some of them, re-export the same model multiple times and so on. In such long projects, assets come and go, and I think more often than not they never go.

More major factors would be briefing videos, which can easily become gigantic if one isn't careful with the quality settings, and mappers not adhering 100% to conventions, i.e. converting as many .tga images as possible to more compact .dds or choosing a sensible resolution. Also, some compression programs (7z) seem to produce more compact zip archives than others (WinRar).

 

Ironically, The Painter's Wife has just now been undergoing optimisations and trimming to bring the file size down. The lowest hanging fruit included checking for images and sounds that were never included in a definition, looking for assets that already exist in core TDM and downscaling oversized custom textures. More intensive methods included loading the FM without the custom texture folders to see which textures the game actually wants to load, according to the console.

As a result, that particular FM has already been brought down from 550 MB to under 200 MB, and still has some ways to go, mostly just by cutting ballast. Loading times dont seem to have improved yet, but that might change once all the custom models are looked at (quite a few seem to have way too many vertices due to an exporter bug that was fixed in DR 2.9).

Down by the Riverside would be another example. Some alpha packages of that FM weighed 150 MB, but that was trimmed down to 56 MB for the beta and release (i.e. by checking which paintings out of a custom paintings pack are actually used).

So there's a surprising amount of potential for savings in FM sizes if an author is willing to put in the extra technical work of doing such trimming. Though I can understand if they'd rather just focus on polishing and releasing the mission after having already worked so long on it. In particular since there isn't really much in the way of tools to assist in the process.

[STiFU]

15 hours ago, Dragofer said:

I think what drives most of the large FMs to be as voluminous as they are is that they contain custom assets which don't get used. Mappers might try out various ambients before choosing one, download texture packs and only use some of them, re-export the same model multiple times and so on. In such long projects, assets come and go, and I think more often than not they never go.

More major factors would be briefing videos, which can easily become gigantic if one isn't careful with the quality settings, and mappers not adhering 100% to conventions, i.e. converting as many .tga images as possible to more compact .dds or choosing a sensible resolution. Also, some compression programs (7z) seem to produce more compact zip archives than others (WinRar).

 

Ironically, The Painter's Wife has just now been undergoing optimisations and trimming to bring the file size down. The lowest hanging fruit included checking for images and sounds that were never included in a definition, looking for assets that already exist in core TDM and downscaling oversized custom textures. More intensive methods included loading the FM without the custom texture folders to see which textures the game actually wants to load, according to the console.

As a result, that particular FM has already been brought down from 550 MB to under 200 MB, and still has some ways to go, mostly just by cutting ballast. Loading times dont seem to have improved yet, but that might change once all the custom models are looked at (quite a few seem to have way too many vertices due to an exporter bug that was fixed in DR 2.9).

Down by the Riverside would be another example. Some alpha packages of that FM weighed 150 MB, but that was trimmed down to 56 MB for the beta and release (i.e. by checking which paintings out of a custom paintings pack are actually used).

So there's a surprising amount of potential for savings in FM sizes if an author is willing to put in the extra technical work of doing such trimming. Though I can understand if they'd rather just focus on polishing and releasing the mission after having already worked so long on it. In particular since there isn't really much in the way of tools to assist in the process.

Checking for unreferenced assets in an FM package is something that could be achieved fairly easily with a python script. Checking for duplicates with core-mod would be a little more advanced in python, but of course also possible.

[OrbWeaver]

17 minutes ago, STiFU said:

Checking for unreferenced assets in an FM package is something that could be achieved fairly easily with a python script. Checking for duplicates with core-mod would be a little more advanced in python, but of course also possible.

The logical place for this would be either DR or the game itself, since both of these are already parsing the asset tree and know what they are loading or not loading in a particular map.

Doing it in a standalone script is of course possible, but then you have to start from scratch with parsing entity defs, MTR files and the like.

[STiFU]

1 hour ago, OrbWeaver said:

The logical place for this would be either DR or the game itself, since both of these are already parsing the asset tree and know what they are loading or not loading in a particular map.

Doing it in a standalone script is of course possible, but then you have to start from scratch with parsing entity defs, MTR files and the like.

I basically agree. However, if I remember correctly the filesystems of both DR and TDM do not actually know where a file came from, i.e., which pk4 (or mod folder), does it? 

Also, you usually don't have to write full parsers for applications like this. Here, it suffices to identify the name of an entity / material / whatever and delete its full definition if you cannot find that name referenced anywhere else. Repeat a couple of times and you got yourself a clean folder.

[OrbWeaver]

2 hours ago, STiFU said:

I basically agree. However, if I remember correctly the filesystems of both DR and TDM do not actually know where a file came from, i.e., which pk4 (or mod folder), does it? 

The location may not be exposed directly to the user, who does not need to care about which PK4 a particular image is in, but the underlying filesystem definitely has to keep track of this information, otherwise it would have no idea how to load resource files.

It's not enough for the engine to know that textures/darkmod/blah.tga is "somewhere in the PK4 structure". It has to know exactly which PK4 and exactly which byte offsets to use when extracting the image data from the ZIP.

2 hours ago, STiFU said:

Also, you usually don't have to write full parsers for applications like this. Here, it suffices to identify the name of an entity / material / whatever and delete its full definition if you cannot find that name referenced anywhere else. Repeat a couple of times and you got yourself a clean folder.

Maybe not a full parser, but you would need some kind of parsing at least, even if it's just keeping track of opening and closing braces so you know where the definition begins and ends (so you can remove it if needed). You'd also need some basic handling of tokens so you know that the string which follows "map" or "diffusemap" refers to an image, although regexes might be powerful enough for this part.

[madtaffer]

Since DarkRadiant 2.11 is released, will next TDM be 2.11 as well ?

[Amadeus]

32 minutes ago, madtaffer said:

Since DarkRadiant 2.11 is released, will next TDM be 2.11 as well ?

Next TDM will be 2.09

[demagogue]

Which is to say, even in "retirement" greebo (who codes DR) is still one of our most prolific workhorses.

[madtaffer]

What is the roadmap for Dark mod past 2.10+ in 2022? Is feature for easier streamlining assets and textures from Blender 3 to DR feasible?