Shared dmapping binary

[stgatilov]

On 3/29/2022 at 6:09 PM, greebo said:

Speaking about sharing code, what would be really nice would be to have a plugin containing the DMAP algorithm. But from what I remember when trying to port this over to DarkRadiant years ago, that code is also tied to the decl system and the materials and even image loading. Maybe @stgatilov, having worked on dmap recently, might share some insight on whether this piece of code would be feasible to isolate and move to a DLL with a nice interface.

I think dmap uses materials/decls/images only because materials are assigned to geometry.
In fact, it does not need actual images, it only needs material flags to determine if material is opaque or not.

Of course, you will have to drag along the whole idlib, and probably all the common/sys classes as well: idList-s are now all over the place, and various common->Printf/Warning are used a lot too.
By the way, recently I merged idlib code into TDM project... and most likely it has to be extracted back in order to be used in DarkRadiant.

On the other hand, won't moving dmap to DarkRadiant cause more problems than if would solve?

For instance, dmap sometimes is changed. One such change during 2.08 was quite breaking. Dmap from TDM is not compatible with dmap from Doom 3 and vice versa (a map which compiles on one game can fail with leak on the other one). How would mappers understand which version they use? What if dmap changes are tied to the changes in the rest of the game?

Quote

Having leak detection and portalisation code available to DarkRadiant would be beneficial for renderer performance too. Right now, it's completely unportalised and slow as heck.

Extracting renderer frontend would be much harder I suppose.
Without it, you can just read .proc files and implement your own areas/portals pruning code.

[greebo]

11 minutes ago, stgatilov said:

On the other hand, won't moving dmap to DarkRadiant cause more problems than if would solve?

For instance, dmap sometimes is changed. One such change during 2.08 was quite breaking. Dmap from TDM is not compatible with dmap from Doom 3 and vice versa

I didn't mean to suggest to move or migrate dmapping code into DR (I already failed once doing so), and you're correct that keeping DR up to date with any dmap changes would be most cumbersome. That's why I just meant to load a dmap.dll module and pump data into it, which would make it spit out some portalling or proc information. Pretty much like the game is using the Maya SDK. It just needs to load the DLL and feed the right data format into it to be able to use it.

[OrbWeaver]

I'd probably approach from a slightly different direction: rather than trying to extract and isolate small parts of the TDM code and call these in a DLL from both the game and DR (which introduces problems with dependencies on other parts of the code), I would try adding a DLL-style build mode for the whole game binary — perhaps chosen with a CMake option — so that you could choose to build either the game itself or a DLL containing most of the same code.

You'd then need a suitable DLL interface on the game side, which I would suggest should be pure C and as simple as possible so that it isn't necessary to expose all of the idLib stuff and deal with the complexities of C++ binary interfaces. So you might end up with an interface a bit like original GTK:

struct TDMInstallation; // opaque type

// Initialise new installation and return object owned by the DLL
TDMInstallation* tdm_installation_new(const char* path);

// Compile a given map, return a status code
int tdm_installation_compile_map(TDMInstallation* installation, const char* mapName);

// Properly dispose of the installation object
void tdm_installation_free(TDMInstallation* installation);

This way you effectively have full encapsulation of the DLL code, and an essentially object-oriented interface using C functions and opaque pointers instead of C++ classes with private members and public methods.

If we were ever going to try this I'd suggest starting with something very simple and self contained. Compiling maps is probably OK, or maybe exposing Tels' i18n system which has never been ported into DR and results in DR not being able to show internationalised names for difficulty settings. Rendering of course would be a much more difficult task.

[stgatilov]

1 hour ago, OrbWeaver said:

I'd probably approach from a slightly different direction: rather than trying to extract and isolate small parts of the TDM code and call these in a DLL from both the game and DR (which introduces problems with dependencies on other parts of the code), I would try adding a DLL-style build mode for the whole game binary — perhaps chosen with a CMake option — so that you could choose to build either the game itself or a DLL containing most of the same code.

One of the problems with DLL-based build is having to use dynamic CRT (recall the nightmare of two separate CRTs when Doom 3 was closed source).

In this case we have to do one of:

1. Put CRT dlls into game directory. This was not an option before 32-bit deprecation because it did not allow 32-bit and 64-bit executables to coexist in the same directory. And changing paths conventions in TDM is something I'd better not attempt
2. Run VCRedist installer on player's machine. I tried doing that but it did not work well... Maybe because I wanted to avoid running the installer when it's already installed, and I did not manage to properly check if compatible version is installed. Anyway, there were a lot of complaints about this thing, so I simply returned full-static build.

Maybe it is possible to have an option (CMake-only) to build something dynamic, but that option would probably get broken regularly, because everyone would forget about it.

Quote

You'd then need a suitable DLL interface on the game side, which I would suggest should be pure C and as simple as possible so that it isn't necessary to expose all of the idLib stuff and deal with the complexities of C++ binary interfaces.

Yes, it makes sense to keep interface pure-C. You won't be able to dynamically load exported C++ function because of name mangling anyway. But are you sure you will be able to expose much stuff without using complex types?

By the way, it is not necessary to create any kind of DLL to do that. In principle, DarkRadiant can import the EXE file as module and use its exported functions! After all, entry point is the only difference between DLL and EXE. Of course, doing so would mean mapping 18 MB file onto DR's virtual memory, but I don't think it is a big problem (well, maybe code cache would suffer because relevant TDM functions would be less localized).

Quote

If we were ever going to try this I'd suggest starting with something very simple and self contained. Compiling maps is probably OK,

Running dmap is now available with the "game connection" feature. Of course, it is more messy and fragile than DLL call, but "game connection" needs to exist anyway (and dmapping in game connection is a bit faster).

Quote

or maybe exposing Tels' i18n system which has never been ported into DR and results in DR not being able to show internationalised names for difficulty settings. Rendering of course would be a much more difficult task.

Ehm... To be honest... we discussed that we should probably reimplement most of this system
But I agree that it would be very hard to force myself doing that

It seems that writing English full-text in translation files instead of relying on these #str_XXX keywords would be more comfortable for mappers in 99% of the cases, and additional "labels" added to the original text would resolve ambiguity in the rest 1%.

[stgatilov]

Ok, it seems that while using EXE as DLL is possible, some pretty artificial issues make this path very hacky:
https://www.codeproject.com/Articles/1045674/Load-EXE-as-DLL-Mission-Possible

And adding optional build with DLLs but deploying fully static build would mean that you'll have to distribute this DLL yourself, i.e. you cannot load this DLL from user's TDM installation.

[OrbWeaver]

20 hours ago, stgatilov said:

One of the problems with DLL-based build is having to use dynamic CRT (recall the nightmare of two separate CRTs when Doom 3 was closed source).

Maybe it is possible to have an option (CMake-only) to build something dynamic, but that option would probably get broken regularly, because everyone would forget about it.

Right, I'm assuming it would need to be a completely different build mode set via CMake, which could then set whatever different options were necessary with regard to static vs dynamic CRT or other dependencies.

20 hours ago, stgatilov said:

Yes, it makes sense to keep interface pure-C. You won't be able to dynamically load exported C++ function because of name mangling anyway. But are you sure you will be able to expose much stuff without using complex types?

In theory I think it should be possible to expose pretty much anything this way — the interface might be a bit more cumbersome to use, but you can always provide a header file with convenient (but optional) C++ wrapper classes which implement more familiar RAII and object-based semantics.

Even lists and maps can be exposed, e.g.

struct TDMStringList; // opaque

// Get list of maps
TDMStringList* tdm_installation_get_map_list(TDMInstallation* inst);

// Manipulate list
int tdm_stringlist_get_item_count(TDMStringList* list);
const char* tdm_stringlist_get_item(TDMStringList* list, int index);
void tdm_stringlist_free(TDMStringList* list);

Obviously I wouldn't want to write code in this style all day, but using it just to traverse a DLL boundary and possibly wrapped in some C++ helper classes would be manageable.

20 hours ago, stgatilov said:

By the way, it is not necessary to create any kind of DLL to do that. In principle, DarkRadiant can import the EXE file as module and use its exported functions! After all, entry point is the only difference between DLL and EXE. Of course, doing so would mean mapping 18 MB file onto DR's virtual memory, but I don't think it is a big problem (well, maybe code cache would suffer because relevant TDM functions would be less localized).

Apparently that's not trivial on Linux: https://stackoverflow.com/questions/6617679/using-dlopen-on-an-executable

If you want to use the executable as a library it seems you need to use the PIE (position-independent executable) compiler option(s), at which point you might as well just build a DLL anyway (unless the binary is always going to be build PIE).

20 hours ago, stgatilov said:

Ehm... To be honest... we discussed that we should probably reimplement most of this system
But I agree that it would be very hard to force myself doing that

It seems that writing English full-text in translation files instead of relying on these #str_XXX keywords would be more comfortable for mappers in 99% of the cases, and additional "labels" added to the original text would resolve ambiguity in the rest 1%.

No disagreement from me there. It seems like a hacky system with little regard to best practices for translation, which is why I've never made any effort to replicate it in DR.

20 hours ago, stgatilov said:

And adding optional build with DLLs but deploying fully static build would mean that you'll have to distribute this DLL yourself, i.e. you cannot load this DLL from user's TDM installation.

Sure, I wouldn't expect the DLL to already be there, it would be something we have to integrate ourselves. Although at that point it might be better to skip the DLL altogether and just do source code or static library integration.