Jump to content
The Dark Mod Forums

Connection to TDM with automation


Recommended Posts

Ah, thanks. Somehow, I did not yet stumble accross std::filesystem. The documentation also directly states, that it is derived from boost.filesystem.

The filesystem library was originally developed as boost.filesystem, was published as the technical specification ISO/IEC TS 18822:2015, and finally merged to ISO C++ as of C++17. The boost implementation is currently available on more compilers and platforms than the C++17 library.

Link to comment
Share on other sites

14 hours ago, greebo said:

Actually, I meant boost headers too. :) At least as far as the Windows compile is conerned, we don't need any boost to compile DR anymore.

After replacing the heavily-used helpers like boost::shared_ptr, boost::noncopyable, and the libs boost::thread and boost::filesystem, there weren't any references left except for a few string-manipulation functions and predicates. At this point I decided to bite the bullet and get rid of the entire boost dependencies - it didn't seem appropriate anymore to ship dozens of MB of headers just for those few string helpers.

Interesting, I didn't realise that.

I'd better check my recent changes, because I might have inadvertently added new minor usages of Boost without realising that the headers wouldn't be available on Windows.

Link to comment
Share on other sites

I'm trying to get camera location as GlobalCamera()->getActiveCamWnd()->get... .

Getting errors like this:

1>GameConnectionDialog.obj : error LNK2019: unresolved external symbol "private: __cdecl Matrix4::Matrix4(double,double,double,double,double,double,double,double,double,double,double,double,double,double,double,double)" (??0Matrix4@@AEAA@NNNNNNNNNNNNNNNN@Z) referenced in function "public: static class Matrix4 __cdecl Matrix4::byColumns(double,double,double,double,double,double,double,double,double,double,double,double,double,double,double,double)" (?byColumns@Matrix4@@SA?AV1@NNNNNNNNNNNNNNNN@Z)
1>GameConnectionDialog.obj : error LNK2019: unresolved external symbol "public: class BasicVector3<double> __cdecl ui::CamWnd::getCameraAngles(void)const " (?getCameraAngles@CamWnd@ui@@QEBA?AV?$BasicVector3@N@@XZ) referenced in function "public: void __cdecl GameConnection::UpdateCamera(void)" (?UpdateCamera@GameConnection@@QEAAXXZ)
1>GameConnectionDialog.obj : error LNK2019: unresolved external symbol "public: class std::shared_ptr<class ui::CamWnd> __cdecl ui::GlobalCameraManager::getActiveCamWnd(void)" (?getActiveCamWnd@GlobalCameraManager@ui@@QEAA?AV?$shared_ptr@VCamWnd@ui@@@std@@XZ) referenced in function "public: void __cdecl GameConnection::UpdateCamera(void)" (?UpdateCamera@GameConnection@@QEAAXXZ)
1>GameConnectionDialog.obj : error LNK2019: unresolved external symbol "public: void __cdecl ui::GlobalCameraManager::addCameraObserver(class CameraObserver *)" (?addCameraObserver@GlobalCameraManager@ui@@QEAAXPEAVCameraObserver@@@Z) referenced in function "public: static void __cdecl GameConnection::EnableCameraSync(class std::vector<class cmd::Argument,class std::allocator<class cmd::Argument> > const &)" (?EnableCameraSync@GameConnection@@SAXAEBV?$vector@VArgument@cmd@@V?$allocator@VArgument@cmd@@@std@@@std@@@Z)
1>GameConnectionDialog.obj : error LNK2019: unresolved external symbol "public: void __cdecl ui::GlobalCameraManager::removeCameraObserver(class CameraObserver *)" (?removeCameraObserver@GlobalCameraManager@ui@@QEAAXPEAVCameraObserver@@@Z) referenced in function "public: static void __cdecl GameConnection::DisableCameraSync(class std::vector<class cmd::Argument,class std::allocator<class cmd::Argument> > const &)" (?DisableCameraSync@GameConnection@@SAXAEBV?$vector@VArgument@cmd@@V?$allocator@VArgument@cmd@@@std@@@std@@@Z)
1>GameConnectionDialog.obj : error LNK2019: unresolved external symbol "class ui::GlobalCameraManager & __cdecl GlobalCamera(void)" (?GlobalCamera@@YAAEAVGlobalCameraManager@ui@@XZ) referenced in function "public: void __cdecl GameConnection::UpdateCamera(void)" (?UpdateCamera@GameConnection@@QEAAXXZ)

I have a feeling that some parts are not opened for plugins yet.

What's the right way to listen for DR camera movements and get position/angles?

Link to comment
Share on other sites

Symbols like Matrix4 are in the math library, which you can add to your link options.

However, GlobalCameraManager is only available within the radiant binary, so your initial assessment is correct — this part is not available for plugins. Historically there has not been any use of external plugins that interface tightly within the main UI (as opposed to adding new dialogs which plugins like Objectives and Stim/Response already do), which is why the CameraManager is not exposed in this way, although it is already a module with a defined interface so it could be refactored if necessary.

For now though, it might be simpler just to write your prototype code as part of the main radiant binary rather than as a separate DLL. It could always be separated out later if there was a need.

Link to comment
Share on other sites

This is how it looks right now.
Also attached the code of main class if anyone is interested.

 

One nuisance is that reloading a map works only after the map is saved. So when mapper moves something, he has to 1) save the map, and 2) click "reload map" button. It is of course possible to send reloadMap command automatically on save, removing one click. Also, reloading map from scratch takes a second or two for a large map (saving probably takes longer).

But I think it would be better if hot-reloading was possible without saving map to disk. In order to achieve that, I need the following features:

  1. Ability to detect which entities has changed (since some mark in the past). Which were removed, which were added, which were modified (it's OK to overestimate the last list).
  2. Ability to save every entity from selected set into in-memory text in .map format (spawnargs, brushes, patches --- all exactly as in .map file).
  3. Probably also some event which signals when an entity has changed.
  4. Worldspawn is out of question yet. But in future it would be necessary to do pretty the same things for world brushes and patches.

 

 

GameConnection.cpp GameConnection.h

  • Like 4
Link to comment
Share on other sites

That actually looks very cool. I was originally assuming you were just going to add an extra button along the lines of "Reload this map in the game", but you've gone all the way to full bidirectional camera synchronisation in real-time which could be extremely useful (and help get around the drawbacks of DR's very limited rendering).

I think there could be some improvements to the UI but that's something we can handle on the DR side.

I would be interested in testing your changes, but I'm not sure how to integrate them with just the new files attached to the forum post (or is it literally just a case of adding them to the build and everything happens automatically?). If changes outside of those files are needed, would you be able to produce a patch using git-format-patch which should cover everything and is very easy for other Git users to apply?

  • Thanks 1
Link to comment
Share on other sites

This looks very cool already. And yeah, reloading map upon saving is probably a good idea, or maybe having a switch / checkbox for that, if mapper wants to control it manually in some cases. This should work even better with for two-monitor setup.

Link to comment
Share on other sites

16 hours ago, OrbWeaver said:

but you've gone all the way to full bidirectional camera synchronisation in real-time which could be extremely useful

It is only DR -> TDM now, but I can definitely add TDM -> DR synchronization using polling. I perhaps it would be enough to have just one button "Sync DR Camera to TDM" for reverse direction?

Quote

I think there could be some improvements to the UI but that's something we can handle on the DR side.

I would be very happy if you take over the code and implement proper GUI. I'm using menu buttons now because that's what plugins had and that's what I quickly made working.

To be honest, I have never intended to do everything "right" in DR. My plan is to implement the underlying system and most important features on top of it, then "surrender" for code review and ask your help in writing the GUI 🥺

Quote

I would be interested in testing your changes, but I'm not sure how to integrate them with just the new files attached to the forum post (or is it literally just a case of adding them to the build and everything happens automatically?). If changes outside of those files are needed, would you be able to produce a patch using git-format-patch which should cover everything and is very easy for other Git users to apply?

The plan was to provide a Pull Request eventually.
Providing changes manually is not very easy, because there is also ZeroMQ addition, which BTW I have only did on Windows x64 platform (and not even committed binaries). If you want to see current state, I can push it to fork.

Link to comment
Share on other sites

1 hour ago, stgatilov said:

It is only DR -> TDM now, but I can definitely add TDM -> DR synchronization using polling. I perhaps it would be enough to have just one button "Sync DR Camera to TDM" for reverse direction?

That would be ideal I think. A toggle setting for "DR camera follows player position" plus a one-shot button for synchronisation in the other direction.

1 hour ago, stgatilov said:

I would be very happy if you take over the code and implement proper GUI. I'm using menu buttons now because that's what plugins had and that's what I quickly made working. To be honest, I have never intended to do everything "right" in DR. My plan is to implement the underlying system and most important features on top of it, then "surrender" for code review and ask your help in writing the GUI 🥺

That certainly makes sense. There's not much point in you spending hours tweaking wxWidgets layouts in DarkRadiant, when your skills are more needed elsewhere in the mod.

1 hour ago, stgatilov said:

The plan was to provide a Pull Request eventually.
Providing changes manually is not very easy, because there is also ZeroMQ addition, which BTW I have only did on Windows x64 platform (and not even committed binaries). If you want to see current state, I can push it to fork.

Ah OK, I didn't realise the implementation was still in progress. I'll wait for the eventual pull request rather than trying to preemptively integrate things now.

Link to comment
Share on other sites

Could you provide any pointers about the questions I posted above?

Basically, I'd like to track entity changes as they happen, and be able to write only entities which has changed. Is it possible?
Is there anything like "time of last modification" in entities?

The idea is to implement a mode which would immediately apply changes from DR to TDM. So that when mapper changes e.g. light radius, the change is immediately transferred to TDM and applied there.

Link to comment
Share on other sites

11 hours ago, stgatilov said:

Basically, I'd like to track entity changes as they happen, and be able to write only entities which has changed. Is it possible?
Is there anything like "time of last modification" in entities?

The idea is to implement a mode which would immediately apply changes from DR to TDM. So that when mapper changes e.g. light radius, the change is immediately transferred to TDM and applied there.

There isn't anything exactly equivalent to a last modification time, but you can listen for modifications to entity key values by constructing an Entity::Observer and attaching it to an Entity with Entity::attachObserver(). However, this will only give you changes to key values on a single entity. While you could theoretically attach an observer to every entity in the map, this would be an O(n) quantity of observers and callbacks and would probably cause performance problems in large maps.

If you're not looking for a notification mechanism but just a way to determine what has changed (effectively a "diff" operation on the entire map's entities), I can't immediately think of a way to do that efficiently in the current codebase, since it's not a problem DarkRadiant has ever needed to solve before.

  • Thanks 1
Link to comment
Share on other sites

14 hours ago, OrbWeaver said:

There isn't anything exactly equivalent to a last modification time, but you can listen for modifications to entity key values by constructing an Entity::Observer and attaching it to an Entity with Entity::attachObserver(). However, this will only give you changes to key values on a single entity. While you could theoretically attach an observer to every entity in the map, this would be an O(n) quantity of observers and callbacks and would probably cause performance problems in large maps.

It won't be a problem. There are at most 8K entities. Even if they are modified all at once, user won't notice it.

Some event after adding entity and before removing entity (or after, but with entity name available) is also needed.

And the hardest thing: write down only specific subset of entities to a in-memory map file. Although a physical file would probably be OK too.

Link to comment
Share on other sites

I think I made it. Will post video in nearest days.

 

I used scene::Graph::Observer with auto-attached Entity::Observer-s to learn about changes to entities.

Then I added customized "map save" in order to get textual representation of only changed entities. More precisely:

  • Added traverseSubset similar to traverseSelected, but allows to explicitly specify the subset.
  • Subclassed Doom3MapWriter with DiffDoom3MapWriter. The only important difference is that it adds header "add entity" / "remove entity" / "modify entity" before every written entity.
  • Added Map::saveMapDiff method, which creates in-memory stream, does some custom stuff, then runs MapExporter::exportMap.

It's probably better to wait with code discussion until PR...

 

Initially I tried to avoid any message-pump-like function, but I'm afraid I need it.
I will use WxWidgets timer to call my GameConnection::Think function periodically.

Link to comment
Share on other sites

I am thinking about some intermediate refactoring.
It turns out I am out of basic DR conventions related to methods naming, indents, namespaces, global variables.

@OrbWeaver, I'd like to ask some questions:

  1. The piece of code I'm writing is currently called "game connection" in DR. I wonder if it is OK or not? I'm going to use namespace gamecon for it.
  2. I need one global object (singleton) for the main class. Should I make it a RegisterableModule with GlobalGameConnection function? Right now it is plain global variable.
  3. If main class should be a registerable module, then should I create a separate interface class (aka IGameConnection)? Should I put it to include project, along with all interfaces?

 

Link to comment
Share on other sites

23 hours ago, stgatilov said:
  1. The piece of code I'm writing is currently called "game connection" in DR. I wonder if it is OK or not? I'm going to use namespace gamecon for it.

Good question. I suppose the feature needs some punchy user-friendly name — "sync" maybe, or "live mode"?

23 hours ago, stgatilov said:
  1. I need one global object (singleton) for the main class. Should I make it a RegisterableModule with GlobalGameConnection function? Right now it is plain global variable.

In general we avoid global variables. If the class needs to expose an interface to other parts of the application (e.g. to allow other classes to add things they wish to sync, or listen for updates), then making it a module is a good choice. If the functionality does not need to be exposed then the class could just be a local member of some other object (e.g. something camera-related).

If it is a module then it should be optional at compile time (since it is TDM-specific), which means the module retrieval function and calling code must be prepared for the module to be nullptr at runtime. Most existing modules are not optional but I believe the sound manager is, so you could use this as a template (basically it returns a pointer rather than a reference, and calling code checks for null before dereferencing it).

23 hours ago, stgatilov said:
  1. If main class should be a registerable module, then should I create a separate interface class (aka IGameConnection)? Should I put it to include project, along with all interfaces?

Yes, if it is a module then it must have an interface defined in a suitable include file within the include directory.

Link to comment
Share on other sites

2 hours ago, OrbWeaver said:

Good question. I suppose the feature needs some punchy user-friendly name — "sync" maybe, or "live mode"?

Most importantly, it is the name for the source code, so it does not need to be user-friendly. The name of menu tab can be changed at any moment.

Also, I believe "hot reload" a suitable fancy name for updating the game immediately when edited. Not for camera sync though...

Quote

In general we avoid global variables. If the class needs to expose an interface to other parts of the application (e.g. to allow other classes to add things they wish to sync, or listen for updates), then making it a module is a good choice. If the functionality does not need to be exposed then the class could just be a local member of some other object (e.g. something camera-related).

There is no need to access it from the outside now (except for attaching the GUI), but such need may arise in future.

Link to comment
Share on other sites

20 hours ago, stgatilov said:

Most importantly, it is the name for the source code, so it does not need to be user-friendly. The name of menu tab can be changed at any moment.

In any case, I think "sync" is a reasonable name: short, simple and I don't think we are using it for anything else at present.

20 hours ago, stgatilov said:

There is no need to access it from the outside now (except for attaching the GUI), but such need may arise in future.

In that case, making it a module is probably unnecessary. The defining test is the interface: what methods, if any, would you expose on the new module interface? If the answer is "none", then there is no need for a new module. Making it a member variable of some other object is probably the way to go, perhaps something in GUI or you could make it a member of the main Radiant module.

Link to comment
Share on other sites

11 minutes ago, OrbWeaver said:

In any case, I think "sync" is a reasonable name: short, simple and I don't think we are using it for anything else at present.

Well... I don't like this name.
It 1) does non clarify what is synced to what, and 2) some features have nothing to do with synchronization (pausing game, respawning entities?).

Quote

In that case, making it a module is probably unnecessary. The defining test is the interface: what methods, if any, would you expose on the new module interface? If the answer is "none", then there is no need for a new module.

Keep in mind that if you leave it without proper shutdown, it will keep alive socket, and maybe also have some code attached to a timer. Sooner or later someone will need to call at least "disconnect" method.

Link to comment
Share on other sites

1 hour ago, stgatilov said:

Well... I don't like this name.
It 1) does non clarify what is synced to what, and 2) some features have nothing to do with synchronization (pausing game, respawning entities?).

I was going to reply that the problem with gamecon is that it might be confused with the existing game namespace, but now I think about it, the game namespace might actually be a good place to put this. Currently it is used for simple things like extracting configuration information from the game-specific XML file, but logically your new code is also game-specific so having a class like game::Connection might make sense. Or if you need several different major classes, a sub-namespace like game::connection could be used.

1 hour ago, stgatilov said:

Keep in mind that if you leave it without proper shutdown, it will keep alive socket, and maybe also have some code attached to a timer. Sooner or later someone will need to call at least "disconnect" method.

What is the expected lifetime of the connection? Is it per-map, per DarkRadiant session, or something else? Do we have to do lifetime management tasks like monitoring for remote process shutdown, initiating a new connection after loading a new map, etc? These might influence the best location for the code (and whether it needs to be a module or not).

One advantage of making it a module is that it would be quite easy to factor it out into a separate DLL later (as you initially tried to do but were limited by the existing module APIs). It might then expose interface methods so that the Map module could check for the connection's existence and send updated map information when appropriate.

Link to comment
Share on other sites

1 hour ago, OrbWeaver said:

What is the expected lifetime of the connection? Is it per-map, per DarkRadiant session, or something else? Do we have to do lifetime management tasks like monitoring for remote process shutdown, initiating a new connection after loading a new map, etc? These might influence the best location for the code (and whether it needs to be a module or not).

Right now I don't manage TheDarkMod process. Unlike Python automation scripts, I don't even start TDM, because in C++ it is a wonderful journey to low-level OS-specific APIs. So the idea is that user manages TDM process himself: he has to start it, and therefore, he has to close it too. I guess it is OK to leave it like that, removing any need to manage process lifetime.

Speaking of socket connection, it is more complicated.
TDM polls socket every frame, and when the opposite side is down the socket is closed, so disconnection happens almost immediately.
Despite using only nonblocking network calls, DR currently hangs if TDM goes away. Because ZeroMQ has its own vision of how sockets should work, and detecting when other side is obviously down is not part of it. ZeroMQ is definitely not a socket library. The thing that is called "sockets" in it can only talk with ZeroMQ endpoints. It took me some time to find a mode which allows talking to raw sockets with it.
Anyway, the plan is to make DR work the same way: when TDM goes down, DR side disables everything in the main class, removes all observers, and kills connection. In the worst case I would use raw socket API or find another socket wrapper library to achieve it.

The TDM connection can survive changing the map and FM. But I guess DR side should close and disable everything when map is closed or loaded. It can reconnect later, but full cleanup is recommended anyway.

I don't think that it is necessary to manage loading map in TDM. Indeed, the Python automation scripts can do it pretty easily: they can change mod, enumerate mods, start game by clicking GUI, etc. So it is possible to do it in DR too, but right now there are more useful things to do. Live editing of brushes and scripts, for example 😉

Link to comment
Share on other sites

17 hours ago, stgatilov said:

Right now I don't manage TheDarkMod process. Unlike Python automation scripts, I don't even start TDM, because in C++ it is a wonderful journey to low-level OS-specific APIs. So the idea is that user manages TDM process himself: he has to start it, and therefore, he has to close it too. I guess it is OK to leave it like that, removing any need to manage process lifetime.

I agree. We don't want to have to manage starting and stopping the game itself, and shouldn't, because we don't know exactly how the game should be run (for example on Linux I sometimes use two shortcuts, one which starts the game fullscreen and another which starts it in a window so it can coexist with the editor).

17 hours ago, stgatilov said:

Speaking of socket connection, it is more complicated.
TDM polls socket every frame, and when the opposite side is down the socket is closed, so disconnection happens almost immediately.
Despite using only nonblocking network calls, DR currently hangs if TDM goes away. Because ZeroMQ has its own vision of how sockets should work, and detecting when other side is obviously down is not part of it. ZeroMQ is definitely not a socket library. The thing that is called "sockets" in it can only talk with ZeroMQ endpoints. It took me some time to find a mode which allows talking to raw sockets with it.
Anyway, the plan is to make DR work the same way: when TDM goes down, DR side disables everything in the main class, removes all observers, and kills connection. In the worst case I would use raw socket API or find another socket wrapper library to achieve it.

Makes me wonder if we are really gaining the benefit of ZeroMQ in this case. The idea behind ZMQ is that it provides a very rich and easy-to-use API for constructing advanced message passing topologies using its out-of-the-box functionality; if all we are using it for is to create raw sockets and send data to them directly, maybe ZMQ isn't the right choice (although I can't immediately think of a good alternative; I'm not that familiar with socket libraries).

17 hours ago, stgatilov said:

The TDM connection can survive changing the map and FM. But I guess DR side should close and disable everything when map is closed or loaded. It can reconnect later, but full cleanup is recommended anyway.

Right, map close/change seems like a good time to do cleanup, because the new map might not be the one loaded into the game at which point any synchronisation will be meaningless or buggy. Is there some sanity check whereby DR can be sure that it is connecting to a running game which is actually showing the same map, or do we just connect and hope for the best? Perhaps the initial handshake protocol over the socket should exchange the map filename so both ends are sure they are looking at the same data.

17 hours ago, stgatilov said:

I don't think that it is necessary to manage loading map in TDM. Indeed, the Python automation scripts can do it pretty easily: they can change mod, enumerate mods, start game by clicking GUI, etc. So it is possible to do it in DR too, but right now there are more useful things to do. Live editing of brushes and scripts, for example 😉

I could see a use for a feature of "If the game isn't running the current map, tell it to switch" but you're right, it's definitely not a critical feature.

Link to comment
Share on other sites

2 hours ago, OrbWeaver said:

Makes me wonder if we are really gaining the benefit of ZeroMQ in this case. The idea behind ZMQ is that it provides a very rich and easy-to-use API for constructing advanced message passing topologies using its out-of-the-box functionality; if all we are using it for is to create raw sockets and send data to them directly, maybe ZMQ isn't the right choice (although I can't immediately think of a good alternative; I'm not that familiar with socket libraries).

Not at all.

I need a raw socket. Just some OS-independent wrapper which makes it easy to connect to conventional representation of network address, maybe even without DNS and domain lookup (I only need localhost). And which makes it easy to listen for TCP connection on specific post.

The problem is that most of the libraries inevitably attempt to do more than that.

Link to comment
Share on other sites

Sounds like what we need then is some kind of "tiny sockets library" rather than a rich API like ZeroMQ.

There's this hideous Geocities-like page with a library that looks like it hasn't been touched since 2009 and might be too lowlevel for our needs, but maybe it's still usable: Simple Sockets Library

Something slightly more modern and available on GitHub: SimpleSockets

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recent Status Updates

    • Ansome

      Finally got my PC back from the shop after my SSD got corrupted a week ago and damaged my motherboard. Scary stuff, but thank goodness it happened right after two months of FM development instead of wiping all my work before I could release it. New SSD, repaired Motherboard and BIOS, and we're ready to start working on my second FM with some added version control in the cloud just to be safe!
      · 0 replies
    • Petike the Taffer  »  DeTeEff

      I've updated the articles for your FMs and your author category at the wiki. Your newer nickname (DeTeEff) now comes first, and the one in parentheses is your older nickname (Fieldmedic). Just to avoid confusing people who played your FMs years ago and remember your older nickname. I've added a wiki article for your latest FM, Who Watches the Watcher?, as part of my current updating efforts. Unless I overlooked something, you have five different FMs so far.
      · 0 replies
    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
×
×
  • Create New...