Jump to content
The Dark Mod Forums

Idea: Collaborative online editing with realtime map synchronization


Recommended Posts

This isn't a feature I personally plan on using as I like to work alone. However I wanted to bring this up as I believe it would help with mission creation and more FM's being developed faster, particularly for people working on FM's in teams rather than alone. I'm imagining a relatively simple system, albeit it may require bigger code changes.

Think of it as multiplayer with a server list but for the map editor: DR instances would have the ability to connect to other DR instances (or different server software) over the internet. Upon connecting the map in cause is cloned locally and loaded. The maps are synced between actively connected instances and any change made on any of the connected DR's is mirrored to the others in realtime, with the local map file being updated on each end as this happens. When an user selects or moves a brush or entity on one instance, it's highlighted and repositioned on all others as fast as the network allows... if done right a DR user should see map components moving on their own in the viewports as they would if being dragged by the local user.

I'm imagining that such an ability could bring FM creation to a new level; Creators of a FM can connect to a central server storing that FM, ideally up for connectivity 24/7... each author can then make changes whenever they feel like working on a given part of the map, with everyone else seeing those changes in realtime (if connected) or when they connect to the project later. We could have larger and more complex FM's as a result! What do you think, and would this be possible without too much development effort?

Trivia: This idea was inspired by the map editor in the Cube / Tesseract engine which is what the popular FPS Red Eclipse uses. It's not just integrated in-game requiring no separate editor at all, but people can start an ordinary multiplayer match to create a map instead of playing Deathmatch. This got me thinking about the possibilities of realtime collaborative map editing, especially for something more complex as TDM FM's where this could accelerate things quite a bit.

Edited by MirceaKitsune
Link to comment
Share on other sites

In principle, there are three known approaches to collaborate on a project:

  1. Lock + modify + unlock. When someone decides to change a file, he locks it, then changes it, uploads a new version, and unlocks it. If file is already locked, then you have to wait before editing it. Or you can learn who holds the lock and negotiate with him.
  2. Copy + modify + merge. When you want to modify a file, you get a local copy of it, change it, then try to upload/commit the changes. If several people have modified the same file in parallel, then the whoever did it last has to merge all changes into single consistent state. Merging is usually possible only for text documents: sometimes it happens automatically (if the changes are independent), sometimes it results in conflicts that people have to resolve manually.
  3. Realtime editing. The document is located on the central server, and everyone can edit it in realtime. Basically, that's how Google Docs work. I'd say internally it is lock+modify+unlock approach, but with very small changes and fast updates.

@MirceaKitsune asks for approach 3. I guess it would require tons of work in DarkRadiant.

Approach 2 can be achieved by using some VCS like SVN, mercurial, git, whatever. But it requires good merging in order to work properly. Text assets like scripts, xdata, defs, materials should merge fine with built-in VCS tools. Binary assets like images, models, videos, are completely unmergeable: for them approach 2 is a complete failure. As far as I understand, @greebo wants to improve merging for map files, which are supposedly the most edited files.

Approach 1 can be achieved by storing FM on cloud disk (e.g. Google Drive) and establishing some discipline. Like posting a message "I locked it" on forum, or in a text file near the FM on Google Drive. SVN also supports file locking, so it can be used for unmergeable assets only and on per-file basis. Git also has extension for file locking, as far as I remember.

Link to comment
Share on other sites

Ok, let's dream a bit 😌

 

Suppose that we have one server, which stores the map in something like SVN repo: initial revision, and a sequence of patches.
The server is going to work as the only source of truth about map state.

Suppose that all assets are stored on a mounted network disk. As far as I understand, google drive can be mounted to local disk.
I guess such a simple approach is enough.

Suppose that we have a framework for map diffs/patches with following properties:

  • It is enough to find diffs on per-entity and per-primitive level, so that a change in entity spawnargs stores all spawnargs in the updated entity, and a change in primitive stores the full text of the updated primitive. No need to go on the level of "add/remove/modify a spawnarg".
  • A modification inside patch should store both the old and the new state. Simultaneously changing spawnargs on the same entity or changing the same primitive results in a conflict, which can be detected reliably by looking at patches.
  • Sequence of compatible patches can be quickly squashed together --- might be convenient in some cases. They can also be reversed easily.
  • It is possible to quickly save a diff between current DR state and the previous such moment. I guess that's what hot reload already does, except that perhaps it is too careless with primitives now.
  • It is possible to quickly apply a patch, as long as it does not result in a conflict (conflict happens e.g. when we modify entity spawnargs which are different from the old state stored in the patch).

Since DarkRadiant is a complex thing, applying patches cannot happen at arbitrary moment. If we have a dialog opened, and we apply some patch which adds/removes some item from this dialog, that could easily result in a crash. Cleaning all the code for such cases is too unreliable and too much work. It is much easier to simply say that patches can only be applied when no dialogs are active, no edit box is active, etc.

Now in order to turn DarkRadiant into a multiplayer game, we only need to invent some protocol 😁

 

The server simply receives all patches from all clients and commits them to its repository in order or arrival (creating new revisions). Every diff should have unique identifier (not just hash of content, but identifier of client + sequence number of diff on his side). If applying a patch results in a conflict, then new revision is still created, but it is marked with "rejected" flag and it does not change anything (empty diff). Storing rejected revisions is necessary in order for the client to know that his patch was rejected.

Also, the server remembers which was the last revision sent to each client. It actively sends patches of newer revisions to the clients.

The client has a persistent thread which communicates with server in an endless loop. It has a queue of outgoing patches, which it sends to server sequentally, and a queue of incoming patches, which it populates as it receives more data from server. This thread never does anything else: the main thread interacts with the queues when it is ready.

The main thread of the client works mostly as usual. When user changes something, we modify all the usual data + add a patch into some patch queue.
Note that this is different from the outgoing patch queue in communication thread. Also, this queue is divided into two sections: some beginning of the queue is "already sent to server", and some tail is "not sent to server yet". When user changes something, we compute diff and add a patch to the "yet-unsent" tail of the queue.

We establish a "sync point" when 1) no dialog is open (i.e. DR is ready to be updated), and 2) patch queue is not empty or 5 seconds has passed since previous sync point.
The sync point is going to be complicated:

  1. Pull all the received ingoing patches from the communication thread.
  2. Compute the patch from current state to the latest state received from the server. Take our patch queue and reverse it, then concatenate ingoing patches, finally squash the whole sequence into one patch. Apply this patch to the current DR state, and you get server revision. That's the most critical point, since all data structures in DR must be updated properly.
  3. Look which of the "already sent" patches in our queue are contained among the ingoing patches. These ones were already incorporated on the server, so we drop them out from our queue. If some of our patches were committed but rejected, we should quickly notify user about it (change dropped).
  4. Go sequentally over the remaining patches in our queue, and apply them to the current state of DR, updating all structures again. If some patch cannot be applied due to conflict, then we drop it and quickly notify user. The already-sent patches remain already-sent, the not-yet-sent patches remain not-yet-sent.
  5. Copy all not-yet-sent patches into the outgoing patch queue of the communication thread, and mark them as "already sent".

Of course, when users try to edit the same thing, someone's changes are dropped. That's not fun, but should be not a big problem for fast modifications. The real problem is with slow modifications, like editing a readable: user can edit it for many minutes, during which no sync points happen. If someone else edits the same entity during this time, then all his modifications will be lost when he finally clicks OK.
I think this problem can be gradually mitigated by adding locks (on entity/primitive) into the protocol: when user starts some potentially long editing operation, the affected entity is locked, and the lock is sent to server, which broadcasts it to all the other clients. If client knows that entity is locked, it forbids both locking it with some dialog and editing it directly. Note that synchronization of locks does not require any consistency in time: it can be achieved by sending entity names as fast as possible without some complex stuff like queues, sync points, etc.

The system sounds like a lot of fun. Indeed, things will often break, in which case DR will crash, and map will become unreadable. Instead of trying to avoid it at all costs, it is much more important to ensure that server stores the whole history, and provides some recovery tools like "export map at revision which is 5 minutes younger than the latest one".

I'm pretty sure starting/ending a multiplayer session would be a torture too. Blocking or adjusting behavior of File menu commands is not obvious too.

 

UPDATE: added illustration:

online_editing_sketch.thumb.png.004b130da2362058096aada024e638ba.png

Link to comment
Share on other sites

Going to look at the other thread as well, thanks for sharing. Letting DR save and load maps over Git / SVN would be a simpler version of this idea, perhaps even a precursor to it. But it does come with issues: Multiple mappers can't easily work on the same FM at the same time! Here's a more in depth analysis on how I see that working out:

The biggest issue is that the moment two or more creators edit the same item, there would be a conflict once the changes are faced with each other and have to be reconciled. We could teach DR how to resolve map conflicts and pick one set of changes, but this means one author's version would be chosen while others have to lose their work. The problem doesn't even end there: Some changes can be reconciled in theory but still lead to inconsistencies in practice! Say one author moves the walls of a building while another handles its furniture entities; DR can agree on which set of changes to keep for each of those movements and everything would seem resolved to the code, however if they aren't made or solved together you're left with an empty house in one place and floating furniture in the other, which a person has to later notice and manually resolve.

There may be ways to reduce this risk. One is attributing specific groups of brushes and entities to different creators: Only the person who spawned an item may edit it, no changes to other people's stuff allowed. This way anyone can push map changes to a Git repository at any point regardless of what other developers may have done during that time, the differences can be merged under any circumstance as no conflicts can occur. But it's a limiting approach since you can't help another mapper with their part of the map, while on the other side they can build over what you're doing and you couldn't get their stuff out of the way.

A good way to take this approach would be allowing maps to be stored in multiple files during development, so changes don't need to be made to the same file altogether. This means that alongside the main myfm.map you can have myfm.bob.map + myfm.mike.map + myfm.alice.map and so on, when loading the main map content is also loaded from those additional files: Now you can push and pull the git repo at your own leisure, your work is always up to date locally while changes made by others become visible whenever you do an update and reload the map! This alone is unlikely to be noticed or used by many creators though, unless encouraged by additional Git tools in the interface; DR could support pushing and pulling changes directly from the toolbar, with an option to automatically update other authors files at an interval (say 5 minutes by default) then refresh the map seamlessly.

While this would all be fun and easier for starters, the holy grail would be realtime editing at the end of the day. That way anyone can work on anything and always see the updated result on their end during all stages... without having to fear emerging conflicts that need to be reconciled later, unless someone's connection goes down and they make changes during that time which should probably be disallowed by the code. If designed properly we could have a powerful and unique system to work with here.

  • Haha 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recent Status Updates

    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
    • OrbWeaver

      I like the new frob highlight but it would nice if it was less "flickery" while moving over objects (especially barred metal doors).
      · 4 replies
    • nbohr1more

      Please vote in the 15th Anniversary Contest Theme Poll
       
      · 0 replies
×
×
  • Create New...