Jump to content
The Dark Mod Forums

Mapping and Version Control


greebo

Recommended Posts

I'd be interested in hearing about how many of our fellow mappers...

  • ...are familiar with version control (Git)?
  • ...are or have been using version control for their mapping projects?
  • ...would be interested in using version control for their mapping projects?
  • ...are or have been mapping alone, using version control?
  • ...are or have been mapping in a team, using version control to collaborate?

I know that frost_salamander used a Github repo for hosting the hare map, and angua and me are using a private one on Gitlab. We discussed a few features that could support common collaboration workflows in DarkRadiant, but I'd like to hear about others first.

  • Like 1
Link to comment
Share on other sites

Thanks for raising this. I think this is a very important topic, and could save mappers a lot of frustration. Reading about mappers occasionally losing all of their work always made me confused, since it ought to be an easily avoided problem, but I guess non-programmers don't naturally gravitate towards version control systems in the way that we do.

Even if VCS features don't become part of DarkRadiant itself, a basic how-to guide on the wiki might be a helpful starting point.

Link to comment
Share on other sites

We've used the betamapper repo from TDM's SVN for developing The Painter's Wife. It was excellent, mainly because it made it much more efficient to synchronise everyone and their input. Could have one person working on the map, another clearing out unused assets and another proofreading readables simultaneously, rather than passing a .pk4 from person to person or trying to make differential update packages.

A problem still was if multiple people wanted to work on the map at the same time i.e. I'd often want to add a handful relays/triggers/script-related spawnargs while bikerdude does architectural work - my workaround was to put all my changed entities into their own layer on my local copy so I can copy them over when I next get the map.

I'd definitely always want to use version control for future collaborative projects.

Link to comment
Share on other sites

I'm currently prototyping a feature to produce a "diff" between two maps, which I figured would be a foundation for other features in this realm. Once a two-way map diff is possible, the next step would be to create a three-way map diff. Once I have that, we can more reliably identify conflicts between different "forks" of the same map, which is something that is happening all the time when angua and me are working on different sections of the same mission (of course we're using the same natural workaround to put stuff into layers and export them to prefabs later for integrating them back in the main line version).

With diff-ing and a conflict resolution UI in place, it's a comparably small step to add a VCS plugin to DarkRadiant, which allows to browse commits/branches and to integrate changes into the currently opened map. It's not necessary to integrate the whole Git feature stack into DarkRadiant's UI like Visual Studio or VS Code, all that pushing, fetching, pulling and rebasing can happen in whatever Git client people are comfortable with - DR would just allow picking a version to merge. With a Git history accessible, it's rather easy to find the common base version of two maps which can be used for a three-way merge.

Even if people are working alone, the benefit of being able to commit versions and have the history stored locally and/or remotely is an improvement and might reduce the amount of hair-pulling occurring after losing work on your map.

Link to comment
Share on other sites

10 hours ago, OrbWeaver said:

Even if VCS features don't become part of DarkRadiant itself, a basic how-to guide on the wiki might be a helpful starting point.

Yes, the downside of the VCS integration is that folks need to learn their way around them - learning Git is something I saw quite a few programmers fail at, so I'm kind of afraid that mappers might not buy into it because of the entry barrier. Mapping itself is full of learning curves, having to work with something like Git might break one's back. Whatever tool support we have, a tutorial-like wiki article would definitely be helpful, I totally agree with you on that.

Link to comment
Share on other sites

SVN also has feature of file locking. I think mappers implement manual locking when they work on a map on google drive: someone says "I'm editing it now", then upload his changes and say "I have done with it, the map is free". That's exactly the locking paradigm, but on the whole map. SVN locking was designed for working with files which are unmergeable (like models or images).

By the way, does anyone know what's the status of betamapper SVN repo?
Who are the intended users of this repo?

 

Speaking of map diff, wouldn't it be enough to normalize order of entities and primitives, and run ordinary diff afterwards?

UPDATE: By the way, hot-reload feature already computes some diff for map file, but it is only interested in entities, whcih can be easily matched by name --- so it is very basic.

 

I think asking mappers to learn git is overkill. If you wrap the whole git into simple commands, then it has some chance.
There are some non-programmer VCS, and they are usually very simple. For instance, Sibelius stores versions in the same file, allows switching between them and viewing diffs. Did not even find any merging there. Word and SharePoint also seems to use sequence of versions, but Word at least has merging (two documents without base version, I suppose).

It is strange to suggest mappers learning the VCS which is hardest to learn/use out of all possibilities.
Git does not provide any additional capabilities over mercurial, but pushes a lot of terms over user which he regularly bumps into and has to learn eventually. Plus it is very easy to lose data. Git ninjas know that deleted branches are not lost: just look at reflog... but for non-ninja it is the same as "lost".

Link to comment
Share on other sites

4 hours ago, stgatilov said:

I think asking mappers to learn git is overkill. If you wrap the whole git into simple commands, then it has some chance.
It is strange to suggest mappers learning the VCS which is hardest to learn/use out of all possibilities.
Git does not provide any additional capabilities over mercurial, but pushes a lot of terms over user which he regularly bumps into and has to learn eventually. Plus it is very easy to lose data. Git ninjas know that deleted branches are not lost: just look at reflog... but for non-ninja it is the same as "lost".

Couldn't disagree with this more, except for this part: "If you wrap the whole git into simple commands, then it has some chance.", and that's exactly what modern IDEs (e.g. VS Code, Github Desktop) and the Github/Gitlab UI do for us.  Using something like Github on top of Git makes things a lot more n00b-friendly and it's important to note that.

If a mapper can learn DarkRadiant and TDM, then learning the basics of Git should be a piece of cake.  I would also suggest if people understand the problems it solves they would be a lot more open to it, but perhaps most mappers don't really see an issue because they are working alone and Google Drive/Dropbox does the trick for them?

Anyways, working alone or not, I wouldn't be caught dead not using a VCS.  I think @greebo and @OrbWeaver are right in that a tutorial would be a great help in showing what it can do and how to use it.  I would be happy to do that if someone could give me a Wiki account.  If I did it, it would be for Git/Github and not SVN however.  I have wanted to do a tutorial anyways, and the reason I haven't is because every time I mention Git I get one or more of the following responses:

  1. crickets
  2. it's not good for binaries
  3. it's overkill

I suspect 1) is due to either 'preaching to the choir' (those who use it already understand the benefits) or something around what @stgatilov is saying in that people might think it's too complex, or don't see the benefits.

2) Just not true, except if you have very large files (and a lot of them), so doesn't really apply to maps.  Large videos are an issue if over 100 MB (with Github anyways).  Yes, you can't 'merge' or diff binaries, but frankly who cares?  I'll just take the latest version, please.  You committed it by mistake?  Just re-add the old version after you rescue it from your history.

3) Learning about few basic features (not even about Git, but VCS in general) and the benefits it provides make it a worthwhile investment.  Nobody needs to become a 'Git Ninja' (I am certainly not one).

Regarding using Git for mapping:

As others have mentioned, it's great and I love it, but yes, what to do about the map-editing problem.  I don't know what to suggest here because I haven't delved into how how DR and doom maps work.  Without even trying it, I would guess that one issue would be with entity/primitive naming?  For example if I was working on the 'main' branch, and and someone else created a new branch off that and we both added a new (but different) entity, it would have the same entity number in both maps (and merge conflict?).  What are the other potential issues?  As I said, I've just avoid the entire scenario because I guessed it would be problematic.  We've just been using the following workflow:

  • Protect 'main' branch in Github so no direct pushes can be done against it
  • I will work on the map on my own branch.  I can/will modify any file in the FM.  Merge stuff into main periodically via pull-request (because main is protected).
  • Kerry000 would periodically pull the latest from main, then create his own branch (I haven't bothered to show him about 'rebase' - maybe later 🙂 )
  • Kerry000 would work on his branch and edit anything except the map file (xdata, textures, models, etc), and raise a PR when finished
  • merge Kerry000's branch into main.

We had a pretty clear separation of duties on Hare in the Snare part 1, so that made it pretty easy.  However for part 2 he's getting into mapping a bit, so we will need to go down the 'import prefab' route I think going forward.

 

 

 

 

Edited by Frost_Salamander
Link to comment
Share on other sites

19 minutes ago, greebo said:

Done, you should have mail.

Thanks Greebo.  Before I start writing anything, is there a way with this Wiki to write and save articles as drafts before publishing anything?

Edited by Frost_Salamander
Link to comment
Share on other sites

16 hours ago, greebo said:

..are familiar with version control (Git)?

I'm familiar with it and use it on other projects.

 

16 hours ago, greebo said:

...are or have been using version control for their mapping projects?

I don't do anything spectacular. I just use the "save a copy" function of DarkRadiant to make numbered copies as I go. Occasionally I will plop a zipped copy on my Dropbox. Maybe it's the wrong way to think, but I can't help feeling things like Git would be overkill for this sort of thing.

 

16 hours ago, greebo said:

...are or have been mapping in a team, using version control to collaborate?

I have only ever worked solo. I'm that bad a mapper that I feel I couldn't contribute anything useful to a team.

My missions:           Stand-alone                                                      Duncan Lynch series                              

                                      Down and Out on Newford Road              the Factory Heist

                                                                                                  A House Call

                                                                                                  The House of deLisle                                                                                                  

                              

Link to comment
Share on other sites

7 hours ago, stgatilov said:

Git does not provide any additional capabilities over mercurial, but pushes a lot of terms over user which he regularly bumps into and has to learn eventually.

Like what? You "commit", you "push", you "pull", you "merge", you "switch" branches (if you use branches, which most mappers probably won't be doing). These terms are pretty much identical between Git and Hg, and provided you are using a decent GUI the workflow is basically the same. Honestly I think the "Git is hard to use" meme dates from about 2006 (when the only way you could use Git was via a bunch of non-obviously-named shell scripts on Linux) and hasn't been updated since.

Not that I've got anything against Mercurial, it is a decent user-friendly VCS which I would be happy to use if it had become the market leader. But the disadvantage is that relatively few public sites offer support for it, since everybody is using GitLab and GitHub these days.

3 hours ago, Frost_Salamander said:

it's not good for binaries

I always used to think this, but recently @stgatilov actually found that SVN sucks for large binaries too, because of the limited window it uses while diffing. So I don't think there's any VCS that is actually good for binaries. SVN does have the advantage that you don't need to clone the entire history when you checkout the latest version, which might make it slightly better for absolutely HUGE projects.

3 hours ago, Frost_Salamander said:

it's overkill

I never really understand this argument either.

"Spending 30 seconds to click the Commit button and enter a brief log message is too complex, therefore I'm going to manually make ZIPs of my map and copy them to another directory because that will be much quicker!" It's actually more work to do the thing which is "not overkill" than to do the thing which is "overkill".

I suppose if you're using something like Dropbox already and you just put your WIP map directory inside your Dropbox folder, that is going to be easier than using any kind of VCS since it's all automatic.

Link to comment
Share on other sites

Getting slightly off-topic here, but when @stgatilov and I were discussing the whole SVN/Git/binaries thing I did some digging around to see how 'real' game companies deal with this.  From what I could gather, it's:

  • SVN for small projects/indie studios
  • Perforce for large studios, or for when SVN will no longer cut it

And that's it.

I also stumbled across this, which looks promising: https://www.plasticscm.com/

They have a free cloud version for projects up to 5 GB (although not sure that would cut it for TDM or not?)

 

 

Link to comment
Share on other sites

How amazing we'd both write about almost the same thing at almost the same time last night 😄 As my thread about full realtime / multiplayer editing is its separate idea, I'll simply quote my post on the matter of non-realtime collaborative editing over Git / SVN and the challenges I see with this idea.

9 minutes ago, MirceaKitsune said:

The biggest issue is that the moment two or more creators edit the same item, there would be a conflict once the changes are faced with each other and have to be reconciled. We could teach DR how to resolve map conflicts and pick one set of changes, but this means one author's version would be chosen while others have to lose their work. The problem doesn't even end there: Some changes can be reconciled in theory but still lead to inconsistencies in practice! Say one author moves the walls of a building while another handles its furniture entities; DR can agree on which set of changes to keep for each of those movements and everything would seem resolved to the code, however if they aren't made or solved together you're left with an empty house in one place and floating furniture in the other, which a person has to later notice and manually resolve.

There may be ways to reduce this risk. One is attributing specific groups of brushes and entities to different creators: Only the person who spawned an item may edit it, no changes to other people's stuff allowed. This way anyone can push map changes to a Git repository at any point regardless of what other developers may have done during that time, the differences can be merged under any circumstance as no conflicts can occur. But it's a limiting approach since you can't help another mapper with their part of the map, while on the other side they can build over what you're doing and you couldn't get their stuff out of the way.

A good way to take this approach would be allowing maps to be stored in multiple files during development, so changes don't need to be made to the same file altogether. This means that alongside the main myfm.map you can have myfm.bob.map + myfm.mike.map + myfm.alice.map and so on, when loading the main map content is also loaded from those additional files: Now you can push and pull the git repo at your own leisure, your work is always up to date locally while changes made by others become visible whenever you do an update and reload the map! This alone is unlikely to be noticed or used by many creators though, unless encouraged by additional Git tools in the interface; DR could support pushing and pulling changes directly from the toolbar, with an option to automatically update other authors files at an interval (say 5 minutes by default) then refresh the map seamlessly.

While this would all be fun and easier for starters, the holy grail would be realtime editing at the end of the day. That way anyone can work on anything and always see the updated result on their end during all stages... without having to fear emerging conflicts that need to be reconciled later, unless someone's connection goes down and they make changes during that time which should probably be disallowed by the code. If designed properly we could have a powerful and unique system to work with here.

 

Edited by MirceaKitsune
Link to comment
Share on other sites

Okay, got a good chunk of the Wiki article done: https://wiki.thedarkmod.com/index.php?title=Git_and_Github_for_Mappers

All feedback, corrections and contributions welcome.  I haven't really scoured it for typos and spelling mistakes, but I'll chip away at that, and there are more sections I am planning as well.

Also it would be cool if someone who hasn't used Git/Github before went through the tutorial to see how it works for them (and provide feedback if they have issues).

Edited by Frost_Salamander
Link to comment
Share on other sites

4 hours ago, Frost_Salamander said:

Okay, got a good chunk of the Wiki article done: https://wiki.thedarkmod.com/index.php?title=Git_and_Github_for_Mappers

I hoped for something simple, but it indeed looks complicated. As expected 😥

Why do you suggest using pull requests?
Reviewing map changes is futile in many cases. Wouldn't it be easier to work on one branch in one repo and push directly? Or is it a workaround for git's ability to easily lose data?

Why do you suggest rebasing?
That's an easy way to lose data. Better just merge everything.

  • Like 1
Link to comment
Share on other sites

One interesting idea for mappers would be to create a repo on GitHub, and then use TortoiseSVN client exclusively to work with it.

No hassle with branches, pulls/pushes, rebases, etc, no chance to lose data.
Just SVN update, then work with FM, then SVN commit.
Indeed, sooner or later you'll get conflicts, and that would be a serious challenge regardless of VCS/client.

It would be perfect if GitHub supported SVN locks too, but I'm afraid it does not.
Real SVN hosting is needed if you want to lock files.

Link to comment
Share on other sites

3 hours ago, stgatilov said:

I hoped for something simple, but it indeed looks complicated. As expected 😥

Why do you suggest using pull requests?
Reviewing map changes is futile in many cases. Wouldn't it be easier to work on one branch in one repo and push directly? Or is it a workaround for git's ability to easily lose data?

Why do you suggest rebasing?
That's an easy way to lose data. Better just merge everything.

It isn't complicated.  It may look like a lot, but this is a tutorial so it's quite verbose.  See this section, as I anticipated that response: https://wiki.thedarkmod.com/index.php?title=Git_and_Github_for_Mappers#Implications.  In reality the entire process is very quick, and when Kerry000 and I use it I usually just merge the PR as an administrator as described here: https://wiki.thedarkmod.com/index.php?title=Git_and_Github_for_Mappers#Merging_your_work:_quick_and_dirty_version

I suggest using pull requests for one very simple reason: the main branch is protected so you can't directly merge to it.  This is a Git best practice when working in teams.  Do you have to do this?  Absolutely not.  But if I was working with a couple of other people who aren't familiar with Git, I would sure as hell do this this.  Here's an example: on Hare in the Snare Part 1, Kerry000 and I had one simple rule: he was not to touch the .map file (because he wasn't mapping, and also I didn't want merge conflicts).  I used the pull request review to ensure the .map file wasn't modified.  If it wasn't, I would just approve the PR. 

If you are working by yourself, just push to main.  Nothing fancy needed there.  But I would caution a little bit.  Say you want to just try something out, like rework an entire section by deleting it first.  You'd want to use a branch for this in case you messed up or ended up discarding the work.  I'm sure you wouldn't want to accidentally push it to main right?  Again, totally up to the user how they want to do this.

The other thing with pull requests is if you raise one, the Github UI will tell you if there is going to be merge conflicts.  If you don't feel like dealing with that, you can just abandon it and fix it in the branch first.

Agree that you don't review .map file changes.  But there are lots of other things in a FM that aren't .map file changes.  Sometimes it's useful to just look at changes before they're merged (e.g. for awareness, or check for typos in an xdata file, to make sure file structure is correct, etc).  Not sure why you are fixating on just the map changes?

The rebase thing: There is much debate on rebase vs. merge, and the truth is I've just always used rebase to get the linear history correct, and never lost any data.  If a merge is better/preferable, then people are certainly welcome to use that, just need a 3rd option in that Wiki section on what to do.

 

Link to comment
Share on other sites

4 hours ago, stgatilov said:

One interesting idea for mappers would be to create a repo on GitHub, and then use TortoiseSVN client exclusively to work with it.

No hassle with branches, pulls/pushes, rebases, etc, no chance to lose data.
Just SVN update, then work with FM, then SVN commit.
Indeed, sooner or later you'll get conflicts, and that would be a serious challenge regardless of VCS/client.

It would be perfect if GitHub supported SVN locks too, but I'm afraid it does not.
Real SVN hosting is needed if you want to lock files.

Does TortoiseSVN work with Git?  I know there is also a TortoiseGit client, but never used it.  This is certainly an option.  I just use VS Code because it is also an editor so I used it for scripts, materials files, skins files, etc.  Either way, this is why I included the 'other tools' section: https://wiki.thedarkmod.com/index.php?title=Git_and_Github_for_Mappers#Other_Tools

In the entire lifecycle of Hare in the Snare part 1, we didn't get a single merge conflict.  As I mentioned before though, we had a clear separation of duties, and used the pull requests to make sure only the proper files were modified.

And we also didn't need to lock any files.  Just not a requirement at all.  I get that you're probably thinking of larger scale using than just two people, but there is no reason that won't work in Git.  Again, just use pull requests - if someone modified a model file that I am also working on all I would need to do is not approve the PR, or comment that I am also working on it so it might be overwritten later, etc.  Again, we're not talking about a professional studio here with 20 devs working on the same map - it's like a few people who are probably in constant communication and are aware of what's going on.

 

Edited by Frost_Salamander
Link to comment
Share on other sites

28 minutes ago, Frost_Salamander said:

I suggest using pull requests for one very simple reason: the main branch is protected so you can't directly merge to it.  This is a Git best practice when working in teams.

Hmm... I don't think it is best practice of Git to be honest. Maybe best practice of GitHub.
I see two reasons for protecting main branch:

  1. External contributions from people who has no access and whom nobody even knows.
  2. Enforcing code review on all changes.

Frankly speaking, I think none of these points apply to our context.
External contributions are maybe possible after the map is released, but not while it is in development. Code review only makes sense for scripts and decls, but I think nobody will bother about it.

20 minutes ago, Frost_Salamander said:

Does TortoiseSVN work with Git?

No, it does not.
GitHub exposes its repositories via SVN protocol, which allows you to use SVN client (until someone decides to use some "fancy" feature like submodule or LFS, of course).
A colleague of mine used TortoiseSVN in one very small project with git repo. He did not notice any problems, even though nobody even knew about it.
Unless people are going to really make many branches, using SVN client might be an option.

Quote

In the entire lifecycle of Hare in the Snare part 1, we didn't get a single merge conflict.  As I mentioned before though, we had a clear separation of duties, and used the pull requests to make sure only the proper files were modified.

And we also didn't need to lock any files.  Just not a requirement at all.  I get that you're probably thinking of larger scale using than just two people, but there is no reason that won't work Git.

Of course locking support is not required: you can always implement it outside of VCS. Just say "I'm going to work on these assets, don't change them please". That's implicit locking. Even with SVN, you can still forget about locking, make your changes, and then find out that they are in conflict with something.

Quote

Again, just use pull requests - if someone modified a model file that I am also working on all I would need to do is not approve the PR, or comment that I am also working on it so it might be overwritten later, etc.

Don't think pull requests are needed here.

Even with SVN, you can simply update the file, see conflict, and click "resolve with mine", and thus happily overwrite it.

  • Like 1
Link to comment
Share on other sites

While it's very interesting to have the predictable discussion about which is the only true way to use git correctly, my original intention before opening this thread was of course more directed towards getting some input about DarkRadiant and maybe what basic functions need to be added to make the VCS benefits more easily accessible to mappers, and/or to make collaborating with other members on the same map file a smoother experience.

Obviously, even with the current state of DR development it's possible to use a VCS to make mapping work more secure or more collaborative, but what functionality would make life easier?

  • Like 1
Link to comment
Share on other sites

16 minutes ago, greebo said:

While it's very interesting to have the predictable discussion about which is the only true way to use git correctly, my original intention before opening this thread was of course more directed towards getting some input about DarkRadiant and maybe what basic functions need to be added to make the VCS benefits more easily accessible to mappers, and/or to make collaborating with other members on the same map file a smoother experience.

Obviously, even with the current state of DR development it's possible to use a VCS to make mapping work more secure or more collaborative, but what functionality would make life easier?

@greebo apologies, we have derailed the thread somewhat.

One thing that comes to mind - I was never quite sure what happens in DR when I switch the underlying branch that contains the .map file.  I eventually got in the habit of closing DR before merging and then reopening it, because sometimes after merging I would have local changes in my branch still.  It might have been a VS Code thing as well.  Anyways - if it just detects changes and reloads the map that's fine - if not maybe something around that?

Edited by Frost_Salamander
  • Like 1
Link to comment
Share on other sites

7 minutes ago, Frost_Salamander said:

One thing that comes to mind - I was never quite sure what happens in DR when I switch the underlying branch that contains the .map file.  I eventually got in the habit of closing DR before merging and then reopening it, because sometimes after merging I would have local changes in my branch still.  It might have been a VS Code thing as well.  Anyways - if it just detects changes and reloads the map that's fine - if not maybe something around that?

Nothing happens immediately when you change the .map file on disk, DR will not care and continue working with what you have loaded into memory. In one of the recent versions, I forget which one, a safeguard has been introduced: once you hit "Save" DarkRadiant will warn you if the file has been changed in the meantime, possibly by some other process like git or a text editor. You can choose to overwrite or cancel (and save it somewhere else if you need to).

When I decide to update the working copy using Git on my end, for instance pulling a change from her laptop to mine, the map file will obviously be changed (which I expect to happen, no problems there). Still, I often had the thought that it'd be nice to have a small status indicator in the toolbar, such that I can see that the .map file has been altered by something other than the current DarkRadiant session. (An option to have those changes automatically integrate the changes from that file to my loaded version is a concept I'm currently trying to prove to be working.)

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recent Status Updates

    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
    • OrbWeaver

      I like the new frob highlight but it would nice if it was less "flickery" while moving over objects (especially barred metal doors).
      · 4 replies
    • nbohr1more

      Please vote in the 15th Anniversary Contest Theme Poll
       
      · 0 replies
×
×
  • Create New...