Jump to content
The Dark Mod Forums

FM database upgrade


stgatilov
 Share

Recommended Posts

I have just found a small bug in my code for missions downloading 😥
So the migration will not go completely unnoticed.

If you try to download FM which is not yet present on all mirrors, and the game randomly selects an out-of-date mirror, then the game will stop the download and show a warning message to you.
This will happen with TDM 2.09. The newer versions (2.10+) will silently try another mirror in such situation, so player won't notice that some mirrors are not yet updated.

Note that it only happens with the new FM system, and only when you try to download FM which has just been added/updated.
The workaround is simple: just retry the download a few times: sooner of later it will randomly choose a good mirror.

  • Like 1
Link to comment
Share on other sites

Converted all 147 FMs, committed them to SVN.
Ready for internal testing by team members.

Problems encountered:

  1. Internal name of @Goldwell's "Lord Edgar's Bathhouse" was changed from "bh" (too short) to "bathhouse".
  2. Internal name of @some1stoleit's "Cleaning Up the Neighbourhood" was changed from "cleaninguptheneighbourhood" (too long) to "cleanneighbourhood".
  3. Dragon's Claw and Elixir were missing on the main mirror, so I had to convert them manually.
  4. Mission "Away 1 - Air Pocket" had internal name "airpocket.pk4", which is invalid. Changed to just "airpocket".
  5. The old FM database contained empty entries for "Training Mission" and "Tears of Saint Lucia", but the new one will not have them (they are bundled with TDM).

The only player-visible thing here is renaming the internal name. It is quite likely that you will be able to download second "Lord Edgar's Bathhouse" FM and have them two at the same time with same title. If you see such problem, just delete both instances of FM and redownload it again.

  • Like 2
Link to comment
Share on other sites

Is there any chance of making the order and names be consistent with the web version while you are doing this? Regarding "a" and "the" and full names being cut on the loading screen and campaign affiliations missing. I fix a lot of the latter in my Unofficial Patch, but it would be much easier if this would be in the core mod...

  • Like 1
Link to comment
Share on other sites

4 hours ago, wesp5 said:

Is there any chance of making the order and names be consistent with the web version while you are doing this? Regarding "a" and "the" and full names being cut on the loading screen and campaign affiliations missing. I fix a lot of the latter in my Unofficial Patch, but it would be much easier if this would be in the core mod...

The missions are sorted by their internal name in the new SVN database (and in XML file).
How various clients reorder missions --- that's out of scope of this process.

Link to comment
Share on other sites

5 hours ago, stgatilov said:

Internal name of @Goldwell's "Lord Edgar's Bathhouse" was changed from "bh" (too short) to "bathhouse".

Thanks for fixing that!

  • Like 1
Link to comment
Share on other sites

We have just got past point 3.

Now in-game downloader takes all the data from the new system.
If you have any problems with it, or something is wrong with some FM, please report here.

Link to comment
Share on other sites

On 5/5/2021 at 2:35 PM, stgatilov said:

I have just found a small bug in my code for missions downloading 😥
So the migration will not go completely unnoticed.

If you try to download FM which is not yet present on all mirrors, and the game randomly selects an out-of-date mirror, then the game will stop the download and show a warning message to you.
This will happen with TDM 2.09. The newer versions (2.10+) will silently try another mirror in such situation, so player won't notice that some mirrors are not yet updated.

Note that it only happens with the new FM system, and only when you try to download FM which has just been added/updated.
The workaround is simple: just retry the download a few times: sooner of later it will randomly choose a good mirror.

Has anyone considered using a CDN (content delivery network) for this instead?  There are free ones for open source.  For example, you could publish the FM on Github as a release artifact (yeah, I know, Github...) and then something like jsDeliver can make those files available over their CDN via the client's closest POP (point of presence).  All free and zero-maintenance.

EDIT: Just noticed that jsdelivr doesn't actually support release assets 🙄, but in general, the idea still stands and there might be other, similar solutions available For example, forget CDN and just link to Github release directly.  There is no limit on the number of files in a release, and they can be up to 2 GB each: https://docs.github.com/en/github/administering-a-repository/about-releases#storage-and-bandwidth-quotas

Edited by Frost_Salamander
  • Like 1
Link to comment
Share on other sites

36 minutes ago, Frost_Salamander said:

Has anyone considered using a CDN (content delivery network) for this instead?

I consider putting FMs to my Azure storage and serving them through Akamai CDN.
If it happens, it will be yet another mirror (maybe with greater weight) in the current system.

Quote

For example, you could publish the FM on Github as a release artifact (yeah, I know, Github...)

I'm not sure GitHub would be happy if we used it as file server, even though they don't post any limits on size of releases. It is abuse of the platform anyway.
Same most likely applies to jsDeliver.

Quote

All free and zero-maintenance.

Well, it is definitely not zero-maintenance, and will likely cease to be free after some moment.

Storing everything on one non-public server, asking some CDN to synchronize its mirrors with the server and serve them all behind one set of URLs with load balancing --- yes, it sounds very cool indeed. Although it most likely means tying to some particular CDN, and having to change everything if it suddenly goes away. And in case end-user links point to CDN domain instead of TDM domain, such migration will also break in-game downloader in old versions of TDM. Not that we specifically support old versions, but today we try not to break them without serious need.

HTTP file servers have been available since TDM was originally released, and they are still here. And I'm sure they will continue to live and work as long as TDM does. That's well worth it. UPDATE: same applies to rsync and cron jobs 😁

By the way, "poor man's load balancing" in TDM was implemented from the very beginning, I just added SHA verification and broke handling of "package missing on mirror" situation. I can't say I spent much time on it.

Link to comment
Share on other sites

21 minutes ago, stgatilov said:

I consider putting FMs to my Azure storage and serving them through Akamai CDN.
If it happens, it will be yet another mirror (maybe with greater weight) in the current system.

I'm not sure GitHub would be happy if we used it as file server, even though they don't post any limits on size of releases. It is abuse of the platform anyway.
Same most likely applies to jsDeliver.

Well, it is definitely not zero-maintenance, and will likely cease to be free after some moment.

 

Would you have to pay for the Azure storage yourself?

Re: the Github 'file server' - why is that abuse?  Each FM is an open source project, and it can have release artifacts.  This ties in with what I am suggesting with the Github organisation for TDM. Each FM can have its own repository in the org with all its source code (i.e the entire FM codebase), with associated release artifacts.

By zero-maintenance, I meant that you shouldn't have to maintain mirrors, worry about hosting, etc. Yes, someone will need to create the release, but that's as simple as clicking a button and uploading the .pk4 (or automate using Github actions)

Also, where are the FMs in SVN?  All I can seem to find is the TDM source itself.  Is there a link floating around somewhere?

 

EDIT: "Although it most likely means tying to some particular CDN" - well this is why you would use Github as the source (jsDelivr is just a layer over top).  Ultimately that's where the file is stored, and it's not going away any time soon.  If Github changed anything in the way they operate, the entire internet would break.

Edited by Frost_Salamander
Link to comment
Share on other sites

10 minutes ago, Frost_Salamander said:

Would you have to pay for the Azure storage yourself?

Yes indeed.

Quote

Re: the Github 'file server' - why is that abuse?  Each FM is an open source project, and it can have release artifacts.

Maybe. It is not source code, and not even code.
At least those hundreds of megabytes have nothing to do with text and with programming 😀

Quote

Also, where are the FMs in SVN?  All I can seem to find is the TDM source itself.  Is there a link floating around somewhere?

Currently it is not public.
And write access will never be public, just like it is now.
Public read access is possible.

 

Are you speaking about more decentralized model?
Like where everyone can release their FM on their own GitHub repo, and us only maintaining a central catalogue of such repos. And the in-game downloader being able to download directly from GitHub according to some protocol.

Link to comment
Share on other sites

3 minutes ago, stgatilov said:

Yes indeed.

That kind of sucks though right?  Azure blob storage is cheap, but still.  What Azure region are you using? You also said Akamai CDN - that won't be free either right? 

3 minutes ago, stgatilov said:

Maybe. It is not source code, and not even code.
At least those hundreds of megabytes have nothing to do with text and with programming 😀

Nothing wrong with that at all.  There is nothing in the terms of use to say Github can't be used for this.   And there is code in it (scripts, etc).  But, this is a silly argument and a non-issue 😃

3 minutes ago, stgatilov said:

Currently it is not public.
And write access will never be public, just like it is now.
Public read access is possible.

Can I have the read-only link please?  I can't find it published anywhere.  I do not want write access, and understand why you would want to keep it that way.  I'm not up to anything sinister, just curious what it looks like.  Dealing with hidden, closed source systems in an environment like this is kind of strange to be honest.

 

3 minutes ago, stgatilov said:

Are you speaking about more decentralized model?
Like where everyone can release their FM on their own GitHub repo, and us only maintaining a central catalogue of such repos. And the in-game downloader being able to download directly from GitHub according to some protocol.

That could be an end-state goal, yes.  What I would suggest for a start is after someone releases an FM, the FM source gets added to the Github org in its own repository, and the FM .pk4 is also added as a release artifact (and yes, a link to that for the in-game downloader).  The source tree and release are synchronized using the usual Git release tagging mechanisms.  A huge number of companies and ecosystems rely on Github to host their releases, I don't think there is much risk involved here?

When you say 'their own Github repo' and 'central catalogue', these would all be in the TDM github organization so everything is in one place.  This is a crucial thing missing from TDM at the moment IMO.  I would also suggest you move all your other tools and things to the TDM organization (e.g. DarkRadiant, the Perl I18N script, in-game downloader, modelling plugins, etc).  Everything is all over the place right now, and between that and these hidden SVN repositories it's not very community-friendly. If some of this already exists in Github, it can just be moved/added to the org and the maintainer can carry on as usual.

Because with a Github organization you can give per-repository permissions, you could give write to the FM author only (and maybe the admin team) so they could maintain it.  Others could still contribute by pull request of course, but it would be up to the author (and whoever else, like admins) to ignore or accept such contributions.  It's very easy to just protect the master branch from pushes and let others write to branches for their PRs.

What if the author knows nothing about Git or Github?  Well that's no different from now where most of them probably know nothing about SVN, where the FMs are maintained now.  I suppose just an admin takes the FM source and updates the repository themselves somehow.

Link to comment
Share on other sites

Here is what an FM looks like in Github (Hare in the Snare, part 1):

code:  https://github.com/thedarkmodcommunity/hits1

pull requests: https://github.com/thedarkmodcommunity/hits1/pulls

project board: https://github.com/thedarkmodcommunity/hits1/projects/1

releases: https://github.com/thedarkmodcommunity/hits1/releases

The release artifacts are automatically generated when you create a release.  Currently, this is just a .zip/tar of the repo source, but a .pk4 can be manually added as well.  However, if all assets are in the repo, then you don't need to manually do anything at all, just download/rename the zip to .pk4).

Another advantage of using Github for the FM is it makes collaboration much easier.  All you need to do is clone the repo into your darkmod/fms folder.  When one person pushes an update, the other person just needs to pull and they will have all the latest changes.  They just need to run DMAP or whatever and off they go.  No more passing .pk4 files around, etc.  

EDIT: updated links, as I just moved the repo into the TDM Github org

 

 

Edited by Frost_Salamander
Link to comment
Share on other sites

That's still unintended usage.
GitHub is intended to be used to store source code, not assets.

There is strict 100 MB limit on a file inside a repo. Several video files already exceed this limit.
You cannot push larger file to GitHub. Without using LFS, which is just one more hack on top of git, which is already over-complicated by itself.

GitHub terms also contain this:

Quote

The Service's bandwidth limitations vary based on the features you use. If we determine your bandwidth usage to be significantly excessive in relation to other users of similar features, we reserve the right to suspend your Account, throttle your file hosting, or otherwise limit your activity until you can reduce your bandwidth consumption. We also reserve the right—after providing advance notice—to delete repositories that we determine to be placing undue strain on our infrastructure. For guidance on acceptable use of object storage in repositories, refer to "What is my disk quota?". For more details on specific features' bandwidth limitations, see the GitHub Additional Product Terms.

If one organization holds 150 repos with 9 GB release packages and 30 GB total size, wouldn't that draw attention?
Recall that GitHub is intended for source code, and it rarely reaches such huge sizes. And when people have gigabytes of source code, they don't use git any more.
And yes: GitHub is commercial thing, and you suggest stressing all limits while being a free user.

Also, here are limits on LFS:

Quote

If you use more than 1 GB of storage without purchasing a data pack, you can still clone repositories with large assets, but you will only retrieve the pointer files, and you will not be able to push new files back up. For more information about pointer files, see "About Git Large File Storage."

If you use more than 1 GB of bandwidth per month without purchasing a data pack, Git LFS support is disabled on your account until the next month.

Somehow, we don't fit into 1 GB limit.

Although there are packs: 5$ per 50 GB storage and bandwidth. Storage is more than enough, but bandwidth is too low. I guess something like 5-6 packs is needed to cover the current bandwidth for FM downloads.
UPDATE: Actually, I just looked into PMs from our admins, and it seems that our bandwidth is on the level of TB/month.

Besides, additional conditions say user should not use Actions for:

Quote
  • any activity that places a burden on our servers, where that burden is disproportionate to the benefits provided to users (for example, don't use Actions as a content delivery network or as part of a serverless application, but a low benefit Action could be ok if it’s also low burden); or
  • any other activity unrelated to the production, testing, deployment, or publication of the software project associated with the repository where GitHub Actions are used.

Using releases as CDN for non-software project may fall into this category.

Link to comment
Share on other sites

Quote

Can I have the read-only link please?  I can't find it published anywhere.  I do not want write access, and understand why you would want to keep it that way.  I'm not up to anything sinister, just curious what it looks like.  Dealing with hidden, closed source systems in an environment like this is kind of strange to be honest.

Here is how it looks like:

missionsSVN.png.7a8a21583df2aa0fb3ee6a97c681a221.png

One directory per FM, each contains an XML with metainfo, pk4 files, and directory with screenshots for in-game downloader.

Plus list of mirrors in a different place.

 

On a related note: I strongly agree that passing files around is not a good thing. I'd say every mapper should use some version control system to store his FM, even if he works on it alone. It can be GitHub if you collaborate, or local git/hg/svn repo if you don't, just use something.
In fact, we have a betamapper SVN repo somewhere, but I don't understand what are access policies behind it, so most mappers probably cannot use it.

  • Like 1
Link to comment
Share on other sites

Do we have any hard numbers related to bandwidth usage?

You're right in that the video files are an issue - you do need LFS for files larger than 100 MB, and then bandwidth is a problem as you say.   I don't know how popular this suggestion will be, but what about a limit on video size?  If they are under 100 MB (very doable) then this goes away potentially.  This post kind of suggests a lot is possible here (although it won't solve the problem with older FMs with large videos mind you).

Some of this wording I think can be worked around.  TDM and FMs ARE 'software projects'.  They just happen to be games.  I'm not sure what the issue is here.  A lot of the assets are binary, yes, but a lot aren't.  Skins, material files, some models, scripts, map files, etc are all text.  At the end of the day, we could always just ask Github themselves about it if there is concern.

Also calling LFS a hack and dismissing Git as over-complicated I think is unfair.  It's very easy to use.  I've been using it for years in a professional capacity and I rarely need to do anything except clone, push, pull and create/merge pull requests, and this is usually through either a code editor or UI (Github or Azure DevOps, for example).  I pretty much never use Git command line, except for the initial clone and it's a copy/paste one-liner.  Also, the bandwidth limitation I think only applies if you are using LFS, so the FM download thing I think would be fine - they wouldn't allow you to attach unlimited 2 GB binary artifacts as releases if they had a problem with this.

Regarding Github actions - that was just a suggestion if anything needed to be done for the packaging/release process.  I don't think packaging and uploading a file on release would put a burden on their servers once in a while, and I wasn't suggesting using Actions as a CDN.

 

 

Edited by Frost_Salamander
Link to comment
Share on other sites

Started final phase: minor updates to FMs.

Updated Matter of Hours FM: now loading GUI shows two tips for approximately half time each.
The mirrors have not updated yet --- to be investigated.

I must admit that downloading this FM in TDM 2.09 when most of the mirrors have not updated yet is not fun.
If we do a hotfix release soon, the fix for downloader should be included.

  • Like 1
Link to comment
Share on other sites

34 minutes ago, stgatilov said:

Started final phase: minor updates to FMs.

Can you take a look at the names of campaigns or similar that do not fit the loading screen? I changed them to fit in my Unofficial Patch and while you are going through FMs you might do the same. You can find my version here: https://www.moddb.com/mods/the-dark-mod/addons/the-dark-mod-unofficial-patch

Edited by wesp5
  • Like 1
Link to comment
Share on other sites

Regarding names that don't fit on the loading screen, it would be helpful if the annotation marks (update/translation/new) could be visible even if the whole title isn't visible.

I also am surprised that I have quite so many updates marked as available; I'm suspicious that TDM is a bit muddled about the missions I already have downloaded.  Is there a good way to check this?

For example.  I saw Requiem marked as available for update, so I took a copy of my existing fms/requiem directory, let the updater do its thing, and the file dates have changed ... but the .pk4 files are identical (well, at least, their md5sum values agree as bea3c38aa9782d592a055af543c5ecb3).

I've also got loads of mission directories with version numbers in their names and an annoying number of duplicates.  I'm kind of reluctant to delete the lot and have to re-download 9G of files, but something seems messed up.  What's the cleanest way to sort this out?

 

  • Like 1
Link to comment
Share on other sites

7 hours ago, Araneidae said:

I also am surprised that I have quite so many updates marked as available; I'm suspicious that TDM is a bit muddled about the missions I already have downloaded.  Is there a good way to check this?

For example.  I saw Requiem marked as available for update, so I took a copy of my existing fms/requiem directory, let the updater do its thing, and the file dates have changed ... but the .pk4 files are identical (well, at least, their md5sum values agree as bea3c38aa9782d592a055af543c5ecb3).

Yes, some spurious updates display for me too.

The game saves information about which version of FM you downloaded in the "missions.tdminfo" file. Unfortunately, it only saves this file when you exit game or restart engine. If TDM crashes, then information is lost. If the file is not saved, then the game will not know which version you have the next time you start it. You will see the FM, but in-game downloader will think it has version = 1.

So you can get unnecessary update for FM if:

  1. you put pk4 file into fms directory directly, without using in-game downloader
  2. the game crashed after download, before you managed to restart engine or exit properly

I'm doing a lot of both😁

Quote

I've also got loads of mission directories with version numbers in their names and an annoying number of duplicates.

A lot of FMs have internal names with some numbers. That's how mappers name them.
Don't bother, unless you see many versions of the same FM.

Quote

I'm kind of reluctant to delete the lot and have to re-download 9G of files, but something seems messed up.

You should not download everything! Download whatever FMs you want in the nearest future and play them.

Damn it... I think we should remove the "download all" button.
I'm pretty sure a lot of people just click it just because they can, and quit or delete FMs before they even try 10% of it.
I bet that would be powerful traffic optimization for mirrors.

Quote

What's the cleanest way to sort this out?

You can update FMs that say they want to update. They should disappear after that.
Does in-game downloader still show Requiem update to you?

Link to comment
Share on other sites

2 hours ago, stgatilov said:

Damn it... I think we should remove the "download all" button.

No, we shouldn't! I always use it and I rarely have problems with duplicate missions. And in most of these cases the problem was that the FM author renamed the mission or something similar.

  • Like 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...