Jump to content
The Dark Mod Forums

I can develop an advanced mission finder


darksilence

Recommended Posts

Let's say that you want to play a city mission with snow, at night, medium-sized and without undead. You add those filters and get a list of missions matching them, as in the attached image.

It's just a rough sketch, but hopefully it portrays the idea I have in mind! More data and filters could be added, but the idea in any case is that players (especially new ones) have an easier time finding missions suited to their taste.

The "mission finder" would be a web application, which could be placed either on a new domain or a subdomain of thedarkmod.com.

I have plenty of experience working as a data systems developer, so I could fully take care of this.

What do you think? Could this be useful? Maybe not as useful as I think and would be a waste of time instead?

Interested in hearing other people's opinions on the subject.

Edit: Maybe this should have been posted in "I want to help"? Please feel free to move this thread there 😕

mission-finder.png

Edited by darksilence
Wrong forum
  • Like 3

The Dark Mod Database: https://tdmdb.com

Link to comment
Share on other sites

I think it is a good idea! Our ingame mission downloader and mission view has long been subject of multiple improvement suggestions. Due to the sheer mass of missions that have been released in the last 15 years, things got really cluttered and especially newcomers will have a hard time finding what they want. However, improving the ingame guis is quite a task, so a web-based application might really suit this scenario well. Maybe we could even add a linke to it from our ingame menus, so user can access it quicker.

Some more things to think about

  • Would users be able to add custom tags and downvote / upvote certain tags, much like the system of Steam? This would also allow to add tags like "beautiful", or "difficult". Actually, the more users can contribute to this system, the better, because it will be automatically maintained then.
  • The browser should also contain a flag for whether the FM belongs to a connected series of FMs or not and have the capability to go to the previous or next FM in that series.
  • There should also be a flag for fully fledges campaigns.
  • Some might like a flag whether or not an FM is "ghostable".
  • Like 1
Link to comment
Share on other sites

2 hours ago, STiFU said:

Would users be able to add custom tags and downvote / upvote certain tags, much like the system of Steam? This would also allow to add tags like "beautiful", or "difficult". Actually, the more users can contribute to this system, the better, because it will be automatically maintained then.

It would also let users add 'ugly', 'worthless', 'DarkLivesMatter' and 'buy-mandrasola.tk' (hopefully enough people would downvote the spam, but I think we might see disagreements about whether e.g. 'ugly' is a legitimate tag - no less a matter of opinion than 'beautiful' but potentially demoralising for mappers, given that it would appear as part of a quasi-official taxonomy).

  • Like 1

Some things I'm repeatedly thinking about...

 

- louder scream when you're dying

Link to comment
Share on other sites

I would be against subjective user tags like "ugly" for the reasons VanishedOne mentions. I think it should be restricted to objective criteria like "has it got spiders".

My missions:           Stand-alone                                                      Duncan Lynch series                              

                                      Down and Out on Newford Road              the Factory Heist

                                                                                                  A House Call

                                                                                                  The House of deLisle                                                                                                  

                              

Link to comment
Share on other sites

If this gets implemented successfully, I see a potential follow-on project that could use some of the same architecture, but for a mapper audience. Namely, find all official FMs that use a specific resource (e.g., texture, material, model, audio file, AI character, script or script function). Essentially, there would be a similar search form, a new backend database with this information, and new indexing scripts to gather it (run whenever an FM is released/updated). Some of this scripting would be easy; some would require inference (e.g., usage of prefabs). The search results would have not just each FM hit, but a link to detail info (e.g., where the resource is defined; where it is deployed in the particular map).

As it stands now, a mapper can use generic full-text-search tools on their dev box (and some manual poking around) to do this, but only over FMs they have downloaded and retained.

  • Like 1
Link to comment
Share on other sites

Thanks for the replies!

16 hours ago, STiFU said:

Would users be able to add custom tags and downvote / upvote certain tags, much like the system of Steam? This would also allow to add tags like "beautiful", or "difficult". Actually, the more users can contribute to this system, the better, because it will be automatically maintained then.

I'm thinking that in order to keep it simple, at first we wouldn't allow users to provide subjective opinions directly, mainly to avoid the issues mentioned above, such as spamming or demoralizing opinions, but I agree that at some point down the road, if this works well, this may evolve into what you are thinking.

One thing that the system could do though is scrape forum threads for information about particular missions. For example, there are many threads asking people what are their top missions; the system could add some kind of tag to missions appearing in those lists (e.g., "Top Mission", or a star or medal icon), and maybe even provide more recognition to those that appear in the majority of top lists.

Another thing that the system could do is pull data from the thread of each mission. For example, in the thread for In The North, there are lots of mentions of the mission being atmospheric and having great architecture. So that could lead to the tags "atmospheric" and "great architecture".

Another thing: in the past, demagogue suggested sending stealth server scores to a server. If we end up having this kind of data for missions, this could be a great way of showing other pieces of objective information about missions, such as average stealth scores, average completion time, etc. which could be very valuable to indicate difficulty and length of a mission, as well as whether a mission is fully ghostable (or whether it has ever been ghosted, to be more precise). 

 

Quote

The browser should also contain a flag for whether the FM belongs to a connected series of FMs or not and have the capability to go to the previous or next FM in that series.

There should also be a flag for fully fledges campaigns.

Good idea! What about a filter such as "Is mission part of a campaign?" This allows finding campaigns in which at least one of the missions meets the filter requirements (for example, one of the missions in the campaign has rain and it's a city mission).

Quote

Some might like a flag whether or not an FM is "ghostable".

Good idea too. Please see my comment a few paragraphs above regarding how to automate this. Seems that at first though we'd rely on forum threads for this information.

 

Quote

Love that undead boolean.

Would like size in mb please  so i can type it myself.

Awesome idea! Good job so far!

🤪 I agree with MBs, although maybe we need two independent filters? One for MBs (which seems like a more advanced filter for experienced players) and another one more "human" (e.g., small/medium/large/very large)? I'm saying this mainly because the size in MBs doesn't really say much to new players.

 

Quote

 

If this gets implemented successfully, I see a potential follow-on project that could use some of the same architecture, but for a mapper audience. Namely, find all official FMs that use a specific resource (e.g., texture, material, model, audio file, AI character, script or script function). Essentially, there would be a similar search form, a new backend database with this information, and new indexing scripts to gather it (run whenever an FM is released/updated). Some of this scripting would be easy; some would require inference (e.g., usage of prefabs). The search results would have not just each FM hit, but a link to detail info (e.g., where the resource is defined; where it is deployed in the particular map).

As it stands now, a mapper can use generic full-text-search tools on their dev box (and some manual poking around) to do this, but only over FMs they have downloaded and retained.

 

 

Raising the stakes, but I like this idea! We could actually use the same application, just use different DB tables for the mapper tools. In some cases we could even interconnect some of the data, and maybe find out some interesting relationships between resources used by FMs and their reception by the public (a simple example: maybe we find out that FMs using a specific set of resources tend to be regarded as "atmospheric").  

Edited by darksilence
  • Like 2

The Dark Mod Database: https://tdmdb.com

Link to comment
Share on other sites

Hi!

I decided to undertake this project, and wanted to show a little bit of what I have so far. Here's a quick video of the app in use: https://streamable.com/2mbs0w.

You may notice that I decided to go for something a bit more appealing to the eye than the original tables I proposed. This also results in a mobile-friendly application (this way, people can search for their next mission while chilling on the sofa). I'm satisfied with how it looks, but I could be wrong, hence any feedback is welcome.

I retrieved all data so far by scraping thedarkmod.com and the wiki with a couple of scripts. I will also be publishing the source code to a repo on GitHub once the project becomes a reality.

I'm working now on the filters/sorting, and after a few more touches the first version should be ready for beta-testing.

All thoughts/feedback appreciated. This is also your time to stop me before I get too attached to this project :P 

 

  • Like 3

The Dark Mod Database: https://tdmdb.com

Link to comment
Share on other sites

  • 2 weeks later...

Looks very nice!

I see the following benefits over what website currently offers:

  1. Some more information about mission type (I mean things like "jail", "platformer", etc). Maybe it should be called tags, but name does not matter.
  2. Provides information about types of monsters available --- useful for everyone, especially great for people with phobia.
  3. Adds link to forum thread.
  4. Adds search/filter panel on all of this.

The information from p.1 and p.2 probably exists only on the wiki, but not in the database. There is organizational problem with official database: the number of people who can edit it (team members) is much less than the number of people who can edit wiki page. We can say "let's add this info to our official database!", but I'm afraid the end result would be that the information would become less rich and useful over time.
Although by looking at wiki page edit history, it seems that it is mostly maintained by @nbohr1more (maybe because he prefers "small commits"😀). So maybe it is not that bad. Wiki page is also good because it is much easier to edit and administrate due to all the power of mediawiki.

So I'm not sure about how to avoiding scraping.
Perhaps it would be more natural to add search abilities to the wiki page, if it is even possible. Although it cannot provide a nice look with screenshots.

  • Like 2
Link to comment
Share on other sites

Thanks @stgatilov

Regarding avoiding scraping, the way to do this is to have an official, trusted database, and then have everywhere else pull data from it through an API request, instead of HTML scraping.

For example, suppose that in the future tdmdb.com becomes the trusted DB and is managed by the TDM team. Then, I could write a simple REST API so that mission data can be obtained through a call that returns a nice JSON with such data.

This way, a lot of the information that appears in the wiki could also be automated, such as the list of missions, without the requirement of having someone manually edit the table every time a new mission comes out.

  • Like 1

The Dark Mod Database: https://tdmdb.com

Link to comment
Share on other sites

3 hours ago, darksilence said:

This way, a lot of the information that appears in the wiki could also be automated, such as the list of missions, without the requirement of having someone manually edit the table every time a new mission comes out.

If you choose between editing a wiki and editing a custom database, it is easier to edit the wiki.
And more people have permission to edit our wiki than anything official.

So while it does not sound right from programming point, wiki table seems to be a better source of truth than official database.

Link to comment
Share on other sites

3 hours ago, stgatilov said:

If you choose between editing a wiki and editing a custom database, it is easier to edit the wiki.
And more people have permission to edit our wiki than anything official.

So while it does not sound right from programming point, wiki table seems to be a better source of truth than official database.

I agree that editing a database manually is not ideal, but this doesn't have to be the case: an interface can be easily developed for that.

For instance, the way TDMDB was engineered, such an interface would be very easy to add (and in fact it's in my TODO list). With such interface, adding/editing missions would imply a few seconds of filling some inputs and marking checkboxes. Quick and easy.

I also agree that the wiki is a great source at the moment, and that's why I retrieved most of the information by scraping it, although some information was more accurate on thedarkmod.com (such as mission size), hence the scripts selected each piece of information from the most reliable source for each field.

Edited by darksilence
  • Like 1

The Dark Mod Database: https://tdmdb.com

Link to comment
Share on other sites

To be honest, I would prefer putting the whole FM database (including the actual pk4 of FMs) into yet another SVN.
After that establish automatic sync from this SVN to http mirrors, from where they can be downloaded by players.

This way:

  1. It is obvious which FMs are where: every FM is on every mirror.
  2. It is not necessary to manually update every mirror (sometimes we forget to update some mirrors, causing random players playing old version). Just commit new version to SVN, and trigger sync task.
  3. It is clear who and when changed every FM. Even though exact changes are not viewable.
  4. It is possible to return back to any old state of any FM.

The metainformation can be stored in the same SVN in e.g. XML format. It can be edited with any XML editor or just text editor, and diffs are easily viewable. You can blame the XML and see for each line who changed it last.

  • Like 1
Link to comment
Share on other sites

7 hours ago, stgatilov said:

To be honest, I would prefer putting the whole FM database (including the actual pk4 of FMs) into yet another SVN.
After that establish automatic sync from this SVN to http mirrors, from where they can be downloaded by players.

This way:

  1. It is obvious which FMs are where: every FM is on every mirror.
  2. It is not necessary to manually update every mirror (sometimes we forget to update some mirrors, causing random players playing old version). Just commit new version to SVN, and trigger sync task.
  3. It is clear who and when changed every FM. Even though exact changes are not viewable.
  4. It is possible to return back to any old state of any FM.

The metainformation can be stored in the same SVN in e.g. XML format. It can be edited with any XML editor or just text editor, and diffs are easily viewable. You can blame the XML and see for each line who changed it last.

Excellent idea!

For starters, I've added an XML feed to TDMDB, which includes all missions data, here: https://tdmdb.com/xml. This can be quickly parsed in order to reuse any of the information that currently is only available in both the wiki and TDMDB.

Note that I added the tag DarkModComId to each mission, which represents the mission id that appears on thedarkmod.com, which I assume is the id for each mission in the official DB (for example, for The Lost Citadel,  the DarkModComId is 150 -> https://www.thedarkmod.com/missiondetails/?id=150).

Edited by darksilence

The Dark Mod Database: https://tdmdb.com

Link to comment
Share on other sites

On 8/28/2020 at 6:11 PM, stgatilov said:

To be honest, I would prefer putting the whole FM database (including the actual pk4 of FMs) into yet another SVN.

  1. It is possible to return back to any old state of any FM.

I'd like to emphasize that this is a serious matter, not a joke.

Right now I need to do minor update to several FMs. And since I know that history is not saved, it is pretty scary.

Imagine the following situation. I create pk4, test it and start uploading, without knowing that my network adapter is broken (or my internet provider has broken outgoing traffic). Since it is absolutely obvious that I must update all mirrors simultaneously, that's what I do: update all of them, replacing each copy with a broken file. As the result, the pk4 of the FM is gone from all official mirrors at once, and can only be salvaged from players' machines (and from author's machine if he is still active).

  • Like 2
Link to comment
Share on other sites

Actually, this would also come in handy for tying missions to specific TDM versions.

If you have TDM 2.02 installed and a mission was updated to work with 2.07 you could use the SVN history

to furnish the original to the TDM 2.02 player. Of course, I guess it would be less expensive to use a home-grown

solution for this use-case than to make the mirrors do all that SVN overhead for individual downloads...

Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Link to comment
Share on other sites

10 hours ago, nbohr1more said:

I guess it would be less expensive to use a home-grown solution for this use-case than to make the mirrors do all that SVN overhead for individual downloads...

My suggestion was to write a script which would extract clean latest revision from SVN, and them push it to all the mirrors. So on the mirrors/game side, nothing will change. SVN is not intended for downloading, it is only for managing the database.

Link to comment
Share on other sites

On 8/29/2020 at 1:58 PM, stgatilov said:

As the result, the pk4 of the FM is gone from all official mirrors at once, and can only be salvaged from players' machines (and from author's machine if he is still active).

None of the official mission servers are backed up? Yikes.

I would have thought you'd want one master server with full backups and maybe version control, and this is the only server you upload to, then a bunch of mirrors which just replicate whatever is on the master server without needing their own backups.

Link to comment
Share on other sites

1 hour ago, OrbWeaver said:

None of the official mission servers are backed up? Yikes.

I have not heard of it, but can't say for sure.

Quote

I would have thought you'd want one master server with full backups and maybe version control, and this is the only server you upload to, then a bunch of mirrors which just replicate whatever is on the master server without needing their own backups.

That's what I suggested.

Link to comment
Share on other sites

  • 8 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recent Status Updates

    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
    • OrbWeaver

      I like the new frob highlight but it would nice if it was less "flickery" while moving over objects (especially barred metal doors).
      · 4 replies
    • nbohr1more

      Please vote in the 15th Anniversary Contest Theme Poll
       
      · 0 replies
×
×
  • Create New...