AngelLoader TDM Support (now released!)

[FenPhoenix]

I'm the developer of AngelLoader, a fan mission loader/manager for Thief 1/2/3/SS2. I recently decided to try adding Dark Mod support.

The latest release is here:

https://github.com/FenPhoenix/AngelLoader/releases

I'm going to keep a log of issues, ideas, notes, etc. here and we can discuss them or whatnot. If anyone has ideas or suggestions, feel free to post them here.

TDM version requirement:

I've only tested it with TDM 2.11. I'm not sure but I dimly remember FMs used to be stored differently at some point in the past? If so, it wouldn't work with those older versions. I assume people would normally keep their TDM install up to date, but just a heads up.

How it works:

In the Settings window -> Paths tab, choose your Dark Mod executable. Click OK. Now your TDM FMs will be detected, scanned, and added to the list.

For the other supported games, there is the concept of "installed" vs "not installed", this is because they normally come in archive files and then the loader can "install" them (extract the file into the "installed FMs" folder). For TDM there is no such concept (FMs are always installed), so the green checkmark "installed" indicator means that that FM is the currently selected one (the one that is loaded up when you start TDM), and only one TDM FM can be "selected" at a time. This indicator is kept in sync with the game, so if you change your selected FM inside TDM, AngelLoader will update its "selected" indicator in-app.

Notes & Issues:

Deleting FMs:
Deleting TDM FMs is not supported yet. I notice TDM itself doesn't allow you to delete FMs either; it has been noted that people have accidentally lost their data and so the option was removed. AngelLoader puts files into the recycle bin when it deletes them, and I could add the option, but figured it was okay to leave it out initially since the game doesn't let you do it either.

Note, though, that you can still delete TDM FMs from AngelLoader's database if you've manually deleted the actual FM from disk first. To do this, first click the "Show only unavailable FMs" button (red X page icon on the top bar).

Last Played and Finished-On-Difficulty:

Finished-on difficulty is now autodetected. Last played date is taken from the game's database only if AngelLoader doesn't have one in its own (its own is more granular).

In the FMs folder, there is a file missions.tdminfo which stores certain pieces of data about FMs. Included among this data is which difficulty the mission has been finished on, and the last played date (down to the day only). AngelLoader also stores this information, but it doesn't get these values from this file at the moment. Instead, the last played date (down to the instant, not just the day) is set whenever an FM is started, and the finished-on state must be set manually.

I could have it watch the file for changes and update the finished-on-difficulty automatically for the current FM. This would override the user setting, but in theory should be accurate anyway. I could also auto-update the last played date, but because it's only down to the day, that would be less optimal than just setting it on FM start, which would give a more granular date and would have sorting working better. So if you played two FMs in one day, the latest played one would sort at the top.

Mods tab is disabled:

For NewDark games, there is built-in support for mods, and they can be enabled or disabled per-FM in an official capacity. I've disabled the Mods tab for TDM because - from a cursory look anyway - I don't think there's such a per-FM method of mod management.

Language selection is disabled:

In the Edit FM tab, there's an option to choose which language to play an FM with. I've disabled it for TDM because there's a language option in-game (not the case with NewDark) and I haven't looked into how I would scan a TDM FM for supported languages anyway.

"Play Without FM" Option:

For the other games, there is the option (in the bottom bar) to play them with no FM. For TDM there is no such thing as "no FM", but the option is still there currently. It will simply start TDM without passing it anything, and whatever FM it has selected will be there. I might remove the option since it doesn't really make sense. This might make people wonder "where's the button for TDM" but on the other hand having it isn't really useful. Not sure.

Mission downloading and updating:

An in-app mission downloader is in progress.

There is no mission downloader or anything like that currently. It simply lists what you've got on disk. This means that to get new FMs or check for updates to existing ones, you still have to go in-game. I don't know how people feel about the convenience factor of this, having to go in-game to get your FMs and then back out to AngelLoader. However, duplicating the TDM download functionality would mean it might well break if TDM ever changes anything about its downloader.

Custom resource stats:

In the Statistics tab, custom resource display is disabled for TDM. I haven't looked into what sort of "custom" things a TDM FM can have and how I can detect them. I may do this in the future.

 

Edited October 31, 2023 by FenPhoenix
AngelLoader v1.7.6 released

[PsymH]

Great to know, that my favorite fm loader now supports TDM. For managment issues I still prefer (new-)dark FM loader tools like AngelLoader, because I can sort and store notes as I wish. I'll gladly test it in the near future.

[Araneidae]

Looks nice.  Is there any prospect of running it on Linux?

[FenPhoenix]

22 minutes ago, Araneidae said:

Looks nice.  Is there any prospect of running it on Linux?

You can try with Wine (some have had success, though there may be visual issues or other imperfections). Oops, I guess you can't use Wine because TDM on Linux will be an actual native version, which AL presumably would not be able to interact with. Other than that, sorry, I'm pretty tied to Windows for AL as a whole. A separate app for TDM only would be much easier to port to Linux, no need for the Windows-only RichTextBox and no need to interact with Wine from the Linux side programmatically either. One could use the Avalonia UI framework for that, I would think. Maybe I'll give it a shot sometime, but for now I'll polish up TDM support on current AL.

Edited October 13, 2023 by FenPhoenix

[nbohr1more]

Not to be a nag, but I was thinking about the columns problem.

If you go to the view source tab in the wiki article:

https://wiki.thedarkmod.com/index.php?title=Fan_Missions_for_The_Dark_Mod&action=edit

The raw table data is accessible directly:

|-
!align=left|{{TDM-FM|written|Written in Stone}}
|Bikerdude, Amadeus, Dragofer
|{{Forumlink|https://forums.thedarkmod.com/index.php?/topic/21265-written-in-stone-beta-210-only-20220128/}}
|2022-01-28
|338
|Yes
|Yes
|CCC 22, Elixir
|City Missions
|Undead, Horror Themes
|-

Each pipe character represents one of the columns.

[FenPhoenix]

But I mean you want me to take a dependency on parsing the edit page of a wiki article though? And trusting it to stay there, stay updated, and the format not change? Heck, the page even changes depending on if you have edit permission or not, and I have no way to know about updates either without just constantly re-downloading it.

[snatcher]

Many thanks for supporting The Dark Mod, @FenPhoenix !

Web searches narrow results by site ttlg.com, you could make an exception for TDM FMs...

EDIT - Ouch, my bad: it can be configured in the settings. Regardless, perhaps this is the time to have per-game search results settings with some defaults.

Edited October 13, 2023 by snatcher
User error

[FenPhoenix]

6 hours ago, snatcher said:

Many thanks for supporting The Dark Mod, @FenPhoenix !

Web searches narrow results by site ttlg.com, you could make an exception for TDM FMs...

EDIT - Ouch, my bad: it can be configured in the settings. Regardless, perhaps this is the time to have per-game search results settings with some defaults.

I noticed that too. I'm definitely wanting to add a second URL field at a minimum, or per-game as you say. It'd be cool to be able to just go straight to the forum thread too, which I could do if I got it from that wiki page. I'm just nervous about getting stuff programmatically from the edit section of a wiki page, that just seems like it could change and the url is like some &action=edit which isn't an explicit filename (it's a command to generate a page maybe?) so maybe that could change too? If the text in the source code box were accessible by itself in some explicit file I guess I'd feel less apprehensive. Then again I'm totally inexperienced when it comes to web stuff.

For example, http://missions.thedarkmod.com/available_missions.xml is where TDM itself gets the available FMs list from, though that list doesn't have any forum links or much extra info. Although incidentally I could use it to populate the FM titles, authors and dates instead of a local scan, maybe get a bit more accuracy.

[AluminumHaste]

Oh hey Fen! Been enjoying your TDM playthroughs.

[FenPhoenix]

Added per-game web search urls and defaulted TDM to site:thedarkmod.com.

I'm working on downloading/parsing FM information and seeing what I can do with it. I really think an in-app downloader would make things a lot more convenient.

[stgatilov]

3 hours ago, FenPhoenix said:

Added per-game web search urls and defaulted TDM to site:thedarkmod.com.

I'm working on downloading/parsing FM information and seeing what I can do with it. I really think an in-app downloader would make things a lot more convenient.

The description of missions on server is located here:

• http://missions.thedarkmod.com/available_missions.xml (cvar tdm_mission_list_urls).

Every mission has "id" field, you can put it into template to see details:

• http://missions.thedarkmod.com/get_mission_details.php?id=25 (cvar cv_tdm_mission_details_url)

And from this XML you can put screenshot path into yet another URL to download screenshot:

• http://missions.thedarkmod.com/screenshots/25_builders_blocks_0.jpg (cvar tdm_mission_screenshot_url)

These are cvars in engine:

idCVar cv_tdm_mission_list_urls("tdm_mission_list_urls",	"http://missions.thedarkmod.com/get_available_missions.php;http://missions.thedarkmod.com/available_missions.xml", CVAR_GAME, "The URLs to check for the mission list XML." );
idCVar cv_tdm_mission_details_url("tdm_mission_details_url", "http://missions.thedarkmod.com/get_mission_details.php?id=%d", CVAR_GAME, "The URLs to check for the mission details XML." );
idCVar cv_tdm_mission_screenshot_url("tdm_mission_screenshot_url", "http://missions.thedarkmod.com/%s", CVAR_GAME, "The URL template to download the mission screenshots." );

This interface for missions database has not changed in ages.

Everything else is kinda additional/unofficial information.
Maybe we'll add it to "details" XML in future, maybe not.

Parsing wiki page is a bad idea for a normal application.
But I think someone else did it in order to create a website with the list of missions.

[stgatilov]

On 10/13/2023 at 12:03 AM, FenPhoenix said:

Deleting FMs:

Deleting TDM FMs is not supported yet. I notice TDM itself doesn't allow you to delete FMs either; it has been noted that people have accidentally lost their data and so the option was removed. AngelLoader puts files into the recycle bin when it deletes them, and I could add the option, but figured it was okay to leave it out initially since the game doesn't let you do it either.

Note, though, that you can still delete TDM FMs from AngelLoader's database if you've manually deleted the actual FM from disk first. To do this, first click the "Show only unavailable FMs" button (red X page icon on the top bar).

There was an in-game button to delete a mission.

It was removed because mappers sometimes accidentally clicked it for their WIP missions, which resulted in removing months of their work.
Since the feature itself is not much needed anyway, we removed it.

In principle, you can simply delete the fms/qwerty directory manually.
One possible issue is that a record in missions.tdminfo will remain. Maybe it will be deleted on next TDM start (there is some kind of cleanup), maybe not --- then it will retain some info when you reinstall the mission back.

[FenPhoenix]

1 hour ago, stgatilov said:

The description of missions on server is located here:

• http://missions.thedarkmod.com/available_missions.xml (cvar tdm_mission_list_urls).

Every mission has "id" field, you can put it into template to see details:

• http://missions.thedarkmod.com/get_mission_details.php?id=25 (cvar cv_tdm_mission_details_url)

And from this XML you can put screenshot path into yet another URL to download screenshot:

• http://missions.thedarkmod.com/screenshots/25_builders_blocks_0.jpg (cvar tdm_mission_screenshot_url)

These are cvars in engine:

idCVar cv_tdm_mission_list_urls("tdm_mission_list_urls",	"http://missions.thedarkmod.com/get_available_missions.php;http://missions.thedarkmod.com/available_missions.xml", CVAR_GAME, "The URLs to check for the mission list XML." );
idCVar cv_tdm_mission_details_url("tdm_mission_details_url", "http://missions.thedarkmod.com/get_mission_details.php?id=%d", CVAR_GAME, "The URLs to check for the mission details XML." );
idCVar cv_tdm_mission_screenshot_url("tdm_mission_screenshot_url", "http://missions.thedarkmod.com/%s", CVAR_GAME, "The URL template to download the mission screenshots." );

This interface for missions database has not changed in ages.

Actually I'd already figured that part out from the source code and a bit of trial-and-error, but thanks nonetheless!

I do have one question. The tdm_mission_list_urls cvar has two urls:

http://missions.thedarkmod.com/get_available_missions.php
http://missions.thedarkmod.com/available_missions.xml

They both appear to lead to the exact same xml file, except the first one has an xml header and the second one doesn't. From looking at the code it seems like it tries them in order. Should I be doing this too? Are they just mirrors and one might be down? Since they're both on the same domain it seems odd they'd be mirrors, surely if one was down the other would be too, but I dunno.

Edited October 14, 2023 by FenPhoenix

[stgatilov]

1 hour ago, FenPhoenix said:

I do have one question. The tdm_mission_list_urls cvar has two urls:

http://missions.thedarkmod.com/get_available_missions.php
http://missions.thedarkmod.com/available_missions.xml

They both appear to lead to the exact same xml file, except the first one has an xml header and the second one doesn't. From looking at the code it seems like it tries them in order. Should I be doing this too?

I think php links are for the "mission database" on TDM website.
You can go there and browse released missions without downloading TDM.

If not, then php is just some remnant of the past

I would say: better use XML links.

[Taffingtaffer]

On 10/13/2023 at 12:11 PM, FenPhoenix said:

You can try with Wine (some have had success, though there may be visual issues or other imperfections). Oops, I guess you can't use Wine because TDM on Linux will be an actual native version, which AL presumably would not be able to interact with. Other than that, sorry, I'm pretty tied to Windows for AL as a whole. A separate app for TDM only would be much easier to port to Linux, no need for the Windows-only RichTextBox and no need to interact with Wine from the Linux side programmatically either. One could use the Avalonia UI framework for that, I would think. Maybe I'll give it a shot sometime, but for now I'll polish up TDM support on current AL.

Someone referred me to this: https://wiki.winehq.org/FAQ#How_do_I_launch_native_applications_from_a_Windows_application.3F

Dunno how that would help on a Windows app, but just so you know.

[datiswous]

---

Edited October 17, 2023 by datiswous

[FenPhoenix]

Beta 7

• TDM FM Scanner now uses a hybrid of local and server data for more accuracy. Titles, authors, and release dates improve.
• AngelLoader will auto-refresh its FMs list when new pk4 FMs are added manually.
• Localization packs (*_l10n.pk4) in the fms directory are now ignored: the game moves these to their appropriate folders on next start.

I reckon this is now pretty usable. I'm working on an in-app downloader, but until that's done, you can use the game to download FMs and AngelLoader to launch them and manage their metadata.

[datiswous]

On 10/14/2023 at 9:44 PM, Taffingtaffer said:

Someone referred me to this: https://wiki.winehq.org/FAQ#How_do_I_launch_native_applications_from_a_Windows_application.3F

Dunno how that would help on a Windows app, but just so you know.

Possibly related?

https://wiki.winehq.org/Winelib_User's_Guide

Quote

What you gain by recompiling your application with Winelib is the ability to make calls to Unix APIs, directly from your Windows source code.
This allows for a better integration with the Unix environment than is allowed by running an unmodified Windows application running in Wine.

 

Edited October 17, 2023 by datiswous

[FenPhoenix]

That's only for native C and C++ I guess.

[Filizitas]

This is a really great idea... Somewhat wants me to play through all missions like in thief 3... just to get my list with all stars.

Really really awesome!

[datiswous]

5 hours ago, FenPhoenix said:

That's only for native C and C++ I guess.

Where does it say that?

Quote

1.2 System requirements

The requirements for Winelib are similar to those for Wine.

 

[FenPhoenix]

Quote

Thanks to the above, Winelib supports most C and C++ source code

No mention of anything else, and

Quote

The main difference is that the compiler becomes much more important. It is highly recommended that you use gcc, g++, and the GNU binutils.

Those are C++ compilers. No mention of .NET anywhere on the page either.

[FenPhoenix]

Anyway, Linux is backburner until I get TDM support release-ready and an AL public release is out. AL on Linux would be awkwardly in between worlds, supporting 4 games that must run on Wine, and one game that's native. That sounds like a headache any way you look at it.

I know some people have been able to run AL on Wine, but from screenshots I've seen it doesn't even look right, some of the GDI drawn stuff looks janky without the high quality scaled filter it's supposed to have, and dark mode doesn't fully work either afaik, etc. So I don't think AL on Wine is that great a solution in the first place, even if it could interface with native TDM.

Now, AL as it stands is stuck on Windows - and WinForms at that - because of the need for the RichTextBox (WinForms has the only acceptably performing one). Making a native Linux version would need a cross-platform UI, so Avalonia or something else as long as it does theming and a virtualizable datagrid I guess. And then I would have to remove rtf support for that version, and then interface with Wine for Thief but native for TDM. Like I said, sounds like a huge headache.

However, a separate Linux app that only supports TDM but otherwise has all of AL's functionality would be MUCH simpler to make. AL's code is already modern-.NET-ready except for the UI, so I would "just" have to learn Avalonia or whatever, and then simply copy AL's code wholesale and rip out anything not TDM-related (which is a large portion of it). That would lessen the "one-stop shop" factor, but would still be way more feature-rich than the in-game mission manager. But that's for the future if I feel like taking it on. For now, I'm just going to get TDM working at a public release level for AngelLoader and offer my sincere apologies to Linux users.

[nbohr1more]

LOL! Don't apologize for volunteer work...

( There are already enough entitled belligerents yelling at TDM devs for not adding features. )

Thank you for your efforts!

[FenPhoenix]

Beta 8

Some fixes...

• Selected TDM FM was not updated on refresh from disk.
• Selected TDM FM was not updated if the FM was deselected in-game.
• An optimization was broken which caused a refresh from disk even when unnecessary.
• We no longer pass the FM as an argument to the game, but rather simply write it out to currentfm.txt. This avoids an issue where if you pass an FM as an argument to the game (+set fs_currentfm some_fm), then you deselect the FM or select another one, upon game restart it will re-select the FM you initially passed, thus effectively preventing any change of selection unless you quit and start again manually.