Jump to content
The Dark Mod Forums

Some enhancements to the Internal Downloader?


Recommended Posts

I have been looking a bit into the code for the internal downloader, and I wonder if some of the enhancements that I have implemented would be of interest to the TDM team.


1. Right now the downloader is not able to detect if a localization pack has become available since the mission got downloaded earlier.

So, the user would never spot that in the page with the available downloads.

Either he would need to download them manually or delete the entire mission and download it from scratch.

(Some simple recreation steps: go to 'In the North' and delete the l10n pack. If you go to the downloads page, there will be no indication that you are missing something)

In the version of the gamex86.dll I am running, the downloader will indicate that mission and allows you to effectively download only the L10N pack (assuming the mission itself is up-to-date already).


Once the localization packs may become available, this should help to get them onto the players TDM installations.


2. The downloader picks any download URL by random. In case it chooses an HTTPS url, it will produce an error message.

My version ignores those HTTPS URLs completely, since they are not useable anyway by the downloader.

Thus, avoiding confusing error messages being presented to the user.


3. Currently the 'version' of the missions is limited to a simple INTEGER.

How should the downloader be able to tell you that mission X has been updated from version 1 to version 1.1 or 1.0.1 ?

It can't tell you!

Actually part of the problem is that the XML received from the server only reads like "1", "2", or "3".

Why is that ? I think that should be changed on the server (and probably in the database) to read a full version string.

The gamex86.dll I have deals with strings just fine. I tested that by redirecting the downloader to a local XML file where I changed the available_missions.xml to read versions like "1.1" instead of a plain "1".

So, if the version handling on the server gets modified to provide proper version info (and I would say that that would make pretty much sense), the internal downloader would be ready for it.


Any interest?

Someone wants to review the code?

  • Like 2
Link to comment
Share on other sites



Thanks for looking over this code, it's clear it needs some love! I think taaaki might be interested in checking out your patches :)



2 - This is not a limitation of the downloader which is essentially just curl, but rather of the compiled lib that someone produced, which seems to exclude SSL support. This should rather be fixed by rebuilding with SSL support enabled.

3 - We really need an 'official' TDM FM versioning system. Ints might be a bit too limiting, but I don't think free-for-all is any better. Maintaining packages, I have found out that if you give people the ability to do whatever they want with versions, they will make you lose faith in humanity. However this is a discussion for another thread :)

Link to comment
Share on other sites

3 - We really need an 'official' TDM FM versioning system. Ints might be a bit too limiting, but I don't think free-for-all is any better. Maintaining packages, I have found out that if you give people the ability to do whatever they want with versions, they will make you lose faith in humanity. However this is a discussion for another thread :)

No, you are absolutely right. Handling versions however you feel like won't work.

Certain standards are required.

And this thread would be the right one to discuss it ... probably.

Neither handling version numbers as a sequence of Integers indicating the Major.Minor version would produce the 'intended' results (1.01 == 1.1),

nor does handling them as String (1.10 < 1.9) really provides the final solution.

So, certain standards would be required... even if they are applied only to the server entries and would require disrespecting the version scheme some authors may choose in their readmes.

But the current handling is really not sufficient in any way at all.

Edited by gnartsch
Link to comment
Share on other sites

Isn't there hashing in tdm_update.exe?


In a way, yes, but they are CRC32 checksums. Easily manipulated and rather just good for checking that a data arrived in good order; Not so good for working out if the file is the one you wanted :)(Which actually matters due to the way we bind to dynamic libraries, which I'm not so chuffed about either ;/)

Link to comment
Share on other sites

Can't you just use hashfunctions which are built into the OS?


Also, please don't use MD5.

Built in where? Each OS handles crypto very differently, and avoiding shells - pretty annoying. Alternative is to use OpenSSL, but that's just overkill.


There are a few reasonably good, 1/2 file implementations floating around, which would be far easier and portable. And no, no MD5/TTH.

Link to comment
Share on other sites

Well, seems like none of the autoritative TDM gurus has shown up here, so I do not even know if it would make any sense at all trying to apply for an SVN account..

Anyway, here is the code (3 classes) and the updated gamex86.dll.

My modifications are on top of the 2.0 release code and my changes are indicated by comments (//gnartsch: plus the topic that is addressed)

Maybe someone will make good use of it.


With that, missing L10N packs are indicated by (#) and you can simply download only those missing packages, which should turn out to be usefull once the L10N packages are made availlable.





The entire discussion about hash support is sort of reasonable, but opens a whole new can of worms.


First of all the server can not tell you what the expected hashvalue would be.

So, that would have to be addressed on the server in the first place.

And I am afraid that this would be a manual task (someone needs to maintain the hashvalues manually in the mission DB), since no mission is located on the official TDM server itself.

All mirrors are 'remote' servers and simply getting a hashvalue on the fly is therefor impossible.


Second, how would you know that the server provides you the hashvalue for the latest version of a mission?

In case you had manually downloaded the latest version, because the server did not get updated, yet, the server would claim having a newer version.

The only way to fix that would be to include some checking on the version of the mission as well, which right now is fairly course grained, but should be improved as well anyway.

Edited by gnartsch
  • Like 1
Link to comment
Share on other sites

Merged the supplied code minus the pieces of code that would exclude https links. I'd rather fix the https problem with the downloader than just avoid it.


Also busy adding an extra legend ("# Translation") to the mission downloader gui just below "* Update". Will commit code and gui + strings updates when I am done.


Edit: on the topic of the mission hashes, etc., I think a bit more work can be done on the mission database to make this easier. Ideally, it would be nice if there was an interface to submit a mission (or an update of a mission), have the mission submission page calculate a SHA256 hash and include the hash in the mission details db so that clients can easily compare the local hash to the one provided by the master server (remember that the mission db is only provided by one server, while the actual mission packages themselves may be distributed).

I am the bat. The night is mine.

Link to comment
Share on other sites

Thanks guys, I really appreciate that!

With respect to https :

I perfectly agree that enabling https would be a better solution, but unless support for that is in,

I feel that having errors being reported during download, because the downloader did chose an https mirror by random, does not really help at all.

So, that temporary fix - in case support can not be added - would deserve to stay in as well.


With respect to updates to the database schemas:

I feel like the L10N packs would actually need some versioning as well.

Translalations for new missions would probably pop in over time for various languages.

And it would be good to know that there is an update as well.

Link to comment
Share on other sites

Well, I'm hoping to resolve the lack of SSL support in CURL before the next release. If I can do that, I will add your workaround.


Anyway, thanks for all your efforts B)


I'll need to ask for more details regarding the current mission db system to see what can be done to add better hash support etc. I'm not sure if versioning is actually required since we can assume that the hash supplied by the mission db is always the latest one (in theory).

I am the bat. The night is mine.

Link to comment
Share on other sites

I'm not sure if versioning is actually required since we can assume that the hash supplied by the mission db is always the latest one (in theory).


Well,... in theory you might be right.

But there are always situations where the servers are not up to date for quite a couple of weeks, e.g. because the author did not notify the admin crew about an update.

And people did download right from the thread instead.

So, the server reporting a different checksum would only tell you that there is a different version availlable, but not whether it would be a newer or older one.

At the end, the checksum would only serve to verify that the actual download was the expected version (and not an outdated mirror or broken file) in case you decided to download anyway.


In order to close that gap, I would guess a dedicated thread for 'New Missions & Update Announcements' would ideal.

Everyone (but preferably the authors) could point out the latest version of a mission and the admins could simply read that thread instead of crawling through the tons of mission threads where such announcements may be found occasionally.

Edited by gnartsch
Link to comment
Share on other sites

  • 5 months later...

For the problem of the internal downloader barely ever noticing that a mission has been updated, a fairly simple fix should be possible.


Let me recall the current problems first:

- the VERSION field is a simple Integer, which barely ever gets updated, since missions most of the time receive only updates to the minor version

(e.g. 1.0 -> 1.0.1 and then 1.1, etc)

Because of that the mission-database still reads nothing else but the major version, and the downloader is clueless abouzt any updates,

- to make it more complicated, there is no consent about how the version schema should look like.

Some author insist on having arbitrary conventions, such as 1.1 --> 1.1b or 1 --> 1.01 --> 1.1

Figuring out how to detect which version would be newer would actually become a mess.


- Because of these limitations, the version number doesn't get updated in the database and players are asked to manually delete the previous version from their harddrive



Current MissionInfo in the XML:

<mission id="14" title="Business as Usual" releaseDate="2010-01-14" size="16" version="3" internalName="business3" type="single" author="Bikerdude">



Now, the solution I propose for these problems would be quite easy, and at the same time would allow authors to use whatever custom versioning schema they want.


- The Integer VERSION should be changed to an INTERNAL_VERSION number, which doesn't require to be visible to any user.

It will in fact be nothing else but a counter which gets updated with each new version.


- A new String DISPLAY_VERSION will be added, which can hold whatever the author wants it to read like. This field will be displayed whereever the user can SEE a version number.


With that you have the flexibility to make sure you get the updates to the players without giving away anything.

And another bonus is, that the actual Update-Check in the Internal Downloader does not need to be touched at all,

because the logic to detect a higher version number on the server is already there.

Only the few places where users can actually see the VERSION info need to be updated.




New MissionInfo schema:

<mission id="14" title="Business as Usual" releaseDate="2010-01-14" size="16" internal_version="3" display_version="3.0" internalName="business" type="single" author="Bikerdude">

After next Update:

<mission id="14" title="Business as Usual" releaseDate="2010-01-14" size="16" internal_version="4" display_version="3.1" internalName="business" type="single" author="Bikerdude">

After yet another Update:

<mission id="14" title="Business as Usual" releaseDate="2010-01-14" size="16" internal_version="5" display_version="3.1.1" internalName="business" type="single" author="Bikerdude">

<mission id="14" title="Business as Usual" releaseDate="2010-01-14" size="16" internal_version="6" display_version="4.0" internalName="business" type="single" author="Bikerdude">

Edited by gnartsch
  • Like 1
Link to comment
Share on other sites

  • 10 months later...

It would be really nice to see the FM versioning being improved some day. (see previous post)


Just yesterday I updated my TDMMissionDb checker tool.

Originally designed to check that all FM mirrors are consistent, it can now also check your local fms folder against the list of official FM mirrors.


As a result I identified a bunch of FM updates, that the internal downloader was not able to inform me about.

(although the original versions had been downloaded that way as well and the 'version' in 'fms/missions.tdminfo' was set)




Simply extract somewhere and make sure you have Java v 1.6 or higher installed.



Call it with your fms folder specified in order to have your local FMs checked against the version(s) on the server.


TDMMissionDbCheck.bat d:\games\darkmod\fms


Note that issues with local fms go to the console window only, whereas the results of checking the consistency of the official FM mirrors goes into the file analysis.txt


If you receive something like the following in the console window, then you know that your local copy is either behind or ahead of what is available on the official mirrors.

Even if the internal downloader does not inform you about an available update.


Mission # 75 - vfat2 : Vengeance for a Thief Part 2
Mismatching local FM size!

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recent Status Updates

    • STiFU

      We are taking our son on his very first holiday trip to see the sea for the first time. 🙂 Will be back in a week.
      · 0 replies
    • Gilkar

      When I was a young man my father was so ignorant I could hardly stand to have him around. As I grew older I was amazed at how much the old man had learned in such a short time.
      · 1 reply
    • jaxa

      RTX 3090 Super, RTX 3070 Ti 16 GB, RTX 2060 12 GB
      · 0 replies
    • duzenko

      CPU benchmark time - compiling DarkRadiant (2nd run)
      i5 8600K 6C/6T@4.4GHz DDR4 2x2133MHz 9MB cache
      Parallel builds: 1. 3:57 Parallel builds: 6 (default). 2:28 r5 1600AF 6C/12T@3.3GHz DDR4 1x2666MHz 16 MB cache, temp folder on HDD
      Parallel builds: 1. 5:05 Parallel builds: 4. 2:47 Parallel builds: 6. 2:55 Parallel builds: 12 (default). 2:57
      · 6 replies
    • nbohr1more

      Status updates are back so it is also a good time to return to contests!
      · 0 replies
  • Create New...