Jump to content
The Dark Mod Forums
Sign in to follow this  
NightStalker

TDM Updater Fails To Create TDM 2.07 Version

Recommended Posts

There's some sort of problem with the TDM updater whereby it won't create a proper 2.07 installation.

Being on a slow Internet connection, my TDM installation/update procedure has always involved downloading just the latest update zip file (currently, 'tdm_update_2.06_to_2.07.zip') from the ModDB site and making that available to the updater locally, to avoid unreliable, slow downloads during the actual update.

I followed the same procedure that I've always (successfully) done in the past:

  1. extract the huge, 2.3-GB 'THEDARKMODVersion2.0-StandaloneRelease.zip' file (downloaded once from ModDB and re-used many times over the years)
  2. change the permissions on all the PK4 files to be write-able (so that the updater can alter the content within, as needed)
  3. delete the existing, obsolete (2013-era), read-only 'tdm_update.log' file (which shouldn't be in the zip file to begin with)
  4. put a copy of all the 'tdm_update_2.0{x}_to_2.0{x+1}.zip' files amassed to date into that extracted directory
  5. download the latest version of the TDM updater app (from the TDM website)
  6. run the updater, with the options '--keep-update-packages --noselfupdate'

 

On the 1st run, it proceeds in what appears to be normal fashion. There's no obvious indication that anything has gone amok. However, if you run the game, it comes up with the main menu showing "TDM 2.06", not the expected "TDM 2.07".

Here's the TDM updater log file for this 1st run:

tdm_update--run-1.log.txt

Some analysis of the directory content and the updater's log file seems to indicate that the installation only made it to 2.06!

As for the 'tdm_update_2.06_to_2.07.zip' file, I've verified the integrity ('zip -T'). I've also verified the filesize and checksum against the updater-downloaded 'tdm_version_info.txt' file:

[UpdatePackage from 2.06 to 2.07]
crc = 159677cb
filesize = 427644498
package = tdm_update_2.06_to_2.07.zip

So I don't think there's anything wrong with that particular file.

If I compare what the updater created as a "2.07" installation with an old, virgin 2.06 installation of mine, the only files that are different are ones that you'd expect to be: 'crc_info.txt', 'tdm_mirrors.txt', 'tdm_update.log', and 'tdm_version_info.txt'.

If I run the updater a 2nd time (and all subsequent times) with the same options, it essentially does this:

 Applying differential update package...
 [=========================] 100.0% Adding: ExtLibs.dll                        
 [=========================] 100.0% Adding: ExtLibsx64.dll                     
 [=========================] 100.0% Adding: TheDarkModx64.exe                  
 [=========================] 100.0% Adding: thedarkmod.x64                     
 [=========================] 100.0% Adding: vcredist_x64.exe                   
 [=========================] 100.0% Adding: vcredist_x86.exe                   
 [=========================] 100.0% Replacing: AUTHORS.txt                     
 [=========================] 100.0% Replacing: TheDarkMod.exe                  
 [=========================] 100.0% Replacing: tdm_update.exe                  
 [=========================] 100.0% Replacing: tdm_update.linux                
 [=========================] 100.0% Replacing: thedarkmod.x86                  
 [=========================] 100.0% File:                                      
 Done applying the differential update.

Due to website file upload limitations, the TDM updater log file for this 2nd run will be attached to a subsequent post.

The updater log files on that 2nd and all subsequent runs are essentially identical, just with different timestamps and potentially use of a different mirror site.

That's the repeatable scenario, the "crux of the matter", if you will. FWIW, I also dug more deeply into the problem. What I describe from here on is less thoroughly documented, but I'll mention what I did, in case it helps....

I tried removing all of the update zip files, hoping it would force the updater to update what is effectively a 2.06 installation but via a slow, painful download of that 408-MB 'tdm_update_2.06_to_2.07.zip'. But instead, it inexplicably wanted to download the 235-MB 'tdm_update_2.05_to_2.06.zip' file, so I aborted the update.

Taking a different tack, I dug into the updater source code a bit and noticed this curious-looking snippet in '.../tdm_update/libtdm_update/Updater/UpdateController.cpp':

//#4529 stgatilov: Since differential update does not uncompress unchanged OGG/ROQ files,
//The hashes of pk4 files are now different from the ones on the server.
//That's why we exit right away to avoid redoing the same differential update and redownloading the unchanged data.
if (info.fromVersion == "2.05" && info.toVersion == "2.06") {
        TryToProceedTo(PostUpdateCleanup);
        break;
}

I played around with various changes to that code (both bumping the 2 version numbers and outright disabling the whole block), but to no avail.

Although I'm running Linux (64-bit Slackware 14.2), I don't immediately see any reason why this would be OS-specific. This issue sounds different from the problem grayman had (running Windows), but I did get an "infinite updater loop" scenario when I hacked that code mentioned above, so maybe there's some correlation.

I also noticed during some debugging that the installation I had was not being correctly recognized by the updater app as a valid 2.06 installation. When I looked into the updater's console output and log file, it seemed that every file about which the installer reported "SIZE MISMATCH" was one of the PK4 files that had embedded OGG files, although I did not check every case.

I'm simply not able to figure out what's going on here, despite much effort, so I have to "punt" before my head explodes. :o Any input on this matter would be very much appreciated!

  • Like 1

Share this post


Link to post
Share on other sites

Fortunately, the details in that bug report mentioned in that code snippet have shed more light on the problem.

After reading it, I realized that there may be some sense in actually (counter-intuitively, I'd argue) running TDM (the game) after the 1st (failed) run of the TDM updater (where it went from 2.00 -> 2.06 instead of 2.07). So I tried that and saw evidence that the game was "Repacking" various PK4 files (presumably the ones containing ROQ/OGG/AVI files).

Then I re-ran the same updater process with the same options.

It acted differently this time, (needlessly?) downloading all 3 of the missions that come with a default TDM installation as well as (needfully, I suspect) 'tdm_base01.pk4'.

But when it was done, the installation did appear to be a true 2.07 install, fortunately. And, in fact, the game now identifies as "TDM 2.07" on the main menu and has 2.07 features, like the new "frob highlighter" (nice addition BTW, STiFU!).

So, although my immediate dilemma seems to be solved, it sounds like anyone who tries to do the upgrade the way that I did (regardless of OS) will encounter this odd glitch, which requires a rather unintuitive work-around. Furthermore, unless a new, large "base" TDM zip file that's built for 2.06 or higher is released, it will continue to be a problem.

It seems to me like this problem would be affecting all new players, ones who have no 2.06 installation to build upon. Or am I missing something?

I don't have the Internet throughput to test it, but I wonder if things would work any differently if I didn't supply a local copy of all the 'tdm_update_2.0{x}_to_2.0{x+1}.zip' files to the updater, thereby (presumably) forcing each update file to be downloaded and applied, 1-by-1.

More broadly, just how do other users (Windows or Linux -- I don't think it matters) install TDM "from scratch"?

Share this post


Link to post
Share on other sites

Typical installation "from scratch" is downloading the more recent version (currently 2.07) directly.

This is what updater does if you run it in an empty directory.

If you badly want to download it directly, I can give you the filelist (i.e. what you should download). You can download packages directly, unpack tdm_shared_blabla.zip and play.

 

Another use case which is supported is the goddamn "differential update", when players upgrade from previous version.
In such case updater downloads single differential package and applies it.

 

The way you describe with installing 2.0 and applying differential updates in sequence is almost never used.

I bet you are the only one who have tried it so far.

 

The reason for the issue around 2.06 that you saw is that starting from 2.06, video and ogg files are stored uncompressed inside pk4.

Aside from faster loading times, this is pretty much required for FFmpeg videos to work properly.

The whole recompression story is a mess, and it gets more complicated because our current updater has two completely separate ways of functioning.

I'm glad it even works if player runs TDM at least once after applying each differential package, and I prefer leaving it as it is.

 

The theoretical plans for new updater take recompression problem into account.

 

Share this post


Link to post
Share on other sites

Many thanks for the info! I'd love to see a list of the files to download to make a complete 2.07 install. But even more than that, I'd like to know how you create such a list (more on that below).

Occasionally, for debugging, I find it useful to have run-able, pure, older TDM versions. So I'd also like to see such a file list for 2.04 - 2.06, although I suspect that it may not be possible or might be impractical if said files no longer exist (i.e. packaged with release-specific, older content, for ready-made download).

Using a crude hack, I've already managed to create "virgin" installations of older TDM releases, simply by manually stopping the incremental update at the proper step. But creating such virgin installations with an actual list of requisite files would be much nicer.

In an attempt to answer my own "I'd like to know how you create such a list" question, I've just now (years overdue, it seems!) done a quick hack to the updater to do a "poor man's equivalent" of the '--dry-run' option that it has annoyingly advertised (via '--help') all along but which has never actually been implemented! Running the updater with that hack let me see just what files the updater wanted to download, without actually downloading all 3.41 GB. It runs in a mere 14 seconds. Having said that, I'd still really like to see your list of files for 2.07, to be sure I'm "on the same page".

It would be heaven for me if I never again needed to run that updater app, save for maybe once each release (with my hack) to quickly generate a "requisite file list". I'll download all the files, slowly but reliably using 'wget' (or possibly even off-site somewhere with a fast Internet connection). Then I'll write a simple shell script to efficiently take care of the installation, with no Internet access required at that point! :awesome:

A point of interest: I ran the updater in an empty directory, but since it does not retain partially downloaded content on subsequent runs, it's simply not a good choice for downloading large files over unreliable and/or slow Internet connections.

BTW, I had actually looked into trying your new 'tdm_sync' capability. But building it requires a too-new version of CMake, which triggers the need for other updates that I cannot abide at the moment, unfortunately.

Share this post


Link to post
Share on other sites

Thanks for the link, freyk. I read it, but it's the same technique that Durandall already suggested in that same thread that I linked to in post #5 of this thread. And, as I mentioned in my reply to his/her post, that method yields a less-than-pure installation. It's a work-able alternative, but it's cleaner to use the "interrupt the updater" technique suggested by nbohr1more and implemented by me (required code can be found in that same linked post).

 

I'd like to have stgatilov's list(s) to confirm that his 2.07 "requisite file list" matches what I'm seeing the (hacked) updater try to download.

Share this post


Link to post
Share on other sites

Here is the list of files which I upload to Azure mirror (for tdmsync prototype and TDM 2.07):

crc_info.txt
tdm_version_info.txt
tdm_update_linux.zip
tdm_update_win.zip
tdm_shared_stuff.zip
fms/newjob/newjob.pk4
fms/stlucia/stlucia.pk4
fms/training_mission/training_mission.pk4
tdm_ai_animals01.pk4
tdm_ai_base01.pk4
tdm_ai_humanoid_builders01.pk4
tdm_ai_humanoid_females01.pk4
tdm_ai_humanoid_guards01.pk4
tdm_ai_humanoid_heads01.pk4
tdm_ai_humanoid_mages01.pk4
tdm_ai_humanoid_nobles01.pk4
tdm_ai_humanoid_pagans01.pk4
tdm_ai_humanoid_townsfolk01.pk4
tdm_ai_humanoid_undead01.pk4
tdm_ai_monsters_spiders01.pk4
tdm_ai_steambots01.pk4
tdm_base01.pk4
tdm_defs01.pk4
tdm_env01.pk4
tdm_fonts01.pk4
tdm_gui01.pk4
tdm_gui_credits01.pk4
tdm_models01.pk4
tdm_models02.pk4
tdm_models_decls01.pk4
tdm_player01.pk4
tdm_prefabs01.pk4
tdm_sound_ambient01.pk4
tdm_sound_ambient02.pk4
tdm_sound_ambient03.pk4
tdm_sound_ambient_decls01.pk4
tdm_sound_sfx01.pk4
tdm_sound_sfx02.pk4
tdm_sound_vocals01.pk4
tdm_sound_vocals02.pk4
tdm_sound_vocals03.pk4
tdm_sound_vocals04.pk4
tdm_sound_vocals05.pk4
tdm_sound_vocals06.pk4
tdm_sound_vocals07.pk4
tdm_sound_vocals_decls01.pk4
tdm_standalone.pk4
tdm_textures_base01.pk4
tdm_textures_carpet01.pk4
tdm_textures_decals01.pk4
tdm_textures_door01.pk4
tdm_textures_fabric01.pk4
tdm_textures_glass01.pk4
tdm_textures_metal01.pk4
tdm_textures_nature01.pk4
tdm_textures_other01.pk4
tdm_textures_paint_paper01.pk4
tdm_textures_plaster01.pk4
tdm_textures_roof01.pk4
tdm_textures_sfx01.pk4
tdm_textures_stone_brick01.pk4
tdm_textures_stone_cobblestones01.pk4
tdm_textures_stone_flat01.pk4
tdm_textures_stone_natural01.pk4
tdm_textures_stone_sculpted01.pk4
tdm_textures_window01.pk4
tdm_textures_wood01.pk4

They are enough for official installer to update.

Here is the anatomy:
1) crc_info.txt --- updater uses it to check pk4 for being correct. Also contains the file list. Not needed to play.
2) tdm_version_info.txt --- another file used by current differential update only. Not needed to play.
3) tdm_update_linux.zip, tdm_update_win.zip --- these contain updaters which new players download from website. Also used for updater self-update.
4) tdm_shared_stuff.zip --- these are all files which should be at TDM root directory unpacked.
5) fms/*/*.pk4 --- these are three core missions.
6) *.pk4 --- these are ordinary packages.

In order to install manually, you take the URL of some mirror (from official tdm_mirrors.txt, or my Azure mirror), for instance http://darkmod.taaaki.za.net/release/.
Then for each file concatenate the mirror URL with the relative file path from the list above, and download the file from there.

After everything is downloaded, unpack tdm_shared_stuff.zip and you are ready to play.

I don't know any other things tdm_update does. Even if it does, they are most likely unnecessary.

 

Note that all mirrors are simple HTTP websites with public files.

This bunch of files is just uploaded to every mirror with the exact directory structure.

You can download anything as you wish.

You cannot see the files (aka listing) only because it is disabled on all HTTP servers (for avoid bots, I guess).

Share this post


Link to post
Share on other sites

Many thanks for that file list, stgatilov! That's all I really needed, since (with the single exception of that 'tdm_shared_stuff.zip' file) I'm intimately familiar with the TDM installation fileset, having spent countless hours with it during the run-up to the 2.05 release. But, for the benefit of any others and for future reference, I'm glad you thoroughly documented it.

I confirmed that your file list exactly matches what the 'dry run' of the TDM updater showed. Also, there actually is one mirror (http://mirror.helium.in-berlin.de/thedarkmod/release) whose content is view-able. That's what gave me a 2nd list (yours being the 3rd) to confirm that I was on the right track. :D

Share this post


Link to post
Share on other sites

Why would you set the updater itself to no self update?

One reason is to avoid having it overwrite the one you built (i.e. from source code) with changes of your own.

 

I find that it's often a different mindset between Linux users and Windows users, but I (being a full-time Linux user) don't like the fact that any executable I run decides to "self-update" and run itself again. It makes it maddening to debug sometimes. At least the authors had the forethought to include the "--noselfupdate" option.

Share this post


Link to post
Share on other sites

Hi there.

I also had issues updating TDM from 2.06 to 2.07, but on Windows 7.

 

Thinking there might be an issue with my 2.06 install, I also tried use a previous copy of 2.00 which I had downloaded back in the days, and to update that instead: it updated up to 2.06 (seemingly without issue) and then failed to update to 2.07 when I ran the updater again, as it kept downloading and trying to apply the 2.05 to 2.06 patch again.

Here are the logs : updating from 2.00 to 2.06 ; updating from 2.06… to 2.06 again ?_?

 

By the way, I've downloaded tdm_update_2.06_to_2.07.zip using a browser from 2 of the available sources, and noticed that their CRC, according to 7-zip, was 44F83C6C, which does not match the one that's written in tdm_version_info.txt and in the first post.

However:
- both 7-zip and WinRAR confirm that there is no error when testing the archive
- both 7-zip and WinRAR show something which seems a bit weird to me: there appears to be two different copies of scripts\tdm_custom_scripts.script within the same archive (see attachment).

post-38427-0-41533500-1549888514_thumb.png

Edited by Moo
  • Like 1

Share this post


Link to post
Share on other sites

Thinking there might be an issue with my 2.06 install, I also tried use a previous copy of 2.00 which I had downloaded back in the days, and to update that instead: it updated up to 2.06 (seemingly without issue) and then failed to update to 2.07 when I ran the updater again, as it kept downloading and trying to apply the 2.05 to 2.06 patch again.

Here are the logs : updating from 2.00 to 2.06 ; updating from 2.06… to 2.06 again ?_?

I think you have to run the game at least once before upgrading from 2.06 to 2.07.

Then the CRC mismatches will go away, and the next differential update should apply, I hope.

 

By the way, I've downloaded tdm_update_2.06_to_2.07.zip using a browser from 2 of the available sources, and noticed that their CRC, according to 7-zip, was 44F83C6C, which does not match the one that's written in tdm_version_info.txt and in the first post.

A CRC checksum written in crc_info.txt and tdm_version_info.txt is not checksum of the archive itself.

It is a composite checksum of all the files within it if they are uncompressed beforehand.

It is normal that CRC reported by 7-zip does not match, it will not match for any CRC written in those files.

 

However:

- both 7-zip and WinRAR confirm that there is no error when testing the archive

- both 7-zip and WinRAR show something which seems a bit weird to me: there appears to be two different copies of scripts\tdm_custom_scripts.script within the same archive (see attachment).

This is really weird.

I wonder which of them gets installed and how the rogue tdm_custom_scripts.script appeared in the first place.

I cannot even recognize its contents: it is different from both 2.06 and 2.07...

  • Like 1

Share this post


Link to post
Share on other sites

OK, I'll try running the game between the updates to 2.06 and 2.07 then.

 

That seems a bit strange though: if running the game does something special that is somehow required before it can be updated further, shouldn't the updater also be able to perform that action by itself?

 

Thanks for the explanation about the CRC, too :)

 

Edit: can confirm, after updating to 2.06 then running the game once, launching the updater again succeeded in updating to 2.07, and notified me that it still needed to update again (downloaded 4 more files), then it seemed to be completely up to date.

Edited by Moo

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...