Jump to content
The Dark Mod Forums

Prototype of time-travelling tdmupdate


stgatilov

Recommended Posts

There are some ideas about changing the way how differential updates work in tdm_update.
The goal is to make any update differential, even if you local copy is dirty, if you update between betas, or update backwards.

Here is the very early prototype of tdm_update with new algorithm (let's say it is "tdmsync"-based).
Ironically, a better approach has already been proposed, so this tdmsync prototype will die as soon as the better one is implemented.
However, this won't happen soon, and there is still some sense in testing this version, because both algorithms rely on the same underlying idea.
Which is: it is possible to download portions of a file from HTTP server.

Note: the prototype is hacky, and using it would be a rough ride!
It is not related to the 2.07 beta testing, and it is unofficial and unsupported.
Unless you have experience of beta-testing and joggling with TDM versions, better not use it.

 



Instructions for updating:

1) Place the tdmupdate_tdmsync.exe into TDM installation directory which you want to update.

2a) If you want to update to the latest 2.07 beta release, then download tdm_mirrors.txt file just as you usually do for beta-testing.

 

2b) If you want to update to some particular version, then create tdm_mirrors.txt file in the directory with the following contents:

[Mirror azure]
url = http://tdmcdn.azureedge.net/beta207-XY
weight = 1

Replace beta207-XY with the codename of the version you want to get.

The list of codenames is provided in the next post in this thread.

3) Start tdmupdate_tdmsync.exe.

4) Go to Advanced settings, and enable two checkboxes there:
--keep-mirrors (if you forget this, you will download the full 2.06 release of TDM!)
--noselfupdate (if you forget this, you will be switched back to stock version of tdm_update)
Click Save.

5) Now click button Continue>>.

6) It will tell you it is going to download huge amount of data --- ignore that.
Check that the number of files to be downloaded is less than 60 (otherwise you have launched updater in the directory without TDM --- better not do that).

7) Click button Continue>>.

8) The progress reports are rather broken during update.
In order to see what's going on, open the directory with TDM installation being updated, and see how new files appear.
The files with extension .tdmsync are metainfo files (about 30 MB total), the files with extension .bindiff are actual pk4 differences downloaded.

9) When update is over, note the number in Total bytes downloaded: XX.X MB report --- let it be D.
Go to TDM directory and check how much space all .tdmsync and .bindiff files take --- let it be S.
Unless I am wrong, the real amount of data downloaded is (S + D).
For instance, updating from release205 to beta207-01 took me roughly 600-700 MB of download size.

10) Now that you have the version you want, note that:
1. Most likely you have to reset darkmod.cfg. The effect of new config on old version and vice versa is unpredictable!
2. You probably have to run vcredist_x??.exe manually if TDM crashes immediately when started.
3. The updater does not remove new files when updating to previous versions. In most cases it has no effect.
One funny case is that if you downgrade from 2.07 to 2.06, you'll get HRTF still enabled, because alsoft.ini will remain =)

11) You can remove all .tdmsync and .bindiff files: you won't need them ever again.
But please don't delete .zip files: they are needed for future differential updates.




Please don't perform full updates with this updater.
If you want to get a fresh 2.05 installation, better copy one of the TDM installs you have (any version except SVN would be OK), then update it using the tool.
  • Like 1
Link to comment
Share on other sites

Here is the list of available codenames.

 

Official releases:

  • release205
  • release206
  • release207

Beta releases:

  • beta207-01
  • beta207-02
  • beta207-03
  • beta207-04
  • beta207-05
  • beta207-06
  • beta207-07
  • beta207-08
  • release207orig
  • beta208-01
  • beta208-02
  • beta208-03
  • beta208-04
  • beta208-05
  • beta208-07
  • beta208-08
  • Like 4
Link to comment
Share on other sites

As a matter of fact, some sound samples were changed in the second beta (codename: beta207-02).

As a result, many sound packages got modified.

With stock updater, it is necessary to redownload all these pk4 files when updating from beta207-01, resulting in about 370 MB download.

With this prototype, only about 80 MB of data is downloaded.

 

=========================================================================

 

Differential update from beta207-02 to beta207-03 takes 40 MB of download on this experimental updater.

Stock updater downloads 750 MB of data.

Link to comment
Share on other sites

I ran a few tests with tdmsync and the regular updater. When starting tdmsync on win10 x64 I first get a warning that Windows does not allow to execute this unknown app. I can choose to execute it anyway in the "Further information" menu.

 

Note: My S-values are incorrect because I had realized too late that they are also generated inside the fms folder.

 

TDMsync: Performed updates in successive order, all with the same directory

207-01 -> 207-02: D = 116.3 MB, S = 49.6 MB

207-02 -> 207-03: D = 9.7 MB, S = 49.6 MB

207-03 -> 206-rc: D = 1.15 GB, S = 4.57 KB (Had to remove keep-mirrors. Just putting 206 in your supplied tdm_mirrors did not work. Looking at the values, the truely differential update probably did not work this way.)

206-rc -> 207-03: D = 16.3 MB, S = 403 MB (Huh? 16.3 MB while 207-01 to 207-02 was 116 MB??? Also that huge S??? I have attached the log file.)

207-03 -> 207-02: D = 9.7 MB, S = 31.5 MB

207-02 -> 207-03: D = 9.7 MB, S = 31.4 MB

 

206-rc_207-03.txt

 

Note: It's a bit of a bummer that tdmsync destroys my alsoft.ini, but I guess there's no way around that.

 

TDMupdate:

207-01 -> 207-03: D = 811.8 MB

206-rc -> 207-03: D = 403.1 MB + 35.1MB for fms (Updater v069->v072 with restart successfull. It's weird that the update from 206 is smaller than from 207-01, but that's probably a limitation of tdmupdate?)

  • Like 1
Link to comment
Share on other sites

I just started the app to look at the GUI.

If i want to backup backwards, i'm missing a tdm-version selectionobject.

And like the updater, would be nice to add a grandtotal progressbar

 

Got the same smart-screen unkown author/publischer warning, but thats always at first-run.

(got that also for my installer. An author certificate is expensive)

How did you do that for the tdm binary?

Edited by freyk

Info: My portfolio and darkmod graphical installer
Amnesty for Bikerdude!

Link to comment
Share on other sites

I ran a few tests with tdmsync and the regular updater. When starting tdmsync on win10 x64 I first get a warning that Windows does not allow to execute this unknown app. I can choose to execute it anyway in the "Further information" menu.

Yes, I saw this. Doesn't matter right now.

 

Note: My S-values are incorrect because I had realized too late that they are also generated inside the fms folder.

FM files rarely change.

 

207-01 -> 207-02: D = 116.3 MB, S = 49.6 MB

I wonder what those 116 MB megabytes are.

I'm afraid my mirror is working weird sometimes...

 

207-03 -> 206-rc: D = 1.15 GB, S = 4.57 KB (Had to remove keep-mirrors. Just putting 206 in your supplied tdm_mirrors did not work. Looking at the values, the truely differential update probably did not work this way.)

Do not remove keep-mirrors!

You did a very ordinary update to 2.06 release: all pk4 files which were different were downloaded from scratch.

The codename of TDM 2.06 is release206, that should be put instead of e.g. beta207-01 in tdm_mirrors.txt.

Check out the original post, codenames release206 and release205 are available in addition to beta packages.

 

206-rc -> 207-03: D = 16.3 MB, S = 403 MB (Huh? 16.3 MB while 207-01 to 207-02 was 116 MB??? Also that huge S??? I have attached the log file.)

If you have not noticed that, 2.07 installation is about 340 MB larger than 2.06 release (starting from the very first beta).

So this is quite expected download size.

 

207-03 -> 207-02: D = 9.7 MB, S = 31.5 MB

207-02 -> 207-03: D = 9.7 MB, S = 31.4 MB

That's normal.

 

Note: It's a bit of a bummer that tdmsync destroys my alsoft.ini, but I guess there's no way around that.

You can probably try to set OpenAL settings in another place?...

Or remove the need for alsoft.ini with integrating HRTF on/off switch into TDM :D

 

207-01 -> 207-03: D = 811.8 MB

206-rc -> 207-03: D = 403.1 MB + 35.1MB for fms (Updater v069->v072 with restart successfull. It's weird that the update from 206 is smaller than from 207-01, but that's probably a limitation of tdmupdate?)

That's expected.

When a release package is created (whether it is beta or final release), a differential update from the previous release is created. When beta207-03 was generated, differential update from release206 to beta207-03 was created. You can use it only if you have exactly the "from" version, and it carries you exactly to the "to" version.

When you updated from TDM 2.06, stock updater applied the differential package. When you updated from previous beta, it simply downloaded complete pk4 files which were changed.

Link to comment
Share on other sites

If i want to backup backwards, i'm missing a tdm-version selectionobject.

And like the updater, would be nice to add a grandtotal progressbar

GUI nicety is definitely missing.

The prototype was created by applying minimal changes possible to make the update work.

Of course, no GUI was added: only the existing one remained.

Link to comment
Share on other sites

As a matter of fact, now the updater works with the official TDM beta-testing location.

See updated point (2a) of the original post.

 

Note: remove the tdmupdate_tdmsync.exe that you have already downloaded and download the updated version.

Otherwise, updates from TDM beta-testing location won't work.

  • Like 1
Link to comment
Share on other sites

This sounds like a problem that rsync/librsync would be ideal for, but perhaps that doesn't work if you're stuck downloading files with HTTP rather than actually running an rsync daemon on the remote server?

That's exactly where I got the idea from.

rsync needs special server, but there is zsync which downloads over HTTP. It looked too Linux-specific, so I rolled my own equivalent.

Anyway, the current plan is to switch to a different synchronization algorithm which is tightly integrated with zip file format.

Link to comment
Share on other sites

  • 2 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recent Status Updates

    • Petike the Taffer  »  DeTeEff

      I've updated the articles for your FMs and your author category at the wiki. Your newer nickname (DeTeEff) now comes first, and the one in parentheses is your older nickname (Fieldmedic). Just to avoid confusing people who played your FMs years ago and remember your older nickname. I've added a wiki article for your latest FM, Who Watches the Watcher?, as part of my current updating efforts. Unless I overlooked something, you have five different FMs so far.
      · 0 replies
    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
    • OrbWeaver

      I like the new frob highlight but it would nice if it was less "flickery" while moving over objects (especially barred metal doors).
      · 4 replies
×
×
  • Create New...