Jump to content
The Dark Mod Forums

A couple of stupid questions regarding TDM_update

gnartsch

By gnartsch
in The Dark Mod

 Share

Recommended Posts

gnartsch

gnartsch

Posted
Posted

Hi all !

 

While trying to figure out some problems with a specific mission,

I came across some some puzzling things regarding the update procedure via tdm_update.exe

 

I realized that some specific packages are apparently different at first glance (using a HEX-file comparison tool) and also produce different CRC values.

However they have the same size and once unpacked, the content seems identical, too.

Anyway, since tdm_update would not consider them worth downloading I had to start the updater from a clean directory and noticed ~10 packages being different then the once I had so far.

 

By name, these are

E:\Games\Doom3-TDM\darkmod\tdm_ai_humanoid_pagans01.pk4

E:\Games\Doom3-TDM\darkmod\tdm_gui_credits01.pk4

E:\Games\Doom3-TDM\darkmod\tdm_models01.pk4

E:\Games\Doom3-TDM\darkmod\tdm_models03.pk4

E:\Games\Doom3-TDM\darkmod\tdm_player01.pk4

E:\Games\Doom3-TDM\darkmod\tdm_sound_ambient02.pk4

E:\Games\Doom3-TDM\darkmod\tdm_sound_vocals05.pk4

E:\Games\Doom3-TDM\darkmod\tdm_textures_base01.pk4

E:\Games\Doom3-TDM\darkmod\tdm_textures_decals01.pk4

E:\Games\Doom3-TDM\darkmod\tdm_textures_fabric01.pk4

E:\Games\Doom3-TDM\darkmod\tdm_textures_paint_paper01.pk4

E:\Games\Doom3-TDM\darkmod\tdm_textures_roof01.pk4

E:\Games\Doom3-TDM\darkmod\tdm_textures_stone_brick01.pk4

E:\Games\Doom3-TDM\darkmod\tdm_textures_stone_sculpted01.pk4

E:\Games\Doom3-TDM\darkmod\tdm_textures_window01.pk4

E:\Games\Doom3-TDM\darkmod\tdm_textures_wood02.pk4

 

However, the list depends on what server the individual files were taken from.

E.g. I can say for sure that tdm_models02.pk4 differs in CRC (altough size and actual content seem identical) between

http://roggen.jmnet.us/ and http://www.fidcal.com/

 

I don't think it makes sense that every mirror packages these files themselves !

There should be a master-server where the mirrors pick up the entire set of files and should not touch them or build them on their own !!!

 

 

So here some questions:

- why did they get repackaged ?

- why does the updater not do a CRC check and how could I force this to happen ?

ok, I guess for performance reasons it checks the size first - but this may result in some files not being updated

- is it granted that all mirrors have the same file ? maybe some mirror MIGHT have still some old files.

 

OK, this did not help the problems I have with the FM, but still it puzzles me.

 

Greetings

Link to comment
Share on other sites
Serpentine

Serpentine

Posted
Posted

Ikatara uncovered a couple of these CRC related things when rewriting the current perl updater to python,here's the thread

 

It doesnt seem to be the exact same issue you are having - I'll fetch and try to confirm your suspicion however.

 

Edit:

Yeah the archive crc's dont match, tho all of the elements do seem to match - still not an ideal situation, maybe Tels knows a bit more tho. Paging Dr Tels to this fine thread.

Link to comment
Share on other sites
Tels

Tels

Posted
Posted

It is possible to re-package the files from a ZIP file into another ZIP file, and since the timestamp on the ZIP file changes, you end up with different ZIP-files (aka containers) but identical files (aka content).

 

However, I am not sure why this would happen here, and it shouldn't happen anyway.

 

Technically, the tdm_updater *should have* used an MD5 checksum on the ZIP file, not just the contents. However, instead it just checks the 32bit checksums of all the contents (and apparently there is also a bug in it when there is only 1 file IIRC).

 

That itself is dodgy (32bit is way to small to guard against anything except an accidental bitswap) and it also doesn't cover the "repackage the same files into a different zip file" case.

 

However, for some reasons it was decided to do it this way (I forgot why it was decided, I think I remember I said that will cause problems like the one you described - but maybe there was a technical reason to not check the container file, anyway).

 

So, if the container are different, but the contents are "good", the updater won't redownload the ZIP files.

 

Edit: Here is a list with the MD5 sums of files on bloodgate.com:

 

http://bloodgate.com/mirrors/tdm/pub/pk4/md5sums.txt

 

http://en.wikipedia.org/wiki/Md5sum

 

 

Files that do NOT match these sums should be taken as broken, please notify us so we can replace them :)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites
Tels

Tels

Posted
Posted

Hm, just had an idea: The "different zip file but same content" can happen if we repackage the next TDM version - the ZIP file gets regenerated but wit the same set of files. If then only one mirror receives the new files, but the other keeps the old file, you end up with two copies.

 

Still, would be better to syncronice the mirrors.

 

For the end-user, it doesn't make a difference, tho. Whatever file you get, as long as the contents are right, it will work.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites
Serpentine

Serpentine

Posted
Posted

Updater etc etc

 

Yeah I thought it might be the unchanged version doing it.

 

In any case, we could quite easily add support for MD5 without breaking the existing framework (CRC32 could still be used for internal checks), since both your perl and the python just parse the lists with Tiny adding the md5 value at the end wouldn't be to any determent as far as I can see.

Link to comment
Share on other sites
gnartsch

gnartsch

Posted
  • Author
Posted (edited)

Hi Tels !

 

As an example, here is the checksums for tdm_textures_wood02.pk4

According to the CRC-file downloaded by the updater from the server its checksum should be:

[File tdm_textures_wood02.pk4]

crc = e6d6c693

size = 857416

I downloaded a recent copy today via TDM-update:

Getting tdm_textures_wood02.pk4 from http://www.keepofmetalandgold.com/

When using some CRC tool on it, I get

CRC32: C91F4C5F

MD5: 36A6B888E49FBF9E8BA101610A4214BD

SHA-1: B3FA81E474727D277E8CFC90A6BF0F17CA1C4220

which does at least match the MD5 value you linked me to:

MD5 (tdm_textures_wood02.pk4) = 36a6b888e49fbf9e8ba101610a4214bd

 

However I doubt that the updater actually cares about CRC.

I can't imagine how it could possibly do the entire CRC-checking within 3 seconds on 1.6GB of data on my elderly equipment!

And since it receives CRC32 info for comparison only, it should have attempted to download it, if it really cared !

But it just seems to ignore anything as long as the size matches.

Right ?

 

 

 

 

EDIT:

 

I just tried editing tdm_textures_nature02.pk4 using a Text-editor (making sure to entirely mess up the file, but keep the size at 555 bytes).

Result with tdm_update: file is ok. no need to update :laugh:

Trying to extract the file: well, it is broken off course !

No sooner then I modified the size, the updater would consider it worth re-downloading.

 

As for the CRC32 checksums : I can't find any match between the files on my disk and the ones reported by the server.

Maybe the algorithm I have is different - or the CRC values reported by the server are wrong.

I don't know.

At least the version I have does care about timestamps inside the pk4 - which off course is not really subject to this topic, but which brought up this topic at first hand.

 

Anyway, all servers should have identical files and the updater should not only pretend checking CRC sums.

Edited by gnartsch
Link to comment
Share on other sites
Tels

Tels

Posted
Posted

Yeah I thought it might be the unchanged version doing it.

 

In any case, we could quite easily add support for MD5 without breaking the existing framework (CRC32 could still be used for internal checks), since both your perl and the python just parse the lists with Tiny adding the md5 value at the end wouldn't be to any determent as far as I can see.

 

AFAIR the problem with MD5 is that the packaging process regenerates new ZIP files with the same content and we need to filter them out before rolling things out. Certainly doable, but atm nobody has time to work on this stuff at all... (we desperately need more manpower!)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites
Tels

Tels

Posted
Posted

Hi Tels !

 

As an example, here is the checksums for tdm_textures_wood02.pk4

According to the CRC-file downloaded by the updater from the server its checksum should be:

[File tdm_textures_wood02.pk4]

crc = e6d6c693

size = 857416

I downloaded a recent copy today via TDM-update:

Getting tdm_textures_wood02.pk4 from http://www.keepofmetalandgold.com/

When using some CRC tool on it, I get

CRC32: C91F4C5F

MD5: 36A6B888E49FBF9E8BA101610A4214BD

SHA-1: B3FA81E474727D277E8CFC90A6BF0F17CA1C4220

which does at least match the MD5 value you linked me to:

MD5 (tdm_textures_wood02.pk4) = 36a6b888e49fbf9e8ba101610a4214bd

 

However I doubt that the updater actually cares about CRC.

 

The updater checks the *internal* CRC of the all the files in the ZIP (actually, their sum), so it cannot match the CRC of the ZIP container (however, you produced that :)

 

Also, for your test, yes, it is true, if you edit the ZIP file container, the updater won''t notice that. If you are lucky, it will still unpack the (still correct) files, if you are unlucky, it crashes. That's expected :)

 

And yes, we should check the SHA1 (not even MD5, as it is trivial to create collisions - see http://www.mscs.dal.ca/~selinger/md5collision/) of the entire file. I am on your side with this :)

 

However, for some reasons greebo did do it otherwise. And also, as I wrote, we don't have anyone who can work on this at the moment. Volunteers welcome :)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites
aidakeeley

aidakeeley

Posted
Posted

I have downloaded the TDM sources... where is the tdm_update source? I'm afraid I've been out of the loop too long and more than that I'm just unreliable these days, but I might give it a looksee because I think this is causing a lot of problems and it would be a less intense thing than jumping in to the DR or TDM code...

 

No promises, not even really a suggestion of a glimmer of hope that I would be able to contribute... but... who knows...

"A Rhapsody Of Feigned And Ill-Invented Nonsense" - Thomas Aikenhead, On Theology, ca. 1696

Link to comment
Share on other sites
Tels

Tels

Posted
Posted

I have downloaded the TDM sources... where is the tdm_update source? I'm afraid I've been out of the loop too long and more than that I'm just unreliable these days, but I might give it a looksee because I think this is causing a lot of problems and it would be a less intense thing than jumping in to the DR or TDM code...

 

No promises, not even really a suggestion of a glimmer of hope that I would be able to contribute... but... who knows...

 

It is only in our SVN, but here is a copy:

 

http://bloodgate.com/mirrors/tdm/pub/tdm_update.txt

 

You need to rename it to .pl

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites
aidakeeley

aidakeeley

Posted
Posted

Well, that just proves how lost I am already :laugh:

 

I thought it'd be in some C; I also thought perl was an interpreted script language; yet tdm_update is an exe complied linked and executable... I pretty far removed obviously. :laugh::laugh:

 

But I've still grabbed what you attached there and will at least get a chance to figure some of my confusion out. So thanks.

"A Rhapsody Of Feigned And Ill-Invented Nonsense" - Thomas Aikenhead, On Theology, ca. 1696

Link to comment
Share on other sites
Tels

Tels

Posted
Posted

Well, that just proves how lost I am already :laugh:

 

I thought it'd be in some C; I also thought perl was an interpreted script language; yet tdm_update is an exe complied linked and executable... I pretty far removed obviously. :laugh::laugh:

 

But I've still grabbed what you attached there and will at least get a chance to figure some of my confusion out. So thanks.

 

We are using a thing called "pp", some sort of perl compiler since Windows doesn't have Perl :/

 

Here is how we build the exe for linux:

 

# New version without progress bar, but with icon:
pp --icon=tdm_update.ico -I devel/lib -o tdm_update.linux devel/tdm_update.pl

 

There was also some talk about rewriting the updater in Python, but I have no idea what came of that (or how that would help us here).

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites
taaaki

taaaki

Posted
Posted (edited)

I've been working on a Python replacement for tdm_update. At the moment it is more or less working the same as the Perl version except for differences with libraries such as the Python zipfile module. All I really need to do is sanitise zip member paths (the Python zipfile module doesn't really make this easy) and test the Windows batch file for restarting the updater. Then I need to test packaging with py2exe for Windows and Python Freeze for Loonix. If it all works out properly (ie. it behaves the same as the current updater in all circumstances), I can look at revamping the file checksum system to use SHA-1 instead. And maybe even look at throwing a GUI on top and supporting multiple download threads (if the mirror hosts have no objections).

 

Edit: the Python rewrite was suggested to me by Serpentine because some people were having issues with Perl libraries on certain Linux distros. I'm not sure if those issues were sorted out using other means or if it really was Perl+distro causing the problem. Also, I needed a side project to keep me coding.

 

 

Edited by taaaki

I am the bat. The night is mine.

Link to comment
Share on other sites
gnartsch

gnartsch

Posted
  • Author
Posted (edited)

Okay, now that makes perfect sense (the pp compiler thing ...not the Python thing ;) )

 

As I say, you may never hear back from me on this but I do appreciate the info.

 

I might be willing to help as well, but I am perfectly unfamiliar with perl & pp.

I downloaded some ActivePerl stuff (or some other perl packages ?) and some stuff from http://search.cpan.o...5_01/script/pp.

Now, I get some odd errors that don't tell me anything.

But I understand I would need to download some sort of perl-packages, define some variables, ...

Plus I need a C-compiler (which one) for PP.

Question would be : WHAT would I need ? WHERE to get it? HOW to install/configure it ?

As an alternative:

Is there a quick way running the perl script manually (without building a EXE) ?

 

I just don't have the time to figure out that stuff on my own from scratch.

 

Anyway, Serpentine pointed out THIS thread earlier, which is concerned about the Python conversion.

But it also points out some issues with the current perl implementation.

It might be a good idea to discuss WHY the conversion to Python will be done/is planned and whether the problems discussed here would be an issue with the Python version as well or not.

 

 

 

EDIT (sorry for the long rant following): seems like taaaki provided some background on the reason for going with Python, before I was able to submit my posting.

 

Since I have no idea/time at all to figure out how to get Perl/Python running:

Maybe I can help by providing some manipulated files to figure out WHY tdm_update does not recognize obviously broken pk4 files ?

 

When randomly editing some, I noticed some could not be opened by 7zip at all, whereas others could be openend, but 7zip would refuse to extract the content since the files inside are considered broken.

At the same time tdm_update would always say "CRC match" !

I just don't get how this may be possible, since CRC would have NO meaning at all here any more.

 

Tels, I understand that ZIPs might be handled different (having CRC for both the entire ZIP+CRC for all files contained within), but I am first of I am referring to the easy case (pk4-files).

The script and the CRC-info obtained from the server denote ONLY CRC info for the entire file.

 

 

Just to cool down the discussion:

I don't experience any issues the CRC check not working and could live with it as is, but I dare to state that it just doesn't work.

I suspect there is simply some problem in the way the checking is implemented.

This MIGHT mean that even switching to some other CRC algorithm might not solve this problem.

 

 

However, might someone look into the problems with these 2 tiny files (and maybe even test with some other CRC algorithm) ?

I have posted two modified versions of tdm_textures_nature02.pk4 here:

 

1. only the file contained in the pk4 was changed.

The PK4 file can be opened, but the file contained can not be extracted due to CRC errors

tdm_updater won't consider it worth updating

http://www.gnartsch....es_nature02.pk4

 

2. PK4 header + file was changed

The PK4-file can not even be opened using 7Zip

tdm_updater won't consider it worth updating

http://www.gnartsch....es_nature02.pk4

 

Greetings,

 

-gnartsch-

 

I would wish I could provide some more practical assistance here.

Edited by gnartsch
Link to comment
Share on other sites
Tels

Tels

Posted
Posted

I might be willing to help as well, but I am perfectly unfamiliar with perl & pp.

I downloaded some ActivePerl stuff (or some other perl packages ?) and some stuff from http://search.cpan.o...5_01/script/pp.

Now, I get some odd errors that don't tell me anything.

But I understand I would need to download some sort of perl-packages, define some variables, ...

Plus I need a C-compiler (which one) for PP.

Question would be : WHAT would I need ? WHERE to get it? HOW to install/configure it ?

As an alternative:

Is there a quick way running the perl script manually (without building a EXE) ?

 

Sure it is :)

 

If you are under Linux, just run:

 

perl tdm_update.pl

 

Under Windows, install StrawberryPerl http://strawberryperl.com/ (5.12 should work) and then open a DOS prompt and run the same.

 

Notable flags are:

 

perl tdm_update.pl --dry-run (so it doesn't download stuff)

 

Anyway, Serpentine pointed out [url="http://forums.thedarkmod.com/topic/11473-problem-with-crcs-and-the-updater/"]THIS[/url] thread earlier, which is concerned about the Python conversion.
But it also points out some issues with the current perl implementation.
It might be a good idea to discuss WHY the conversion to Python will be done/is planned and whether the problems discussed here would be an issue with the Python version as well or not.

 

It was basically because:

 

* we have issues with the Perl code (mainly due to the fact that the binary has problems on 32/64 bit Linux - and as an exe on windows)

* there is no "GUI" (and users love a GUI more than the functionality)

* He can do Python and believes these things will get fixed that way

 

Of course the downside is that we now have Python and Perl code and I don't know Python at all...

 

I would wish I could provide some more practical assistance here.

 

 

Just please file a bug tracker entry here http://bugs.angua.at/my_view_page.php - if you can't, post here and I'll do it. I'll take care of adding whole-file summing to the updater, but it ight take a while - I am in the midst of a huge patch atm and want to finish it first.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites
greebo

greebo

Posted
Posted

I'll take care of adding whole-file summing to the updater, but it ight take a while - I am in the midst of a huge patch atm and want to finish it first.

 

We didn't run a checksum over the entire PK4 file because it's useless. We're packaging from our local SVN repositories, where the timestamps of the files are set to the time the SVN checkout was performed. When packaging the files from the SVN working copy the PK4 "whole-file" checksum will differ from system to system, even if the file contents are bit-wise identical. The only information of the PK4 files themselves is their size and the filename which should not differ.

 

Here are the quotes from one year ago:

 

I fixed the MD5 sum problem, but here's another topic to consider.

 

I realised that the MD5 hashing is maybe not the best way to qualify PK4s as "to be updated" or "not changed". Reason: on my end, when I check out the darkmod repository from SVN, all the files receive the timestamp of "now", i.e. when I hit SVN Checkout, the file change date is set to now.

 

When I zip up the files into a PK4 file, the timestamp is saved into the ZIP archive. Hence, if I build a PK4 using the package script on two different systems, I will always get two different MD5 sums, as the check out dates and therefore the file timestamps will differ, and so will the resulting ZIPs.

 

This means, that as soon as I build a new package on my home systems, all the MD5 sums will be different, i.e. all files appear as "dirty" to tdm_update.

 

Crud. I just tried copying a folder and zipping it up. As soon as the time stamp of the folder is changed (the contained file timestamps are identical) the MD5 sum of the ZIP archive is different.

 

So I guess we need a different validity check. I see that although the MD5 hash is differen, the CRC (using 7-zip's "Info" feature) is identical. I'll check if we can extract the CRC via script and use that as check.

 

edit: counter-checking, it seems that the CRC is the same even if a folder name in the ZIP is changed, so it seems the CRC is calculated on the file contents only. Probably not a big deal.

 

Ok, I reorganised the tdm_update algorithm such that it inspects the actual content CRC of the PK4s. The new sum file on the server is named "crc_info.txt", which contains the CRCs for all the PK4s on the server.

 

To generate such a crc_info.txt file, there is a new perl script "devel\calculate_crcs.pl" in SVN.

 

I replaced a few routines in tdm_update.pl to take the CRCs into account, this worked fine so far with my system here. I tried altering a PK4 locally and the updater correctly downloaded the PK4 from the server again.

 

Thread: http://modetwo.net/d...752-tdm-update/

Link to comment
Share on other sites
Tels

Tels

Posted
Posted

We didn't run a checksum over the entire PK4 file because it's useless. We're packaging from our local SVN repositories, where the timestamps of the files are set to the time the SVN checkout was performed. When packaging the files from the SVN working copy the PK4 "whole-file" checksum will differ from system to system, even if the file contents are bit-wise identical. The only information of the PK4 files themselves is their size and the filename which should not differ.

 

Yeah, and the reason why we still didn't go with MD5/SHA1 the whole file (it would work) is that each SVN check out and then packaging the package will create a "new" archive for PK4s that haven't actual changed content. And this means someone (e.g. greebo :) would need to re-upload them, which is a major bandwidth and time headache.

 

However if we make the packager script smart enough to not create these files, you would end up with only the new files to upload, so everything would be fine. How does that sound?

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites
greebo

greebo

Posted
Posted

However if we make the packager script smart enough to not create these files, you would end up with only the new files to upload, so everything would be fine. How does that sound?

I don't think this makes things easier for anybody, as the packager is slow enough already. Comparing each file in the existing PK4 set with each SVN file to be packaged will take a lot of time, and the packager already takes ten minutes for me (and each release takes a lot of packaging runs, believe me). Plus, I need to manually copy over the files into the existing TDM PK4 set each time.

 

What's the problem at hand we're trying to solve anyway?

Link to comment
Share on other sites
Tels

Tels

Posted
Posted

I don't think this makes things easier for anybody, as the packager is slow enough already. Comparing each file in the existing PK4 set with each SVN file to be packaged will take a lot of time,

 

I don't expect that, not really, because it needs to compare/compress it, anyway. It might even be faster NOT to create the new PK4 if no file in it has changed, anyway.

 

and the packager already takes ten minutes for me (and each release takes a lot of packaging runs, believe me). Plus, I need to manually copy over the files into the existing TDM PK4 set each time.

 

I am pretty sure we can automate the copy, to. (Btw, I didn't think you'd handle v1.03 anyway :)

 

What's the problem at hand we're trying to solve anyway?

 

That the updater fails to spot corrupted/modified PK4 files and thinks they are okay. Which is a total fail in my eyes :)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites
taaaki

taaaki

Posted
Posted

Another way to do the checks would be to actually have the updater client unpack each file in a Zip/PK4 and check if: a.) it is possible to even open the archive, b.) it is possible to extract the file and c.) the extracted CRC matches the CRC value stored in the archive (or do a running sum of extracted CRCs). The Python libraries allow you to extract in chunks to memory so temp files/dir hassles shouldn't be a problem. To some extent the updater already does this after it downloads a Zip file. But doing the full check for all files at the start of the update would probably take a very long time, so maybe make this an option like --deep-check.

 

The main reason for going Python in my mind would be for the GUI stuff. Serps and I seem to have found a reasonably portable gui library for Python so I'm going to start experimenting with that soon.

 

 

I am the bat. The night is mine.

Link to comment
Share on other sites
Tels

Tels

Posted
Posted

Another way to do the checks would be to actually have the updater client unpack each file in a Zip/PK4 and check if: a.) it is possible to even open the archive, b.) it is possible to extract the file and c.) the extracted CRC matches the CRC value stored in the archive (or do a running sum of extracted CRCs).

 

That wouldn't work as a corrupted ZIP file can easily cras either the zip library or the updater. Plus it would be dead-slow. It would be much better to check a hash of the entire file before trying to handle it.

 

The main reason for going Python in my mind would be for the GUI stuff. Serps and I seem to have found a reasonably portable gui library for Python so I'm going to start experimenting with that soon.

 

Don't let me hinder you :)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites
greebo

greebo

Posted
Posted

I don't expect that, not really, because it needs to compare/compress it, anyway. It might even be faster NOT to create the new PK4 if no file in it has changed, anyway.

Before not-creating the PK4 the comparison must be done somehow, so this might not be a huge time saver. But I won't argue against that if you're going to do the work and the comparison.

 

That the updater fails to spot corrupted/modified PK4 files and thinks they are okay. Which is a total fail in my eyes :)

Yes, this is definitely a fail. I agree that an overall TDM 1.03 checksum (post-processing the full PK4 set) would be enough to resolve that then.

 

Btw, I didn't think you'd handle v1.03 anyway :)

And you are trying to say what? That I shouldn't participate in the discussion because I won't do the actual release footwork? That'd probably be fair enough, but I remember that in the past not doing the work didn't hinder anyone from furiously discussing stuff in the mod.

 

Thanks for reminding me though. Have fun with the release, I'm anxious to see how things will be turning out.

Link to comment
Share on other sites
Tels

Tels

Posted
Posted

And you are trying to say what? That I shouldn't participate in the discussion because I won't do the actual release footwork?

 

No, I was just trying to say that if you don't do the release, it wouldn't matter to you how long things take :)

 

That'd probably be fair enough, but I remember that in the past not doing the work didn't hinder anyone from furiously discussing stuff in the mod.

 

Thanks for reminding me though. Have fun with the release, I'm anxious to see how things will be turning out.

 

Given that we have exactly zero persons who can and want to work on the release, I think nothing will happen at all.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing

  • Recent Status Updates

    • nbohr1more

      nbohr1more

      The FAQ wiki is almost a proper FAQ now. Probably need to spin-off a bunch of the "remedies" for playing older TDM versions into their own article.
      · 1 reply
    • nbohr1more

      nbohr1more

      Was checking out old translation packs and decided to fire up TDM 1.07. Rightful Property with sub-20 FPS areas yay! ( same areas run at 180FPS with cranked eye candy on 2.12 )
      · 3 replies
    • taffernicus

      taffernicus

      i am so euphoric to see new FMs keep coming out and I am keen to try it out in my leisure time, then suddenly my PC is spouting a couple of S.M.A.R.T errors...
      tbf i cannot afford myself to miss my network emulator image file&progress, important ebooks, hyper-v checkpoint & hyper-v export and the precious thief & TDM gamesaves. Don't fall yourself into & lay your hands on crappy SSD
       
      · 7 replies
    • OrbWeaver

      OrbWeaver

      Does anyone actually use the Normalise button in the Surface inspector? Even after looking at the code I'm not quite sure what it's for.
      · 7 replies
    • Ansome

      Ansome

      Turns out my 15th anniversary mission idea has already been done once or twice before! I've been beaten to the punch once again, but I suppose that's to be expected when there's over 170 FMs out there, eh? I'm not complaining though, I love learning new tricks and taking inspiration from past FMs. Best of luck on your own fan missions!
      · 4 replies
×
×
  • Create New...