Jump to content


Photo

Bogus Timestamps In TDM Update Files Cause TDM Crashes Under Linux


  • Please log in to reply
14 replies to this topic

#1 NightStalker

NightStalker

    Linux hero

  • Development Role
  • PipPip
  • 146 posts

Posted 29 December 2016 - 06:33 PM

*
POPULAR

In the beta-testing forum, 'Baal' reported TDM 2.05-beta crashing under Linux.  This is my report on the investigation of that problem, with suggested fixes and work-arounds.
 
TLDR:

Believe it or not, this all comes down to a bunch of bad timestamps in some zip files! :blink: :wacko: :o

Three of the four 'tdm_update_2.0{0-3}_to_2.0{1-4}.zip' files used to update TDM from a "base" 2.00 installation to version 2.04 have illegal timestamps on several of the files within the zip file!

These need to be fixed and the mirrors need to update before this problem will start to go away. Also, this problem may linger, semi-dormant, for any Linux users who have already updated their TDM installation using any of the 3 bad files (or for any Linux 2.05-beta testers since that update file is affected too).

But the back-story is far more involved, so for those interested, please read on....

Details:

This gets a bit hairy, so bear with me, folks! :)
 
Baal had helpfully reported both the output of the debugger after a crash and several missions for which the crash occurred.  Using that information, I was able to easily duplicate the crashing. (The simpler missions that I had been testing with prior to that time did not exhibit the sort of crashes that Baal was seeing.) Running the same missions as Baal, I was greeted with the exact same sort of crashing (under 32-bit Slackware 14.2 Linux in my case). With taaaki having granted me access to the 2.05-beta source code, I was able to track this down more precisely by building a proper 'debug' build.

Even though it was obvious where the crash was occurring now, the cause was not immediately evident. (This was made more difficult by my total inexperience with the TDM code, especially in the area of the crash -- resource file processing).

Of course, a simple hack to prevent the crash was possible, but that would simply be "treating the symptoms" rather than "curing the disease" and I really hate it when people do that (myself included!). And even such a quick hack would have meant that the resource in question failed to load ("WARNING:Couldn't load image: models/darkmod/props/textures/gaslight1_d"), which would then have unknown (to me, as a non-modeler) ramifications later on during game-play. I suppose a missing resource would simply not be rendered, but for all I know, that might have just caused a crash later on in some other part of the code!

So the "root" cause had to be determined, which meant digging deeper.

All of the crashing missions seemed to crash right after loading the "gaslight1" resource, so I spent some time trying to understand what might be wrong with that particular resource and the files it's comprised of. That turned out to be a "red herring", unfortunately. There's nothing special about that resource -- it just happened to be the 1st one affected by this issue that was loaded in the missions that Baal happened to be testing.

Eventually, by digging around more in the code (oh, how I hate C++) and testing, I became suspicious of the timestamps on the resource files. It's not intuitive at all (to me), but the processing of some of these TDM "resources" is based on timestamps.

More specifically, getting that bogus timestamp from a file within a PK4 file results in the low-level 'ReadFile()' routine returning a timestamp of -1, which screws things up further down the line in very subtle ways that ultimately manifests itself as an unprotected attempt to write to a null pointer, thereby crashing the game!

Once I understood that, I soon realized that there were several "child" resource files (i.e. files within the PK4 "parent" file) with "weird" timestamps, including one of the aforementioned "gaslight1" files.

A digression to explain something important for those unaware.... Unix (and Linux) OSes commonly store time values as "seconds since UTC midnight on 01 Jan 1970". Under a 32-bit Linux OS (regardless of whether it runs on a 64-bit CPU or a 32-bit CPU), the timestamps are stored as a 32-bit signed integer. This leaves a maximum date/time of '2^31 - 1' (i.e. 2,147,483,647) seconds since 01 Jan 1970. That means that it's only practical until the date/time shown:

==> TZ='-0000' date --date '@2147483647'
Tue Jan 19 03:14:07  2038

This is the well-known "Y2038" problem.

On a 64-bit Linux OS, the timestamps are, unsurprisingly, stored as a 64-bit signed integer. This leaves a maximum date well past all of our lifetimes, even for those who plan to have a Futurama-style head-in-a-jar someday. ;)

So until TDM becomes a native 64-bit app on Linux, that particular part of this problem could continue, due to inattention to timestamping of files.

---------------------------------
Now back to the main story...

When I looked at several of the resource files contained within the various PK4 files, I saw many with the timestamp "12-31-2077 18:59". That's completely illegal and unrepresentable on a 32-bit Linux system!

But I found that when I extracted the resource files from those PK4 files, those odd timestamps were conveniently(?) converted to the maximum timestamp value, yielding the aforementioned date/time of 19 Jan 2038 03:14:07. And TDM runs fine like that! Furthermore, if you replace the original PK4 file with one that you create yourself (after Linux has "fixed" the timestamps by way of file extraction) by re-zipping the extracted resource files, then TDM will run fine with that new PK4 file too.

OK, those are nice work-arounds, but it's still just "treating the symptoms".

Aside: My cursory testing with Windows (XP Home SP3) shows that it seems to treat timestamps differently. The PK4 files with the bogus timestamps show up in Windows (using WinZip) as "12/31/2077 6:59 PM", so that is presumably a legal timestamp to Windows. (A quick Internet search turned up nothing obvious.)

Unfortunately, I next turned my attention toward the 'tdm_update' app, thinking that it was the most likely culprit, since I'd seen some console output about it "Performing cleanup steps and correcting bad dates in PK4 files".

I spent a long time looking through that mostly uncommented, C++ code (did I mention how much I hate C++ code?), hacking away to add console and log output, and hacking it to only run the "correct bad dates in PK4 files" step. That turned out to be mostly a waste of time. I could not get it to corrupt any of the files it handled such that they had an egregiously wrong timestamp (but more on that later).

I next started downloading, from each of the common TDM mirrors, various PK4 files which, on my system, contained files with bogus timestamps. But I saw no evidence of bogus timestamps in any of the downloaded files, suggesting that something else was afoot here.

So I went back and did a test install of 2.00 followed by a 2.00 -> 2.04 upgrade. When it finished, there were many files within various PK4 files that had the bogus ("31-Dec-2077 18:59:58") timestamps!

Now, finally, I could smell blood!

At this point, I was still suspicious of the 'tdm_update' process and its functionality designed to alter the timestamps of the files in the PK4 file. But some more checking and pondering led me to the zip files that the 'tdm_update' process uses.

Eureka!!! That's it! Those 'tdm_update_2.0{0-3}_to_2.0{1-4}.zip' files are full of bad timestamps!

Now I was finally at the real source of these problems! Those bad timestamps are filtering down into the resource files causing mayhem for Linux users who just happen to load the wrong mission!

The bad timestamps in the 'tdm_update_2.0{0-3}_to_2.0{1-4}.zip' files are especially egregious! They're essentially all "15-31-2107 31:63" (the 15th month, 31st day of year 2107 at 31:63)! What the heck? Various expletives deleted here... ;)

A little research (actually done well prior to this point, while I was still head-scratching) told me that Zip files use the old IBM/MS-DOS format for date+time representation (for those of us old enough to have worked with it but just young enough to still remember having done so!). In essence, those bogus timestamps have every bit set! So, basically:

  • Year (7 bits): 127 (2107-1980)
  • Month (4 bits): 15
  • Day (5 bits): 31
  • Hour (5 bits): 31
  • Minute (6 bits): 63
  • Seconds (5 bits): {seconds}/2

Why those timestamps are getting set that way is still a mystery to me.

Suggested Fix:

Whoever built those 5 'tdm_update_2.0{0-4}_to_2.0{1-5}.zip' files needs to take a look at their system and/or the source (another SVN tree?) for those files used in the zip files!

Find the reason for the bogus, illegal timestamps on the input (resource) files and re-generate all 5 zip files. Actually, the 1st one ('tdm_update_2.00_to_2.01.zip') has no bad timestamps and could be skipped.

Make sure the updated files get picked up by all known TDM mirror sites.

Then we need to strongly suggest to all Linux users of TDM that they revert to the TDM 2.00 distribution and upgrade again, using only newly downloaded 'tdm_update_2.0{0-3}_to_2.0{1-4}.zip' files, to TDM 2.04.

Aftermath:

Now, having identified the real problem, what are the ramifications?

Fortunately, this doesn't seem to affect Windows users, presumably due to Windows' use of a different timestamp method than Unix/Linux.

I didn't know initially how or if this affects users on a 64-bit Linux OS but with 32-bit ("multi-lib") capability installed. All my testing was done on native 32-bit Slackware 14.2. But I went ahead and installed 64-bit Slackware 14.2 with 32-bit "multilib" capability added on, using a spare partition of my "TDM testing" HDD. My suspicion was confirmed -- that environment is equally (i.e. badly) affected by these bogus timestamps, so I suspect that these problems would, left unchecked, affect all Linux users who load the "wrong" missions with 2.05.

I downloaded the same file ('tdm_update_2.03_to_2.04.zip') from 3 mirrors ('mirror.helium.in-berlin.de', 'www.fidcal.com', and 'darkmod.taaaki.za.net') and they all have the same bogus, illegal timestamps, so this is not just a bad mirror somewhere.

A potentially bigger issue is that all those "broken" PK4 files are presumably out there "in the wild". And, even in 2.04, this problem exists! Therefore, I suspect some of these resource files with a bad timestamp are little "time-bombs" just waiting for a Linux user to run a mission that (either now or in the future) uses that object, especially if it applies either the 'makeAlpha' or 'makeIntensity' operation. In other words, this problem could, in theory, be the source of any number of TDM crashes at mission loadup on Linux systems. Eventually, this problem will work itself out, but it could linger for a long time unless Linux users do a wholesale re-install from 2.00 up, and only after all the update zip files have been repaired.

Fortunately, we may have been a bit luckier with the 2.04 resources -- I loaded over a dozen missions in 2.04 and none of them crashed, indicating that none of them accessed any of the objects (at least on initial load) using the bad timestamps, at least in ways that trigger the crash (i.e. by applying either the 'makeAlpha' or 'makeIntensity' operation). But of course, there are far more untested missions that could cause a crash -- I simply don't know how to generate a list of resources that a given mission uses, otherwise I'd simply scan them all looking for "bad" objects.

As hinted at above, this problem also exists in the zip file that updates a 2.04 installation to 2.05-beta, so that file needs to be fixed as well as the 4 previous ones.

Work-Arounds:

Until the relevant zip files are "repaired", Linux users with adequate disk space could simply extract the content of all the PK4 (zip-format) files into the same directory containing the PK4 files -- i.e. their run-time directory -- optionally deleting the original PK4 files. My tests show that there should be no need to eliminate, move, or re-name the PK4 files. The extracted content seems to be used in preference to equivalent content buried in a PK4 file.

Optionally, those extracted files could be re-compressed into PK4 files, but there seems to be little point in doing so.

Alternatively, a hack could be placed into the TDM code to allow TDM to continue running when it encounters any resources with one of these bogus timestamps, but that's not something I'd recommend (and it would be required in at least 2 spots).

Other Issues and Thoughts:

  • The 'tdm_update' app, while not the prime culprit here, is certainly still "suspect" in that it failed to correct the bad timestamps. I need to look into that more at some later date.

    It also seems to needlessly alter files' timestamps by 1 hour, which may be due to its failure to respect 'daylight saving time'. But I've seen differences of more than 1 hour too, always an integer number of hours (2 or 3, but never more, IIRC).

    In short, I'm still somewhat suspicious of that app's alteration of timestamps.

    And I wonder what led to the 'tdm_update' process being given the capability to re-generate PK4 files with embedded files having "bad" ('timestamp_year = 1980' or 'timestamp is in the future') timestamps. Why should that even be needed? It seems like another case of "treat the symptoms" without "curing the disease", but I'll admit that I don't know the history there and could be wrong.
     
  • In looking into this, I've discovered some timestamp code with origins in the Doom3 engine that is not ready for 64-bit OSes!
     
  • It probably wouldn't hurt to put some more protection ("hardening") in various spots in the code, like this spot where a null pointer is not checked for.

    Far better, IMHO, to abort TDM abruptly on someone's PC with an intelligent error message (hoping TDM will have been run from a shell window of sorts, so that such a message will be seen) than to crash and expect them (or someone else) to run TDM under a debugger.
     
  • I've seen all kinds of weird warnings in the TDM console and in the shell's console from which I run TDM. I suspect that some of them may have been manifestations of this same problem, just for resources that didn't happen to cause a crash.

    There's so much "noise" in both the TDM (in-game) console and in the shell window from which it runs that it's hard to see legitimate problems!
     

Closing Thoughts:

My thanks to Baal for bringing this problem to our attention, running under the debugger to narrow it down, and reporting various missions that were affected by this issue.

To those who've read this whole report, you have both my respect and gratitude! :)

I've undoubtedly missed something in this lengthy explanation, so if anyone has questions, please let me know.


  • New Horizon, Bikerdude, duzenko and 3 others like this

#2 gnartsch

gnartsch

    Member

  • Member
  • PipPip
  • 478 posts

Posted 30 December 2016 - 06:52 AM

Regarding these bad timestamps, there is definetly a bug open already for about a year (?).

As far as I recall it was scheduled to be implemented in 2.05, and basicly should make the tdm_update correct all bad timestamps.

Can't point you to the bug-tacker right now though, because the Bugtracker is 'currently down for maintenance'.



#3 grayman

grayman

    Master Builder, Coder

  • Active Developer
  • PipPipPipPipPip
  • 11620 posts

Posted 30 December 2016 - 08:09 AM

Talk to taaaki about bad time stamps. The zip files are built on the package server.

#4 grayman

grayman

    Master Builder, Coder

  • Active Developer
  • PipPipPipPipPip
  • 11620 posts

Posted 30 December 2016 - 08:21 AM

Another thing to ask taaaki about is whether older zip files can be recreated. We might be beyond that point. The recommendation could be to download a full TDM 2.05 and then go back to downloading the update package in 2.06, with the problem fixed.

#5 Bikerdude

Bikerdude

    Mod hero

  • Member
  • PipPipPipPipPip
  • 18356 posts

Posted 30 December 2016 - 09:26 AM

  • This leaves a maximum date well past all of our lifetimes, even for those who plan to have a Futurama-style head-in-a-jar someday. ;)
  • So until TDM becomes a native 64-bit app on Linux,
  • heh, nice.
  • Is this something you could attempt..? and could we have x86 & x64 binaries side by side..?


#6 damiel

damiel

    Member

  • Member
  • PipPip
  • 11 posts

Posted 30 December 2016 - 12:04 PM

Maybe we can find out more if we look at the revision which introduced the date correction for zip files? Revision 6588 seems to be the commit which introduced the date correction.

 

EDIT:

 

I was able to dig out a bugreport about the problem: http://bugs.thedarkm...iew.php?id=4167


Edited by damiel, 30 December 2016 - 12:14 PM.

  • VanishedOne and NightStalker like this

#7 NightStalker

NightStalker

    Linux hero

  • Development Role
  • PipPip
  • 146 posts

Posted 30 December 2016 - 03:26 PM

OK, many thanks for the comments, folks, especially to those pointing me to the BugTracker.  I (foolishly) didn't even think to check it with regard to this issue, but in a preliminary check just now, I can see that there's some important, relevant history there.

I'm a bit dismayed at how much time I had to spend to find what is, essentially, a long-standing, still-outstanding issue that was recently discussed, but it's "water under the bridge" now, I guess.  In the future, I'll be sure to peruse the BugTracker, you can bet!

I've PMed taaaki asking him to read and comment here.

Until he comments and until I've had a chance to fully read up on the history of this issue, I'll probably be mostly silent, since actually fixing this issue is mostly out of my hands now.

Is this something you could attempt..? and could we have x86 & x64 binaries side by side..?

"Could"?  Yes.  "Will"?  Sorry, no.  I simply have too many other projects going and, for the last month, I've been neglecting them in favor of TDM and that cannot continue.  I may have inputs and offer occasional assistance if someone spearheads the drive to 64-bit, but that's the best I can offer ATM.  Sorry.  :(

Whatever happened to 'NagaHuntress'?  It seemed like she was well along the way to converting TDM to a 64-bit app and I was impressed with her contributions.


  • Anderson likes this

#8 Bikerdude

Bikerdude

    Mod hero

  • Member
  • PipPipPipPipPip
  • 18356 posts

Posted 30 December 2016 - 06:03 PM

  • but that's the best I can offer ATM.  Sorry.  :(
  • Whatever happened to 'NagaHuntress'?  It seemed like she was well along the way to converting TDM to a 64-bit app and I was impressed with her contributions.
  • Fair enough.
  • Good question..


#9 taaaki

taaaki

    Forum Hoster

  • Root
  • 818 posts

Posted 02 January 2017 - 05:01 PM

NightStalker:

 

As you probably gleaned from the bug tracker issue, the changes to the updater were implemented as a workaround. This was because the only way to have the packager build "correct" pk4s was to force a situation where the afflicted files would have to be regenerated completely due to an "update" to an existing file and we couldn't ensure this for all the afflicted files. As I recall, this was implemented as an 11th hour kind of thing, so I admit that I probably didn't test it as well as I should have (and I don't think I got around to testing it on Linux at all) - hence the lack of issues on Windows.

 

In theory, it shouldn't matter that the tdm_update_* files have bad dates as the cleanup step in the updater should sort this out. But it is quite likely that I may have missed something in relation to the differential updates, which would not surprise me given how difficult it is to follow the updater code (as you can attest). I'll have to try and fix my Linux environment and run some some tests on there to see if I can isolate why the cleanups didn't work (in particular, I'll need to see why the differential updates still have this issue since the packager was "fixed" prior to the release of 2.04). And I agree that this is treating the symptoms, but unless we can actually track down and properly resolve the issue in the underlying zip library (or switch to a more modern version or alternative library), I'm not sure how to finally resolve this.

 

I assume that you also checked this bug for background: http://bugs.thedarkm...iew.php?id=4110

In that issue, you can see that neither gnartsch, nor myself could figure out why the dates were getting our of whack in the zip/pk4 files in the first place, but my suspicion was that the minizip library used by Doom3 is at fault.

 

I think it might be possible to regenerate older releases, but I'll have to make sure that the packager is actually buillding valid differential update zip files. Anyway, I'll take another look at this during the week and see what I can come up with taking your suggestions into account.


I am the bat. The night is mine.


#10 NightStalker

NightStalker

    Linux hero

  • Development Role
  • PipPip
  • 146 posts

Posted 02 January 2017 - 07:28 PM

Taaaki,

Many thanks for your input on this!  I actually have not had time to go back and read all the relevant stuff in the BugTracker, but I will accelerate that plan now that you've commented.

I would very much appreciate you fixing up your Linux environment because the input from someone much closer to the matter than myself would be helpful.

So, IIUC, there is no issue with the timestamps on the actual "TDM resource" files but they somehow get corrupted only when they're packaged into PK4 files?

I'm starting to think that your reference to "the packager" is the 'tdm_update/tdm_package' tree in SVN.  I didn't have time to dig into that when I was researching these Linux crashes, but it sounds like that may be at the heart of the problem, possibly combined with the 'minizip' code.

I, too, was suspicious of the 'minizip' code in the Doom3/TDM project, but never actually took any time to dig into it -- I was simply overwhelmed by all the other things, mired in deep confusion and unfamiliarity.

I've got some new things to think about and look into now....

Thanks again for your inputs -- much appreciated!

 


  • Anderson likes this

#11 NightStalker

NightStalker

    Linux hero

  • Development Role
  • PipPip
  • 146 posts

Posted 05 January 2017 - 11:29 PM

Good news... I believe I've finally gotten to the bottom of what is corrupting the timestamps on the TDM update zip files ('tdm_update_2.0{0-4}_to_2.0{1-5}.zip').  As is often the case, it's a chain of unfortunate events, in this case triggered by some sloppy coding.

TLDR:

There are calls to 'mktime()' in 3 spots in the TDM code where the Daylight Saving Time (DST) input is not properly set up, essentially causing random cases where the returned time value is off by 1 hour.

By itself, that would be relatively harmless.  But in the "right" circumstances, coupled with another factor (explained below) it's dangerous.

One of these 3 cases is in the TDM "packager" code, which can result in bad timestamps being generated in the zip files distributed for "incremental" TDM updates.  This in turn corrupts several TDM resource files used by the game itself, causing crashes for Linux users when running certain missions which use certain resource files afflicted with bad timestamps, as explained in my 1st post of this thread.

BTW, none of these cases of inadequate 'mktime()' setup can be blamed on the Doom3 source code.  These are all errors introduced by TDM folks.  Moral of the story: Please be careful with your inputs, people!

Backstory:

I was able to duplicate the problem reasonably quickly, by running a customized version of 'tdm_package' with a customized PK4 file full of game resource files.

By simply altering one of the TDM resource files so that it had a timestamp of 01 Jan 1980 00:00:00, I was able to generate a TDM update zip file that had the bogus, illegal (all bits set) timestamp on that custom-dated file.

The bug in the 'mktime()' setup mentioned above caused the time to sometimes jump back by 1 hour, which effectively changed the timestamp on my "test" file to 31 Dec 1979 23:00:00.  The 1-hour alteration is quite unpredictable, since the 'is DST' flag is not initialized before the call and contains, essentially, garbage.  Therefore, sometimes the garbage causes 'mktime()' to think that DST is in effect and other times not.  So, running 'tdm_package' twice in a row on identical input can result in files with different timestamps!

But there is another element to this problem....

The TDM "updater" app and the TDM "packager" app both use a 'minizip' source code "library" of sorts to do various zip/unzip operations.  It contains questionable code in some spots.  It expects the caller-supplied "year" value will be 1980-2107 (an absolute year, with century) or 80-207 (an absolute year, without century) or simply 0-127 (relative to 1980), subtracting either 1980, 80, or nothing to make the range 0-127 (and thereby legal for a zip file's 7-bit year specification).  It simply was not written to tolerate a date with any other year.  So our "1979" case (from the bad 'mktime()' call setup) causes it to erroneously subtract 80 years, resulting in a year of 1899!  When this value is used to compute a DOS/ZIP-style timestamp, expected to be using a year in the range of 0-127, it computes a timestamp that exceeds the 4-byte (32-bit) allotment.  Rather unfortunately, this further triggers some curious code in the 'minizip' library with the terse comment "data overflow - hack for ZIP64".  This in turn causes 0xFF to be written to all 4 bytes of the date/time area of the zip file!

From there, it was easy to see what effect this had on the timestamps.  That is, the timestamps had every bit in the 32-bit value set, which computes to a particular (grossly illegal!) timestamp of "15-31-2107 31:63:62", exactly as described in my 1st post.

The mystery was finally solved!

Solution:

Modify TDM code to set the 'tm_isdst' (DST) flag properly in every case where 'mktime()' is called.  Suggested setting would be -1, which causes 'mktime()' to figure out the appropriate value.

This won't eliminate the odd "data overflow" code in the 'minizip' library, but it should prevent it from being invoked.  That is, if we don't pass the 'minizip' library any dates with an effective year below 1980 (or beyond the 2107 [1980+127] limit of the zip file format), then all should be well.

The real problem now is dealing with all of the bad resource files (and to some extent, bad TDM upgrade zip files) that are already "out in the wild".  This is the same issue as I mentioned in my 1st post, so I won't repeat it here, but it will need to be dealt with somehow.

Other Thoughts:

Without having seen any of the timestamps on the original resource files used to create these 'TDM upgrade' zip files, I cannot be 100% certain that this fix will solve all the bad timestamps.  But I suspect that it's a major source of problems, and quite possibly the only source.

This "mktime() DST setup" bug is in 2 other spots in the TDM source code, so it may be affecting something else entirely.  These 2 cases will need to be investigated further at some point.

I need to give it more thought, but when the fix described above has been in place for a while and has been fully verified, there really should be no need for the 'tdm_update' utility to be checking and altering timestamps in the PK4 "child" files any longer.  That was/is essentially just "treating the symptoms", I believe, and I think we now know the "cure for the disease" (I hope!).  So, at some point, we should give some thought to actually removing the code in the 'tdm_update' utility that tweaks timestamps.

The question arose earlier: Why didn't the 'tdm_update' app fix the broken timestamps in the PK4 child files given that it's specifically designed to do so?  Without looking into it, I'm not sure but I'll hazard a guess that those broken dates may have triggered the very same "data overflow - hack for ZIP64" code in the 'minizip' library.  Or something similar.

If I don't hear from taaaki (who I've PMed) reasonably soon (2 days?), my plan is to implement the simple fix and commit it to SVN, unless I hear otherwise.

As always, if I've left anything out, please let me know.


  • New Horizon, nbohr1more, Anderson and 1 other like this

#12 grayman

grayman

    Master Builder, Coder

  • Active Developer
  • PipPipPipPipPip
  • 11620 posts

Posted 06 January 2017 - 04:41 AM

Excellent find!!
  • New Horizon likes this

#13 taaaki

taaaki

    Forum Hoster

  • Root
  • 818 posts

Posted 07 January 2017 - 06:25 PM

Nice find indeed. I have no issue with you committing the mktime() related changes. Once you do, I'll just need to know so that I can recompile the packager binary on the server (have to do it there since the server runs FreeBSD) and recommit the bin to SVN. I can then investigate rebuilding the differential updates and syncing to the mirrors.

 

We might still need some process for fixing a user's existing install (for cases where there are bad PK4s from previous updates). So shouldn't we leave the timestamp altering stuff in the updater for a release or two (after fixing it for Linux)? Or do you see another way, like fixing the file reading code in TDM itself so that it doesn't get tripped up by the bad dates?


I am the bat. The night is mine.


#14 NightStalker

NightStalker

    Linux hero

  • Development Role
  • PipPip
  • 146 posts

Posted 07 January 2017 - 11:10 PM

I have no issue with you committing the mktime() related changes.

Ok, I will try to get the update done sometime in the next 24 hours.

Once you do, I'll just need to know so that I can recompile the packager binary on the server [...]

Wilco.  Will PM you when I'm ready.

We might still need some process for fixing a user's existing install (for cases where there are bad PK4s from previous updates). So shouldn't we leave the timestamp altering stuff in the updater for a release or two (after fixing it for Linux)? Or do you see another way, like fixing the file reading code in TDM itself so that it doesn't get tripped up by the bad dates?

I don't have any objection to leaving the timestamp check/tweak in place for a while, but since it didn't work (in my case, anyway), I think someone needs to investigate why not. I suppose I can do that, but, again, I'm both lousy at C++ and really hate it!

 

As for "another way", it's certainly possible, but despite initially digging deeply into the non-primary cause (i.e. what turned out to be just "the symptoms"), I still have no strong understanding of why the resources are time-checked, let alone the confidence to start modifying that C++ code! :o

 

Give me some time to think about this last quoted bit of yours some more.  Will comment more when I can.

 

Thanks for your input, taaaki!



#15 NightStalker

NightStalker

    Linux hero

  • Development Role
  • PipPip
  • 146 posts

Posted 13 January 2017 - 04:31 PM

*
POPULAR

Just a quick update for any interested parties....

I believe that all the pieces are in place now for this problem to go away (hopefully for good!).  I have applied fixes to both the 'tdm_package' app and (just now) to the 'tdm_update' app.  Taaaki has re-generated and carefully tested new 'tdm_update_*_to_*.zip' files which should eventually be propagated to all the mirrors.

So, whether we're supporting new users, installing from scratch, or old-timers updating from an existing install, there should be no more bad timestamps once people run the 'tdm_update' process after 2.05 is released.


  • Springheel, Bikerdude, nbohr1more and 5 others like this




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users