NightStalker

This reminded me about the time when I first got involved with TDM (Dec 2016), in the era of TDM 2.04/2.05-beta, building under Linux natively (i.e. no VM). It should be noted that the incredibly annoying issue with SCons rebuilding everything all the time was easily fixed with essentially a 1-line tweak. Reading my notes from back then, I see that when building "from scratch", use of PCH nicely lowered the build time (from 16m39s without PCH to 9m01s with PCH), but only for the 1st build. Subsequent builds' speed improvement with PCH would obviously depend somewhat on what changed in the code, but the improvement was never anywhere near as dramatic, often showing no measurable improvement. Fixing that SCons issue saved me about 9 minutes every single time I'd build TDM. Using PCH saved me a little less than 8 minutes, but only on the very 1st build and never much again.

I don't generally like jumping into the middle of a conversation when someone is already being assisted, but I fear that things might be going astray here. Freyk, IIUC, all 'fraten' wants to do is install a desktop icon. It sounds like the game itself is already installed correctly... it's just lacking a "launch" icon. Is that the case, fraten? That is, can you run the game from a command prompt? If you don't know, please check that first. Also, I don't use Ubuntu (I'm a Slackware user) and I don't use desktop icons, but I suspect that you could find plenty of information online in a generic search for "Ubuntu desktop icons" or similar. Or simply try to find where/how an existing desktop icon is configured. BTW, the advice you got from OrbWeaver is quite sound, based on my past experience. If his instructions failed you, then please be more specific about that so that we can give better guidance. Furthermore, I agree that the instructions are not fool-proof. In all seriousness, if you succeed at this, you might be able to help us with some improved instructions, so please don't give up yet!

Correct. (That's the latest version, which comes with TDM 2.07, for those unaware.)

One reason is to avoid having it overwrite the one you built (i.e. from source code) with changes of your own. I find that it's often a different mindset between Linux users and Windows users, but I (being a full-time Linux user) don't like the fact that any executable I run decides to "self-update" and run itself again. It makes it maddening to debug sometimes. At least the authors had the forethought to include the "--noselfupdate" option.

Many thanks for that file list, stgatilov! That's all I really needed, since (with the single exception of that 'tdm_shared_stuff.zip' file) I'm intimately familiar with the TDM installation fileset, having spent countless hours with it during the run-up to the 2.05 release. But, for the benefit of any others and for future reference, I'm glad you thoroughly documented it. I confirmed that your file list exactly matches what the 'dry run' of the TDM updater showed. Also, there actually is one mirror (http://mirror.helium.in-berlin.de/thedarkmod/release) whose content is view-able. That's what gave me a 2nd list (yours being the 3rd) to confirm that I was on the right track.

Thanks for the link, freyk. I read it, but it's the same technique that Durandall already suggested in that same thread that I linked to in post #5 of this thread. And, as I mentioned in my reply to his/her post, that method yields a less-than-pure installation. It's a work-able alternative, but it's cleaner to use the "interrupt the updater" technique suggested by nbohr1more and implemented by me (required code can be found in that same linked post). I'd like to have stgatilov's list(s) to confirm that his 2.07 "requisite file list" matches what I'm seeing the (hacked) updater try to download.

Many thanks for the info! I'd love to see a list of the files to download to make a complete 2.07 install. But even more than that, I'd like to know how you create such a list (more on that below). Occasionally, for debugging, I find it useful to have run-able, pure, older TDM versions. So I'd also like to see such a file list for 2.04 - 2.06, although I suspect that it may not be possible or might be impractical if said files no longer exist (i.e. packaged with release-specific, older content, for ready-made download). Using a crude hack, I've already managed to create "virgin" installations of older TDM releases, simply by manually stopping the incremental update at the proper step. But creating such virgin installations with an actual list of requisite files would be much nicer. In an attempt to answer my own "I'd like to know how you create such a list" question, I've just now (years overdue, it seems!) done a quick hack to the updater to do a "poor man's equivalent" of the '--dry-run' option that it has annoyingly advertised (via '--help') all along but which has never actually been implemented! Running the updater with that hack let me see just what files the updater wanted to download, without actually downloading all 3.41 GB. It runs in a mere 14 seconds. Having said that, I'd still really like to see your list of files for 2.07, to be sure I'm "on the same page". It would be heaven for me if I never again needed to run that updater app, save for maybe once each release (with my hack) to quickly generate a "requisite file list". I'll download all the files, slowly but reliably using 'wget' (or possibly even off-site somewhere with a fast Internet connection). Then I'll write a simple shell script to efficiently take care of the installation, with no Internet access required at that point! A point of interest: I ran the updater in an empty directory, but since it does not retain partially downloaded content on subsequent runs, it's simply not a good choice for downloading large files over unreliable and/or slow Internet connections. BTW, I had actually looked into trying your new 'tdm_sync' capability. But building it requires a too-new version of CMake, which triggers the need for other updates that I cannot abide at the moment, unfortunately.

Fortunately, the details in that bug report mentioned in that code snippet have shed more light on the problem. After reading it, I realized that there may be some sense in actually (counter-intuitively, I'd argue) running TDM (the game) after the 1st (failed) run of the TDM updater (where it went from 2.00 -> 2.06 instead of 2.07). So I tried that and saw evidence that the game was "Repacking" various PK4 files (presumably the ones containing ROQ/OGG/AVI files). Then I re-ran the same updater process with the same options. It acted differently this time, (needlessly?) downloading all 3 of the missions that come with a default TDM installation as well as (needfully, I suspect) 'tdm_base01.pk4'. But when it was done, the installation did appear to be a true 2.07 install, fortunately. And, in fact, the game now identifies as "TDM 2.07" on the main menu and has 2.07 features, like the new "frob highlighter" (nice addition BTW, STiFU!). So, although my immediate dilemma seems to be solved, it sounds like anyone who tries to do the upgrade the way that I did (regardless of OS) will encounter this odd glitch, which requires a rather unintuitive work-around. Furthermore, unless a new, large "base" TDM zip file that's built for 2.06 or higher is released, it will continue to be a problem. It seems to me like this problem would be affecting all new players, ones who have no 2.06 installation to build upon. Or am I missing something? I don't have the Internet throughput to test it, but I wonder if things would work any differently if I didn't supply a local copy of all the 'tdm_update_2.0{x}_to_2.0{x+1}.zip' files to the updater, thereby (presumably) forcing each update file to be downloaded and applied, 1-by-1. More broadly, just how do other users (Windows or Linux -- I don't think it matters) install TDM "from scratch"?

The log file from the 2nd run of the updater: tdm_update--run-2.log.txt

There's some sort of problem with the TDM updater whereby it won't create a proper 2.07 installation. Being on a slow Internet connection, my TDM installation/update procedure has always involved downloading just the latest update zip file (currently, 'tdm_update_2.06_to_2.07.zip') from the ModDB site and making that available to the updater locally, to avoid unreliable, slow downloads during the actual update. I followed the same procedure that I've always (successfully) done in the past: extract the huge, 2.3-GB 'THEDARKMODVersion2.0-StandaloneRelease.zip' file (downloaded once from ModDB and re-used many times over the years) change the permissions on all the PK4 files to be write-able (so that the updater can alter the content within, as needed) delete the existing, obsolete (2013-era), read-only 'tdm_update.log' file (which shouldn't be in the zip file to begin with) put a copy of all the 'tdm_update_2.0{x}_to_2.0{x+1}.zip' files amassed to date into that extracted directory download the latest version of the TDM updater app (from the TDM website) run the updater, with the options '--keep-update-packages --noselfupdate' On the 1st run, it proceeds in what appears to be normal fashion. There's no obvious indication that anything has gone amok. However, if you run the game, it comes up with the main menu showing "TDM 2.06", not the expected "TDM 2.07". Here's the TDM updater log file for this 1st run: tdm_update--run-1.log.txt Some analysis of the directory content and the updater's log file seems to indicate that the installation only made it to 2.06! As for the 'tdm_update_2.06_to_2.07.zip' file, I've verified the integrity ('zip -T'). I've also verified the filesize and checksum against the updater-downloaded 'tdm_version_info.txt' file: [UpdatePackage from 2.06 to 2.07] crc = 159677cb filesize = 427644498 package = tdm_update_2.06_to_2.07.zipSo I don't think there's anything wrong with that particular file. If I compare what the updater created as a "2.07" installation with an old, virgin 2.06 installation of mine, the only files that are different are ones that you'd expect to be: 'crc_info.txt', 'tdm_mirrors.txt', 'tdm_update.log', and 'tdm_version_info.txt'. If I run the updater a 2nd time (and all subsequent times) with the same options, it essentially does this: Applying differential update package... [=========================] 100.0% Adding: ExtLibs.dll [=========================] 100.0% Adding: ExtLibsx64.dll [=========================] 100.0% Adding: TheDarkModx64.exe [=========================] 100.0% Adding: thedarkmod.x64 [=========================] 100.0% Adding: vcredist_x64.exe [=========================] 100.0% Adding: vcredist_x86.exe [=========================] 100.0% Replacing: AUTHORS.txt [=========================] 100.0% Replacing: TheDarkMod.exe [=========================] 100.0% Replacing: tdm_update.exe [=========================] 100.0% Replacing: tdm_update.linux [=========================] 100.0% Replacing: thedarkmod.x86 [=========================] 100.0% File: Done applying the differential update.Due to website file upload limitations, the TDM updater log file for this 2nd run will be attached to a subsequent post. The updater log files on that 2nd and all subsequent runs are essentially identical, just with different timestamps and potentially use of a different mirror site. That's the repeatable scenario, the "crux of the matter", if you will. FWIW, I also dug more deeply into the problem. What I describe from here on is less thoroughly documented, but I'll mention what I did, in case it helps.... I tried removing all of the update zip files, hoping it would force the updater to update what is effectively a 2.06 installation but via a slow, painful download of that 408-MB 'tdm_update_2.06_to_2.07.zip'. But instead, it inexplicably wanted to download the 235-MB 'tdm_update_2.05_to_2.06.zip' file, so I aborted the update. Taking a different tack, I dug into the updater source code a bit and noticed this curious-looking snippet in '.../tdm_update/libtdm_update/Updater/UpdateController.cpp': //#4529 stgatilov: Since differential update does not uncompress unchanged OGG/ROQ files, //The hashes of pk4 files are now different from the ones on the server. //That's why we exit right away to avoid redoing the same differential update and redownloading the unchanged data. if (info.fromVersion == "2.05" && info.toVersion == "2.06") { TryToProceedTo(PostUpdateCleanup); break; }I played around with various changes to that code (both bumping the 2 version numbers and outright disabling the whole block), but to no avail. Although I'm running Linux (64-bit Slackware 14.2), I don't immediately see any reason why this would be OS-specific. This issue sounds different from the problem grayman had (running Windows), but I did get an "infinite updater loop" scenario when I hacked that code mentioned above, so maybe there's some correlation. I also noticed during some debugging that the installation I had was not being correctly recognized by the updater app as a valid 2.06 installation. When I looked into the updater's console output and log file, it seemed that every file about which the installer reported "SIZE MISMATCH" was one of the PK4 files that had embedded OGG files, although I did not check every case. I'm simply not able to figure out what's going on here, despite much effort, so I have to "punt" before my head explodes. Any input on this matter would be very much appreciated!

I agree with 'gnartsch'. I don't think there's any need to try to fix anything in this regard. The packager app had the bug originally. Unfortunately, it went undetected and a "band-aid" fix (which itself had a flaw) was applied to the updater app. Shortly after I became involved with TDM, I eventually found and fixed both flaws (back in late Dec 2016). IMHO, now that we've been through 2 TDM releases with the fixes in place and no reports of problems, we should be able to remove (at any convenient time) the entire "band-aid" (i.e. the entire block of code that does anything to the timestamps) in the updater app. Since the packager app (the true source of the problem) has been fixed, there should be no problems going forward, I believe.

I explained in a general way in the 1st post of this thread: Refreshing my memory by re-reading my notes on this from Dec 2016, I'll point you to the 'R_ParseImageProgram_r()' routine in the 'renderer/Image_program.cpp' file. Beyond that, I'll leave it to the author of that code to explain.

Apologies for not "closing the loop" on this. Only recently did I finally get back to re-visiting this issue. I wound up simply hacking the 'tdm_update' C++ code to crudely inject a large, obvious console message and a 10-second delay at the end of each version's update. This allowed me to simply monitor the installation and break out at the appropriate time. Very ugly, but it sufficed. Here's the actual code, injected at the end of the 'Updater::PerformDifferentialUpdateStep()' routine in the 'tdm_update/libtdm_update/Updater/Updater.cpp' file: Thanks again for all the suggestions.

Although that file that you referenced which comes with the source code ('COMPILING.txt' is the actual name) is pretty useless (IMHO), it does give the link to the Wiki "Compilation Guide" for TDM. Although that Wiki page certainly could use some attention, it does show one variant of a "scons" command. But, frankly, it's buried in the noise and easy to miss. I'm not a huge fan of wikis, especially for something as important as publishing a project's building instructions. Good that you've found the 'linuxBuild.sh' script. After my first few compiles of TDM, I've never used that script. It might be useful to others, though. I think some of the commands in it were superseded by developments since 2.05 was released. A lot has changed since the last time I was heavily into building TDM and I've also forgotten a lot of details. The official Linux builds are currently done by 'grayman', but (IIRC) he's said that he's offered to let someone else take that task over and nobody ever has. (I suspect he'd still be happy to offload that.) So anything new would undoubtedly need grayman's blessing. And with TDM 2.06 being beta-tested now in the push for its release, it's probably not the best time to ask him to devote time considering any future build changes, even if they might improve his situation in the long run. Taaaki is also involved in the official releases, so you'd probably need to understand what he does. AFAIK, none of that is published as a set of instructions and getting a reply from taaaki can sometimes take a while. I hope I don't sound too discouraging. I like what you're doing so far but I don't call any of the shots. I'm just trying to give a realistic picture of the situation so that you don't waste time and/or get frustrated. And, yes, the SVN trees are a mess, IMHO. This is not how it should be set up!

I think many of your questions are best answered by someone else, so I hope others will comment here. I can only speak for the Linux build. I've never compiled TDM under Windows (or Mac OSX). And I have no experience with Continuous Integration (CI) systems. (In fact, your use of the acronym sent me searching to understand it! ) So, heeding the words of Mark Twain, I will keep my mouth shut and let everyone think I'm a fool rather than open my mouth and remove all doubt. I will say that the various disorganized build instructions are undoubtedly somewhat obsolete, misleading, and incomplete. There are files in the source code, Wiki pages, and $DEITY-knows-what lurking elsewhere. Yikes, what a mess! Given the wide variety of Linux distributions, it's very difficult to thoroughly document the process of TDM building. Improving this has been discussed before. The TDM content is already split into source code and "assets". The source code tree is public (as seen in my earlier 'svn' command). The much larger "assets" tree is currently private. I suspect that the suggestion earlier in this thread that free hosting sites are too limited for TDM repositories might only apply to the "assets" tree. Site hosting is done by 'taaaki', so that's his area. Oh, and just for the record, my "build process" on TDM is simply: scons -Q -j2 BUILD_GAMEPAK=0 NO_GCH=1 BUILD=release There's certainly room for improvement in many areas of TDM, so I'd encourage any/all interest and suggestions.