NightStalker

This reminded me about the time when I first got involved with TDM (Dec 2016), in the era of TDM 2.04/2.05-beta, building under Linux natively (i.e. no VM). It should be noted that the incredibly annoying issue with SCons rebuilding everything all the time was easily fixed with essentially a 1-line tweak. Reading my notes from back then, I see that when building "from scratch", use of PCH nicely lowered the build time (from 16m39s without PCH to 9m01s with PCH), but only for the 1st build. Subsequent builds' speed improvement with PCH would obviously depend somewhat on what changed in the code, but the improvement was never anywhere near as dramatic, often showing no measurable improvement. Fixing that SCons issue saved me about 9 minutes every single time I'd build TDM. Using PCH saved me a little less than 8 minutes, but only on the very 1st build and never much again.

I don't generally like jumping into the middle of a conversation when someone is already being assisted, but I fear that things might be going astray here. Freyk, IIUC, all 'fraten' wants to do is install a desktop icon. It sounds like the game itself is already installed correctly... it's just lacking a "launch" icon. Is that the case, fraten? That is, can you run the game from a command prompt? If you don't know, please check that first. Also, I don't use Ubuntu (I'm a Slackware user) and I don't use desktop icons, but I suspect that you could find plenty of information online in a generic search for "Ubuntu desktop icons" or similar. Or simply try to find where/how an existing desktop icon is configured. BTW, the advice you got from OrbWeaver is quite sound, based on my past experience. If his instructions failed you, then please be more specific about that so that we can give better guidance. Furthermore, I agree that the instructions are not fool-proof. In all seriousness, if you succeed at this, you might be able to help us with some improved instructions, so please don't give up yet!

Correct. (That's the latest version, which comes with TDM 2.07, for those unaware.)

One reason is to avoid having it overwrite the one you built (i.e. from source code) with changes of your own. I find that it's often a different mindset between Linux users and Windows users, but I (being a full-time Linux user) don't like the fact that any executable I run decides to "self-update" and run itself again. It makes it maddening to debug sometimes. At least the authors had the forethought to include the "--noselfupdate" option.

Many thanks for that file list, stgatilov! That's all I really needed, since (with the single exception of that 'tdm_shared_stuff.zip' file) I'm intimately familiar with the TDM installation fileset, having spent countless hours with it during the run-up to the 2.05 release. But, for the benefit of any others and for future reference, I'm glad you thoroughly documented it. I confirmed that your file list exactly matches what the 'dry run' of the TDM updater showed. Also, there actually is one mirror (http://mirror.helium.in-berlin.de/thedarkmod/release) whose content is view-able. That's what gave me a 2nd list (yours being the 3rd) to confirm that I was on the right track.

Thanks for the link, freyk. I read it, but it's the same technique that Durandall already suggested in that same thread that I linked to in post #5 of this thread. And, as I mentioned in my reply to his/her post, that method yields a less-than-pure installation. It's a work-able alternative, but it's cleaner to use the "interrupt the updater" technique suggested by nbohr1more and implemented by me (required code can be found in that same linked post). I'd like to have stgatilov's list(s) to confirm that his 2.07 "requisite file list" matches what I'm seeing the (hacked) updater try to download.

Many thanks for the info! I'd love to see a list of the files to download to make a complete 2.07 install. But even more than that, I'd like to know how you create such a list (more on that below). Occasionally, for debugging, I find it useful to have run-able, pure, older TDM versions. So I'd also like to see such a file list for 2.04 - 2.06, although I suspect that it may not be possible or might be impractical if said files no longer exist (i.e. packaged with release-specific, older content, for ready-made download). Using a crude hack, I've already managed to create "virgin" installations of older TDM releases, simply by manually stopping the incremental update at the proper step. But creating such virgin installations with an actual list of requisite files would be much nicer. In an attempt to answer my own "I'd like to know how you create such a list" question, I've just now (years overdue, it seems!) done a quick hack to the updater to do a "poor man's equivalent" of the '--dry-run' option that it has annoyingly advertised (via '--help') all along but which has never actually been implemented! Running the updater with that hack let me see just what files the updater wanted to download, without actually downloading all 3.41 GB. It runs in a mere 14 seconds. Having said that, I'd still really like to see your list of files for 2.07, to be sure I'm "on the same page". It would be heaven for me if I never again needed to run that updater app, save for maybe once each release (with my hack) to quickly generate a "requisite file list". I'll download all the files, slowly but reliably using 'wget' (or possibly even off-site somewhere with a fast Internet connection). Then I'll write a simple shell script to efficiently take care of the installation, with no Internet access required at that point! A point of interest: I ran the updater in an empty directory, but since it does not retain partially downloaded content on subsequent runs, it's simply not a good choice for downloading large files over unreliable and/or slow Internet connections. BTW, I had actually looked into trying your new 'tdm_sync' capability. But building it requires a too-new version of CMake, which triggers the need for other updates that I cannot abide at the moment, unfortunately.

Fortunately, the details in that bug report mentioned in that code snippet have shed more light on the problem. After reading it, I realized that there may be some sense in actually (counter-intuitively, I'd argue) running TDM (the game) after the 1st (failed) run of the TDM updater (where it went from 2.00 -> 2.06 instead of 2.07). So I tried that and saw evidence that the game was "Repacking" various PK4 files (presumably the ones containing ROQ/OGG/AVI files). Then I re-ran the same updater process with the same options. It acted differently this time, (needlessly?) downloading all 3 of the missions that come with a default TDM installation as well as (needfully, I suspect) 'tdm_base01.pk4'. But when it was done, the installation did appear to be a true 2.07 install, fortunately. And, in fact, the game now identifies as "TDM 2.07" on the main menu and has 2.07 features, like the new "frob highlighter" (nice addition BTW, STiFU!). So, although my immediate dilemma seems to be solved, it sounds like anyone who tries to do the upgrade the way that I did (regardless of OS) will encounter this odd glitch, which requires a rather unintuitive work-around. Furthermore, unless a new, large "base" TDM zip file that's built for 2.06 or higher is released, it will continue to be a problem. It seems to me like this problem would be affecting all new players, ones who have no 2.06 installation to build upon. Or am I missing something? I don't have the Internet throughput to test it, but I wonder if things would work any differently if I didn't supply a local copy of all the 'tdm_update_2.0{x}_to_2.0{x+1}.zip' files to the updater, thereby (presumably) forcing each update file to be downloaded and applied, 1-by-1. More broadly, just how do other users (Windows or Linux -- I don't think it matters) install TDM "from scratch"?

The log file from the 2nd run of the updater: tdm_update--run-2.log.txt

There's some sort of problem with the TDM updater whereby it won't create a proper 2.07 installation. Being on a slow Internet connection, my TDM installation/update procedure has always involved downloading just the latest update zip file (currently, 'tdm_update_2.06_to_2.07.zip') from the ModDB site and making that available to the updater locally, to avoid unreliable, slow downloads during the actual update. I followed the same procedure that I've always (successfully) done in the past: extract the huge, 2.3-GB 'THEDARKMODVersion2.0-StandaloneRelease.zip' file (downloaded once from ModDB and re-used many times over the years) change the permissions on all the PK4 files to be write-able (so that the updater can alter the content within, as needed) delete the existing, obsolete (2013-era), read-only 'tdm_update.log' file (which shouldn't be in the zip file to begin with) put a copy of all the 'tdm_update_2.0{x}_to_2.0{x+1}.zip' files amassed to date into that extracted directory download the latest version of the TDM updater app (from the TDM website) run the updater, with the options '--keep-update-packages --noselfupdate' On the 1st run, it proceeds in what appears to be normal fashion. There's no obvious indication that anything has gone amok. However, if you run the game, it comes up with the main menu showing "TDM 2.06", not the expected "TDM 2.07". Here's the TDM updater log file for this 1st run: tdm_update--run-1.log.txt Some analysis of the directory content and the updater's log file seems to indicate that the installation only made it to 2.06! As for the 'tdm_update_2.06_to_2.07.zip' file, I've verified the integrity ('zip -T'). I've also verified the filesize and checksum against the updater-downloaded 'tdm_version_info.txt' file: [UpdatePackage from 2.06 to 2.07] crc = 159677cb filesize = 427644498 package = tdm_update_2.06_to_2.07.zipSo I don't think there's anything wrong with that particular file. If I compare what the updater created as a "2.07" installation with an old, virgin 2.06 installation of mine, the only files that are different are ones that you'd expect to be: 'crc_info.txt', 'tdm_mirrors.txt', 'tdm_update.log', and 'tdm_version_info.txt'. If I run the updater a 2nd time (and all subsequent times) with the same options, it essentially does this: Applying differential update package... [=========================] 100.0% Adding: ExtLibs.dll [=========================] 100.0% Adding: ExtLibsx64.dll [=========================] 100.0% Adding: TheDarkModx64.exe [=========================] 100.0% Adding: thedarkmod.x64 [=========================] 100.0% Adding: vcredist_x64.exe [=========================] 100.0% Adding: vcredist_x86.exe [=========================] 100.0% Replacing: AUTHORS.txt [=========================] 100.0% Replacing: TheDarkMod.exe [=========================] 100.0% Replacing: tdm_update.exe [=========================] 100.0% Replacing: tdm_update.linux [=========================] 100.0% Replacing: thedarkmod.x86 [=========================] 100.0% File: Done applying the differential update.Due to website file upload limitations, the TDM updater log file for this 2nd run will be attached to a subsequent post. The updater log files on that 2nd and all subsequent runs are essentially identical, just with different timestamps and potentially use of a different mirror site. That's the repeatable scenario, the "crux of the matter", if you will. FWIW, I also dug more deeply into the problem. What I describe from here on is less thoroughly documented, but I'll mention what I did, in case it helps.... I tried removing all of the update zip files, hoping it would force the updater to update what is effectively a 2.06 installation but via a slow, painful download of that 408-MB 'tdm_update_2.06_to_2.07.zip'. But instead, it inexplicably wanted to download the 235-MB 'tdm_update_2.05_to_2.06.zip' file, so I aborted the update. Taking a different tack, I dug into the updater source code a bit and noticed this curious-looking snippet in '.../tdm_update/libtdm_update/Updater/UpdateController.cpp': //#4529 stgatilov: Since differential update does not uncompress unchanged OGG/ROQ files, //The hashes of pk4 files are now different from the ones on the server. //That's why we exit right away to avoid redoing the same differential update and redownloading the unchanged data. if (info.fromVersion == "2.05" && info.toVersion == "2.06") { TryToProceedTo(PostUpdateCleanup); break; }I played around with various changes to that code (both bumping the 2 version numbers and outright disabling the whole block), but to no avail. Although I'm running Linux (64-bit Slackware 14.2), I don't immediately see any reason why this would be OS-specific. This issue sounds different from the problem grayman had (running Windows), but I did get an "infinite updater loop" scenario when I hacked that code mentioned above, so maybe there's some correlation. I also noticed during some debugging that the installation I had was not being correctly recognized by the updater app as a valid 2.06 installation. When I looked into the updater's console output and log file, it seemed that every file about which the installer reported "SIZE MISMATCH" was one of the PK4 files that had embedded OGG files, although I did not check every case. I'm simply not able to figure out what's going on here, despite much effort, so I have to "punt" before my head explodes. Any input on this matter would be very much appreciated!

I agree with 'gnartsch'. I don't think there's any need to try to fix anything in this regard. The packager app had the bug originally. Unfortunately, it went undetected and a "band-aid" fix (which itself had a flaw) was applied to the updater app. Shortly after I became involved with TDM, I eventually found and fixed both flaws (back in late Dec 2016). IMHO, now that we've been through 2 TDM releases with the fixes in place and no reports of problems, we should be able to remove (at any convenient time) the entire "band-aid" (i.e. the entire block of code that does anything to the timestamps) in the updater app. Since the packager app (the true source of the problem) has been fixed, there should be no problems going forward, I believe.

I explained in a general way in the 1st post of this thread: Refreshing my memory by re-reading my notes on this from Dec 2016, I'll point you to the 'R_ParseImageProgram_r()' routine in the 'renderer/Image_program.cpp' file. Beyond that, I'll leave it to the author of that code to explain.

Apologies for not "closing the loop" on this. Only recently did I finally get back to re-visiting this issue. I wound up simply hacking the 'tdm_update' C++ code to crudely inject a large, obvious console message and a 10-second delay at the end of each version's update. This allowed me to simply monitor the installation and break out at the appropriate time. Very ugly, but it sufficed. Here's the actual code, injected at the end of the 'Updater::PerformDifferentialUpdateStep()' routine in the 'tdm_update/libtdm_update/Updater/Updater.cpp' file: Thanks again for all the suggestions.

Although that file that you referenced which comes with the source code ('COMPILING.txt' is the actual name) is pretty useless (IMHO), it does give the link to the Wiki "Compilation Guide" for TDM. Although that Wiki page certainly could use some attention, it does show one variant of a "scons" command. But, frankly, it's buried in the noise and easy to miss. I'm not a huge fan of wikis, especially for something as important as publishing a project's building instructions. Good that you've found the 'linuxBuild.sh' script. After my first few compiles of TDM, I've never used that script. It might be useful to others, though. I think some of the commands in it were superseded by developments since 2.05 was released. A lot has changed since the last time I was heavily into building TDM and I've also forgotten a lot of details. The official Linux builds are currently done by 'grayman', but (IIRC) he's said that he's offered to let someone else take that task over and nobody ever has. (I suspect he'd still be happy to offload that.) So anything new would undoubtedly need grayman's blessing. And with TDM 2.06 being beta-tested now in the push for its release, it's probably not the best time to ask him to devote time considering any future build changes, even if they might improve his situation in the long run. Taaaki is also involved in the official releases, so you'd probably need to understand what he does. AFAIK, none of that is published as a set of instructions and getting a reply from taaaki can sometimes take a while. I hope I don't sound too discouraging. I like what you're doing so far but I don't call any of the shots. I'm just trying to give a realistic picture of the situation so that you don't waste time and/or get frustrated. And, yes, the SVN trees are a mess, IMHO. This is not how it should be set up!

I think many of your questions are best answered by someone else, so I hope others will comment here. I can only speak for the Linux build. I've never compiled TDM under Windows (or Mac OSX). And I have no experience with Continuous Integration (CI) systems. (In fact, your use of the acronym sent me searching to understand it! ) So, heeding the words of Mark Twain, I will keep my mouth shut and let everyone think I'm a fool rather than open my mouth and remove all doubt. I will say that the various disorganized build instructions are undoubtedly somewhat obsolete, misleading, and incomplete. There are files in the source code, Wiki pages, and $DEITY-knows-what lurking elsewhere. Yikes, what a mess! Given the wide variety of Linux distributions, it's very difficult to thoroughly document the process of TDM building. Improving this has been discussed before. The TDM content is already split into source code and "assets". The source code tree is public (as seen in my earlier 'svn' command). The much larger "assets" tree is currently private. I suspect that the suggestion earlier in this thread that free hosting sites are too limited for TDM repositories might only apply to the "assets" tree. Site hosting is done by 'taaaki', so that's his area. Oh, and just for the record, my "build process" on TDM is simply: scons -Q -j2 BUILD_GAMEPAK=0 NO_GCH=1 BUILD=release There's certainly room for improvement in many areas of TDM, so I'd encourage any/all interest and suggestions.

While I'd never discourage anyone from getting involved with TDM, the primary (IMHO) concern which prompted Tels to start this thread has been addressed just over a year ago: -> svn log -r 6733 https://svn.thedarkmod.com/publicsvn/darkmod_src/trunk ------------------------------------------------------------------------ r6733 | grayman | 2016-12-22 07:42:03 -0500 (Thu, 22 Dec 2016) | 1 line Apply "reduce build time" linux patch from NightStalker. ------------------------------------------------------------------------When I first built TDM (under Linux) back then, it was utterly intolerable that the entire system was getting rebuilt every time, no matter how small the alterations, so I dug in and found the fix. Actually, I'm surprised to see this ancient (5+ years!) thread and to realize only now just how long that problem went unfixed, especially given how annoying it was and that the fix was (essentially) a 1-line change! As for using SCons, I realize that others may disagree, but even though I'd never used it prior to my involvement with TDM, I don't mind it at all. In fact, I rather like it. So, FWIW, I would discourage any change to the TDM build system that would replace SCons. Of course, a good, healthy discussion about potential improvements of any sort is always welcomed, IMO.

I just tried frobbing that item, running TDM 2.05 under (32-bit Slackware 14.2) Linux and it was exactly as you experienced. No matter what I did, it would not highlight. Curiosity got the best of me and I dug into it. It turns out that the fix is very simple. After a lot of trial-and-error testing, I eventually got the idea for the simplest fix from the last paragraph of the Wiki page about frobbing. Basically, when this line is added to the problematic entity (named 'atdm_loot_spilt_purse_silver_1'), it becomes properly frob-able once again: "frobbox_size" "0"To the map's authors: If you decide to make a change, I think you'll find that the "DresserFrobControl" entity is unneeded. In fact, removal of that entity, even without the addition of a "frobbox_size" specification on the loot purse entity, will make the purse frob-able as well. I'm no mapper, but if I understand correctly, the loot purse's default frob-box area intersects with the clip area of that "DresserFrobControl" entity. To CountMorillonite: If you'd like to try the fix, extract the contents of the mission's PK4 file (it's just a zip file with a ".pk4" extension) in place. To be safe, rename (or move) the original mission PK4 file once you've done that to avoid any possible "collision". Edit the (text-format) 'allthewayup.map' file. Find this entity (the un-frob-able loot purse): // entity 409 { "classname" "atdm:loot_spilt_purse_silver" "name" "atdm_loot_spilt_purse_silver_1" "inv_loot_value" "25" "origin" "4051.08 1667.42 -115.399" "rotation" "0.819153 -0.573577 0 0.573577 0.819153 0 0 0 1" } If, say, after the "inv_loot_value" line, you add the line mentioned earlier, the purse should become frob-able once again. Good luck (and thanks for reporting the bug)!

Possibly. I'd thought about that, but since the "assets" SVN tree is non-public (IIRC), I didn't want to consider that avenue, at least for purposes of this (public) thread. So, although I could (and might eventually) try the SVN route, it's not really ideal, for multiple reasons, unfortunately. But I do appreciate your suggestion and comment. It just seems to me like creating a "fully authentic" runnable older version of TDM should be far more straightforward than it is.

Thanks, Durandall. I didn't consider extracting the update files over an earlier installation although I seem to recall having done something like that many months ago, for a different reason. Today, I downloaded one of the so-called "Windows" moddb 'update' files and it had the exact same content as the one I use (under Linux, downloaded from one of the TDM mirrors) with 'tdm_update.linux'. So I would say that those files are actually meant to be used with the updater app, but... I suppose that if one were to take the somewhat unusual step of extracting all of the PK4 files that come in the base 2.00 installation, one could then do as you suggest and extract (without using the updater app) the "update" file(s) on top of the 2.00 installation. Technically, that would not yield the same set of files as when the updater app is run for a given 2.0x release because the app will delete old files (IIRC, both intra-PK4 files and PK4 files themselves) when they're obsolete. But, if one isn't a purist, that would probably yield a workable 2.0x installation, so thanks again for the suggestion. I'll try both nb1m's suggestion and Durandall's when I get a chance. Thanks, folks. If anyone has any other ideas, I'm still open to them.

Sorry, I may not have made myself totally clear. It's not the TDM executable that's the problem. So the availability of source code for old versions of TDM doesn't help (in this case). It's the whole "environment". Think "directory content", "resources", or "assets". How do I set up a directory with, say, the TDM 2.03 release, with the executable (of course) and with TDM-2.03-era PK4 files ("assets") and no other (newer or older) assets? Before I started this thread, I searched the TDM forums and found nothing helpful. I also tried various mild hacks to the 'tdm_update' source code and to its input files, but I was thwarted at each step (a longer story). I still figure that someone, somewhere, over the years of TDM must have had the same dilemma I'm in, no?

Missed your post initially, nb1m. Thanks for the info. But, short of trying to halt the updater at the proper point, which gives me pause, I think most of what I said in my reply to freyk is still relevant and I'm still quite befuddled. I may have to try what you suggest, though, because I see no easy way (ATM) to get what I want. I can't see wanting to go back much beyond TDM 2.03, but I can't even yet accomplish that!

Thanks, freyk. I got the large TDM 2.00 "base" zip file from the moddb site a couple years ago, but didn't think to look there for anything else. Unfortunately, maybe I'm misunderstanding your suggestion but, after looking at what's available on the moddb site, I still don't see how I can install an older version of TDM (to be precise, older than 2.05, but newer than 2.00). The "standalone" file seems to be for the 2.00 version only. And if one is to download an update (say the "2.00 to 2.01" update), IIUC it still requires one to run the TDM updater exec (under Windows or Linux) and that's the crux of the problem -- the updater app has no ability to do anything but install the latest version (2.05 ATM), AFAICT. If I supply it with the 2.00 to 2.01 upgrade file only, it will merrily go off and (painfully) download all the other upgrade files and (against my preference, in this case) bring the installation up to 2.05! Am I missing something here? And, in fact, I want to do this under Linux, not Windows, a fact I intentionally omitted from my post because I thought that it should not matter. I still think it doesn't matter, but I'm prepared to be wrong. Lastly, I'm wondering why those "The Dark Mod 2.0{0-4}-2.0{1-5} Update Package (Windows)" files on moddb all say "(Windows)" when they seem to be the exact same update files one would use with Linux (and, presumably, Mac OSX). It seems to me like someone would have wanted to install an old version of TDM at some point over the years, but it escapes me how to do it without some very ugly and/or tedious work-arounds. So, still very puzzled here... just what am I missing?

Pray tell, how does one go about installing an older version of TDM?

Excellent! Glad to hear this. I seem to recall that the Linux installation instructions were in need of some attention. But although I'm sure they could (should) be improved, it's probably quite difficult to come up with instructions that work well across all Linux variants. I'm not saying we shouldn't improve them, but it might be a challenge. I don't have a surplus of time, but I'll try to look into that issue at some point. What might be nice (if I can dream for a minute) would be to have a small test app that would demonstrate whether someone's hardware and OS setup was capable of running TDM, without having to first download the massive set of TDM installation files. I'm always happy to see more Linux TDM users, regardless of preferred distribution!

Excellent! Thank you. That's good to see because it tells me that it's running, but what I really want to see is the output of 'glxinfo' whenever you get the chance. OK, very good to have ruled that out as a possible issue. Sounds good. Fingers still crossed...