Jump to content
The Dark Mod Forums

Fixing the TDM build system


Tels
 Share

Recommended Posts

TDM currently uses Scons as the build system to build the execuables from the source code for Linux and Mac. (I guess the Windows side uses whatever MS uses).

 

This has been working more or less for a few years, and one shouldn't "fix" a working system. However, the merging of the D3 source code and the TDM source also brougth along a change to try to use pre-compiled headers. This, however, does somehow not work in the combination of scons, and gcc. Everytime a single line is changed anywhere (in fact, even if 0 lines are changed), the entire source code is recompiled from scratch. Due to the precompiled header, this goes a bit faster than before (16 mins vs 22 mins on my old PC), but it is still annoying having to wait that long, not to mention not being able to use the system during that time.

 

So, it is highly desirable to fix this problem. Here is your chance to fame. Go fix it!

 

Related thoughts:

 

* the minimal invasive solution is prefered - if it can be made with scons, the better! There is no real need to rip out out and replace scons with another build system, unless this brings massive advantages (frankly, the only one I can see is "make it work with precompiled headers"). However, if you do so, prepare to get swear words thrown at because a new build system is likely to bring other, yet unknown problems :) Nobody will complain, tho, if you make the change to whatever, and it just works, tho

 

Bonus points for:

 

* Creating an automated build system. Something living on the greater internet, syncing SVN every minute, and when detecting a new checkin, re-compiling the source automatically, then publishing the build log on the web, and make the result downloadable.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

I've been compiling and debugging some stuff using the Intel compiler set lately, having to fudge the solution each time is extremely annoying and a waste of 5 min :/

 

So yeah - CMake for a build system would be lovely. Windows users only need to generate the solutions once (and they could always just share one like atm).

 

You also have the option of CDash, which gives you your desired web interface and whatever ever.

 

Since I now know that at least there's someone else who'd like an alternative, I'll see if I can start hacking together the CMakeLists files, tho I have never written from scratch... should be interesting.

Link to comment
Share on other sites

 

(The post greebo linked to refers to that precompiled headers are now optional)

 

Unfortunately, this does not work - it is still recompiling everything (and/or not working at all, I forgot which, but I tried it and it did not work).

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Okay, good. Serpentine has the time to change the system entirely and get away from SCons. Sounds like a solution to the terribly hampering and threatening problem. Can't have Tels anywhere near the verge of quiting -- talk about the "End" being "Nigh".

 

Go Serpentine!

Edited by BlueSkyBullet
Link to comment
Share on other sites

TDM currently uses Scons as the build system to build the execuables from the source code for Linux and Mac. (I guess the Windows side uses whatever MS uses).

 

This has been working more or less for a few years, and one shouldn't "fix" a working system. However, the merging of the D3 source code and the TDM source also brougth along a change to try to use pre-compiled headers. This, however, does somehow not work in the combination of scons, and gcc. Everytime a single line is changed anywhere (in fact, even if 0 lines are changed), the entire source code is recompiled from scratch.

 

Is the repository available somewhere? I only see a 1.07 download. Some kind of github presence would be nice and would likely attract developers more easily.

Link to comment
Share on other sites

Is the repository available somewhere? I only see a 1.07 download. Some kind of github presence would be nice and would likely attract developers more easily.

 

We can also give you SVN access if you want. (Tho I am not sure if there is read-only access or if it is always full access). You'd need to pm/ask greebo for access.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

  • 3 weeks later...
  • 2 weeks later...

Outside the ongoing work to change the build system for the executable, would it

be possible to break more of the engine code into a DLL (etc)?

 

...such that the "old build method" could be used for prototyping while having the

nearly the same level of functional access to the formerly hidden parts of the code?

Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Link to comment
Share on other sites

That will not help, as the build builds executable and DLL at the same time on linux.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

  • 2 weeks later...

So it seems both icculus and dhewg have cmake Doom 3 builds?

 

I will post the icculus git in case it helps:

 

http://git.icculus.org/?p=icculus/iodoom3.git;a=summary

Please visit TDM's IndieDB site and help promote the mod:

 

http://www.indiedb.com/mods/the-dark-mod

 

(Yeah, shameless promotion... but traffic is traffic folks...)

Link to comment
Share on other sites

  • 5 years later...

Just wondering if this ever went anywhere or could use improvement? I've been wanting to offer to help out with this project for ages, but wasn't sure what I could offer. I have a lot of experience with stuff like Docker and automated build systems which might help here. Is there anything I could look at?

Link to comment
Share on other sites

Just wondering if this ever went anywhere or could use improvement? I've been wanting to offer to help out with this project for ages, but wasn't sure what I could offer. I have a lot of experience with stuff like Docker and automated build systems which might help here. Is there anything I could look at?

While I'd never discourage anyone from getting involved with TDM, the primary (IMHO) concern which prompted Tels to start this thread has been addressed just over a year ago:

-> svn log -r 6733 https://svn.thedarkmod.com/publicsvn/darkmod_src/trunk
------------------------------------------------------------------------
r6733 | grayman | 2016-12-22 07:42:03 -0500 (Thu, 22 Dec 2016) | 1 line

Apply "reduce build time" linux patch from NightStalker.
------------------------------------------------------------------------

When I first built TDM (under Linux) back then, it was utterly intolerable that the entire system was getting rebuilt every time, no matter how small the alterations, so I dug in and found the fix. Actually, I'm surprised to see this ancient (5+ years!) thread and to realize only now just how long that problem went unfixed, especially given how annoying it was and that the fix was (essentially) a 1-line change! :)

 

As for using SCons, I realize that others may disagree, but even though I'd never used it prior to my involvement with TDM, I don't mind it at all. In fact, I rather like it. So, FWIW, I would discourage any change to the TDM build system that would replace SCons. Of course, a good, healthy discussion about potential improvements of any sort is always welcomed, IMO.

Link to comment
Share on other sites

OK thanks for the reply. I have no preferences for the 'build system' (i.e. scons), but looking at the Wiki and the COMPILE.txt file in the source it looks like (and correct me if I'm wrong) a fairly manual build process, involving all the usual headaches regarding OS dependencies, configuration drift, 3rd party libraries that have to be compiled, library paths, etc, etc.

 

What is used for a consistent build environment? How do you know if the build is broken? What if it works on one user's PC and not the others?

 

Would using a free CI service (e.g. Travis or Circle) be desirable? I also think I saw that using Github isn't doable because of the size of the repository. Would splitting up the repo between code and assets help?

 

I'm wary of making suggestions without having the full picture - all I have to go by is the Wiki and the readmes in the source code, but if any of this sounds like it's worth looking into I'd be happy to do it.

Link to comment
Share on other sites

I think many of your questions are best answered by someone else, so I hope others will comment here. I can only speak for the Linux build. I've never compiled TDM under Windows (or Mac OSX). And I have no experience with Continuous Integration (CI) systems. (In fact, your use of the acronym sent me searching to understand it! :blush:) So, heeding the words of Mark Twain, I will keep my mouth shut and let everyone think I'm a fool rather than open my mouth and remove all doubt. ;)

I will say that the various disorganized build instructions are undoubtedly somewhat obsolete, misleading, and incomplete. There are files in the source code, Wiki pages, and $DEITY-knows-what lurking elsewhere. Yikes, what a mess! :wacko: Given the wide variety of Linux distributions, it's very difficult to thoroughly document the process of TDM building. Improving this has been discussed before.

The TDM content is already split into source code and "assets". The source code tree is public (as seen in my earlier 'svn' command). The much larger "assets" tree is currently private. I suspect that the suggestion earlier in this thread that free hosting sites are too limited for TDM repositories might only apply to the "assets" tree. Site hosting is done by 'taaaki', so that's his area.

Oh, and just for the record, my "build process" on TDM is simply:

scons -Q -j2 BUILD_GAMEPAK=0 NO_GCH=1 BUILD=release

There's certainly room for improvement in many areas of TDM, so I'd encourage any/all interest and suggestions.

Link to comment
Share on other sites

OK. That's the first time I've seen that build command, so either I didn't look hard enough or the docs are out of date. I'll give that a try.

 

For the record, I'm only looking at Linux at the moment as well, as that's where all my experience with this stuff is. But obviously getting something working for Windows as well would be just as valuable.

 

What I was thinking of doing was just creating a Docker image for the build environment, and seeing if we could use it in a free hosted CI service (or one that is free for open-source projects at least). Haven't really looked at any of them yet - just wanted to see if there was interest first. Also for it to be truly useful I would also probably want all the steps involved in packaging it up for distribution as well.

Link to comment
Share on other sites

OK got it to build in my Docker image (using the linuxBuild.sh script), but might be missing a couple of files...?

scons: done building targets.

real    3m12.136s
user    31m43.112s
sys     3m35.660s
mv: cannot stat 'gamex86-base.so': No such file or directory
cp: cannot stat 'tdm_game02.pk4': No such file or directory

EDIT: I also notice there are a lot of binaries included in the source, which helps contribute to the large repository size. This is another area where Docker would help - all this stuff would be in the Docker image instead of the source repository and you would have clear separation of code and dependencies. You could also have a different build image for each target platform (Debian, Ubuntu, MacOS, etc).

 

So what this would provide is:

 

- isolated, immutable build environments for all target platforms

- reduce repository size by moving binaries and other dependencies to Docker build images (and possibly pave the way to moving to Github - which may help attract more contributors, and make it easier to use free online services. I'm not sure how many support SVN for example...)

- provide an easy way for anyone to build the project without having to worry about missing libraries, dependencies, etc

 

 

Happy to share what I've got so far, and provide more info for anyone who's interested...

Edited by demilich666
  • Like 1
Link to comment
Share on other sites

Although that file that you referenced which comes with the source code ('COMPILING.txt' is the actual name) is pretty useless (IMHO), it does give the link to the Wiki "Compilation Guide" for TDM. Although that Wiki page certainly could use some attention, it does show one variant of a "scons" command. But, frankly, it's buried in the noise and easy to miss. I'm not a huge fan of wikis, especially for something as important as publishing a project's building instructions.

Good that you've found the 'linuxBuild.sh' script. After my first few compiles of TDM, I've never used that script. It might be useful to others, though. I think some of the commands in it were superseded by developments since 2.05 was released. A lot has changed since the last time I was heavily into building TDM and I've also forgotten a lot of details.

The official Linux builds are currently done by 'grayman', but (IIRC) he's said that he's offered to let someone else take that task over and nobody ever has. (I suspect he'd still be happy to offload that.) So anything new would undoubtedly need grayman's blessing. And with TDM 2.06 being beta-tested now in the push for its release, it's probably not the best time to ask him to devote time considering any future build changes, even if they might improve his situation in the long run. Taaaki is also involved in the official releases, so you'd probably need to understand what he does. AFAIK, none of that is published as a set of instructions and getting a reply from taaaki can sometimes take a while.

I hope I don't sound too discouraging. I like what you're doing so far but I don't call any of the shots. I'm just trying to give a realistic picture of the situation so that you don't waste time and/or get frustrated.

And, yes, the SVN trees are a mess, IMHO. This is not how it should be set up!

Link to comment
Share on other sites

stgatilov and duzenko worked on the build changes for 2.06.

They are now stable and the Linux builds are easier to do than in 2.05.

I'm not in favor of replacing what we currently have. It works well when all the necessary parts are in place.

Any future changes from 2.07 on must have buy-in from the coders.

There were at least 3 efforts mounted in the past to move off of scons for the Linux build, and they all died. I see no reason to start another such effort, since the current system works fine.

If you want to do something for your personal use or a learning exercise, that's fine. But none of it goes into SVN unless it's been vetted by the team.

And I seriously doubt if anyone will accept moving off of MSVS; we have too much invested in that tool.

  • Like 1
Link to comment
Share on other sites

Grayman: believe me I understand where you are coming from....

 

This wouldn't require replacing scons or MSVS (I'm assuming this means Visual Studio). Rather it still uses them and complements them. Think of it as abstracting the build process up into another layer - one that is far easier to manage (both for newbies and experiences devs), as well as paving the way for other possible improvements like mentioned earlier.

 

Also I'm not interested in suggesting anything goes into SVN - this is just a demonstration on how things can be improved.

 

For those interested, I've put what I have up on Github, with a readme: https://github.com/timwebster9/darkmod-build

  • Like 2
Link to comment
Share on other sites

I know nothing about docker except a read a few months ago where it was ridiculed.

I don't see how installing a few more tools (cmake, docker, etc) makes life easier for a newbie like me, not that I have any experience or interest in linux builds. Things are definitely easier on Windows, where I can just download VS and open a solution in it.

Link to comment
Share on other sites

I'm not suggesting CMake. I'm not suggesting replacing scons. All I'm suggesting is a way to create and use a build environment that is instantly available to everyone and will work every single time.

 

It would be useful for those interested in Linux builds, or those who work on Mac or Linux. You can even run Docker on Windows, and build for Linux from your Windows environment. Or run in it Mac and also build for Linux. That's kind of the point.

 

You can think of Docker has an extremely lightweight Virtual Machine that you can spin up instantly (literally milliseconds) with any OS in order to run a particular task (in this case, to build Darkmod), and then it disappears. The OS (or image) you can build yourself and put anything you want in it (see the Dockerfile in the Github link I provided). Again, this is just one use case for Docker - the others aren't really applicable here.

 

If you don't have any interest or experience with Linux builds, then it won't be of interest - totally understandable. But for someone like me, who wants to build the project in the quickest and easiest way possible, and have it work every single time, it is very useful.

 

Regarding Windows - it's a different story and not really part of this Docker thing. But I was also looking at this whole thing from an automation angle like Tels mentioned in the first post in this thread. This is also doable with VS but like I said it would require a different approach.

 

You would get a better idea of how it's helpful if you worked through the Github example I provided - but if it's not your area of interest then by all means give it a miss.

 

Also - I would be interested in reading whatever article you said ridiculed Docker. The only people that I have heard ridicule Docker are those who have never used it and don't really understand the benefits of it. I've worked in software development for 18 years, and container technology is the single most useful thing to emerge in the mainstream in the past few years (along with cloud computing).

  • Like 1
Link to comment
Share on other sites

To better illustrate, let's contrast the different ways of doing this:

 

The Old Way

 

1. Obtain a build machine (PC, VM, or server)

2. Install required OS (Ubuntu, Debian, Windows, Mac, etc)

3. Update OS

4. Download and install required OS dependencies (e.g. apt install mesa-common-dev, scons, subversion, etc).

5. Download and install required project dependencies (e.g. Boost). May require building from source, etc

6. Download project source

7. Run the build script, and hope (or pray) that it 'just works'. Which it won't.

8. Spend the next 3 hours figuring out what's wrong with your setup.

9. finally get it working

10. Update some OS dependency, and find that your project build no longer works.

11. Someone updates the build script or adds a new dependency, and the build no longer works because they haven't updated the Wiki

12. etc, etc

 

Note that instead of step 5. people may opt to include libraries in the source tree. This leads to repository bloat.

 

The New Way (with Docker installed)

 

1. Download project source

2. Run single docker command. This always works because the 'build image' will have been updated with any required changes as well - if it hadn't the code wouldn't build for the person who made changes to the source code.

 

Note the 'build image' doesn't have to be built by the user - it can be shared via dockerhub and 'pulled' locally. This is done implicitly if it doesn't exist locally when you do a 'docker run'. The user doesn't have to know anything about it at all, in fact.

  • Like 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

  • Recent Status Updates

    • STiFU

      We are taking our son on his very first holiday trip to see the sea for the first time. 🙂 Will be back in a week.
      · 0 replies
    • Gilkar

      When I was a young man my father was so ignorant I could hardly stand to have him around. As I grew older I was amazed at how much the old man had learned in such a short time.
      · 1 reply
    • jaxa

      RTX 3090 Super, RTX 3070 Ti 16 GB, RTX 2060 12 GB
      https://wccftech.com/nvidia-launching-rtx-3090-super-rtx-3070-ti-16gb-and-rtx-2060-12gb-by-january-2022/
      · 0 replies
    • duzenko

      CPU benchmark time - compiling DarkRadiant (2nd run)
      i5 8600K 6C/6T@4.4GHz DDR4 2x2133MHz 9MB cache
      Parallel builds: 1. 3:57 Parallel builds: 6 (default). 2:28 r5 1600AF 6C/12T@3.3GHz DDR4 1x2666MHz 16 MB cache, temp folder on HDD
      Parallel builds: 1. 5:05 Parallel builds: 4. 2:47 Parallel builds: 6. 2:55 Parallel builds: 12 (default). 2:57
      · 6 replies
    • nbohr1more

      Status updates are back so it is also a good time to return to contests!
      https://forums.thedarkmod.com/index.php?/topic/21095-christmas-connections-contest-2021
       
      · 0 replies
×
×
  • Create New...