Jump to content


Photo

Fixing the TDM build system

[censored]

  • Please log in to reply
33 replies to this topic

#1 Tels

Tels

    Mod hero

  • Member
  • PipPipPipPipPip
  • 15024 posts

Posted 11 April 2012 - 12:54 AM

TDM currently uses Scons as the build system to build the execuables from the source code for Linux and Mac. (I guess the Windows side uses whatever MS uses).

This has been working more or less for a few years, and one shouldn't "fix" a working system. However, the merging of the D3 source code and the TDM source also brougth along a change to try to use pre-compiled headers. This, however, does somehow not work in the combination of scons, and gcc. Everytime a single line is changed anywhere (in fact, even if 0 lines are changed), the entire source code is recompiled from scratch. Due to the precompiled header, this goes a bit faster than before (16 mins vs 22 mins on my old PC), but it is still annoying having to wait that long, not to mention not being able to use the system during that time.

So, it is highly desirable to fix this problem. Here is your chance to fame. Go fix it!

Related thoughts:

* the minimal invasive solution is prefered - if it can be made with scons, the better! There is no real need to rip out out and replace scons with another build system, unless this brings massive advantages (frankly, the only one I can see is "make it work with precompiled headers"). However, if you do so, prepare to get swear words thrown at because a new build system is likely to bring other, yet unknown problems :) Nobody will complain, tho, if you make the change to whatever, and it just works, tho

Bonus points for:

* Creating an automated build system. Something living on the greater internet, syncing SVN every minute, and when detecting a new checkin, re-compiling the source automatically, then publishing the build log on the web, and make the result downloadable.
"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

"Remember: If the game lets you do it, it's not cheating." -- Xarax

#2 greebo

greebo

    Heroic Coder

  • Root
  • 16001 posts

Posted 11 April 2012 - 03:20 AM

http://forums.thedar...post__p__275410

#3 Serpentine

Serpentine

    Uber member

  • Active Developer
  • PipPipPipPip
  • 2903 posts

Posted 11 April 2012 - 04:52 AM

I've been compiling and debugging some stuff using the Intel compiler set lately, having to fudge the solution each time is extremely annoying and a waste of 5 min :/

So yeah - CMake for a build system would be lovely. Windows users only need to generate the solutions once (and they could always just share one like atm).

You also have the option of CDash, which gives you your desired web interface and whatever ever.

Since I now know that at least there's someone else who'd like an alternative, I'll see if I can start hacking together the CMakeLists files, tho I have never written from scratch... should be interesting.

#4 Tels

Tels

    Mod hero

  • Member
  • PipPipPipPipPip
  • 15024 posts

Posted 11 April 2012 - 02:26 PM

http://forums.thedarkmod.com/topic/13275-how-to-use-i18n-script/page__view__findpost__p__275410


(The post greebo linked to refers to that precompiled headers are now optional)

Unfortunately, this does not work - it is still recompiling everything (and/or not working at all, I forgot which, but I tried it and it did not work).
"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

"Remember: If the game lets you do it, it's not cheating." -- Xarax

#5 BlueSkyBullet

BlueSkyBullet

    Member

  • Banned
  • PipPip
  • 77 posts

Posted 12 April 2012 - 02:58 PM

Okay, good. Serpentine has the time to change the system entirely and get away from SCons. Sounds like a solution to the terribly hampering and threatening problem. Can't have Tels anywhere near the verge of quiting -- talk about the "End" being "Nigh".

Go Serpentine!

Edited by BlueSkyBullet, 12 April 2012 - 03:01 PM.


#6 Serpentine

Serpentine

    Uber member

  • Active Developer
  • PipPipPipPip
  • 2903 posts

Posted 12 April 2012 - 05:00 PM

Well, I spent some time moving over to qmake... now to move that over to CMake.

#7 Sgt Pinback

Sgt Pinback

    Newbie

  • Member
  • Pip
  • 7 posts

Posted 12 April 2012 - 06:09 PM

TDM currently uses Scons as the build system to build the execuables from the source code for Linux and Mac. (I guess the Windows side uses whatever MS uses).

This has been working more or less for a few years, and one shouldn't "fix" a working system. However, the merging of the D3 source code and the TDM source also brougth along a change to try to use pre-compiled headers. This, however, does somehow not work in the combination of scons, and gcc. Everytime a single line is changed anywhere (in fact, even if 0 lines are changed), the entire source code is recompiled from scratch.


Is the repository available somewhere? I only see a 1.07 download. Some kind of github presence would be nice and would likely attract developers more easily.

#8 Serpentine

Serpentine

    Uber member

  • Active Developer
  • PipPipPipPip
  • 2903 posts

Posted 12 April 2012 - 07:59 PM

Nope, but if you're interested in helping out - let us know.

We use subversion and don't host publicly as the repositories are all far larger than any of the acceptable free services allow.

#9 Tels

Tels

    Mod hero

  • Member
  • PipPipPipPipPip
  • 15024 posts

Posted 13 April 2012 - 12:26 AM

Is the repository available somewhere? I only see a 1.07 download. Some kind of github presence would be nice and would likely attract developers more easily.


We can also give you SVN access if you want. (Tho I am not sure if there is read-only access or if it is always full access). You'd need to pm/ask greebo for access.
"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

"Remember: If the game lets you do it, it's not cheating." -- Xarax

#10 Springheel

Springheel

    Creative Director (retired)

  • Admin
  • 36763 posts

Posted 28 April 2012 - 11:46 AM

Any progress here?

#11 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8214 posts

Posted 06 May 2012 - 04:30 PM

Outside the ongoing work to change the build system for the executable, would it
be possible to break more of the engine code into a DLL (etc)?

...such that the "old build method" could be used for prototyping while having the
nearly the same level of functional access to the formerly hidden parts of the code?
Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#12 Tels

Tels

    Mod hero

  • Member
  • PipPipPipPipPip
  • 15024 posts

Posted 07 May 2012 - 12:39 AM

That will not help, as the build builds executable and DLL at the same time on linux.
"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

"Remember: If the game lets you do it, it's not cheating." -- Xarax

#13 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8214 posts

Posted 21 May 2012 - 06:04 PM

So it seems both icculus and dhewg have cmake Doom 3 builds?

I will post the icculus git in case it helps:

http://git.icculus.o...3.git;a=summary
Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#14 demilich666

demilich666

    Member

  • Member
  • PipPip
  • 16 posts

Posted 26 December 2017 - 06:20 PM

Just wondering if this ever went anywhere or could use improvement?  I've been wanting to offer to help out with this project for ages, but wasn't sure what I could offer.  I have a lot of experience with stuff like Docker and automated build systems which might help here.  Is there anything I could look at?



#15 NightStalker

NightStalker

    Linux hero

  • Development Role
  • PipPip
  • 162 posts

Posted 27 December 2017 - 08:42 AM

Just wondering if this ever went anywhere or could use improvement?  I've been wanting to offer to help out with this project for ages, but wasn't sure what I could offer.  I have a lot of experience with stuff like Docker and automated build systems which might help here.  Is there anything I could look at?

While I'd never discourage anyone from getting involved with TDM, the primary (IMHO) concern which prompted Tels to start this thread has been addressed just over a year ago:

-> svn log -r 6733 https://svn.thedarkmod.com/publicsvn/darkmod_src/trunk
------------------------------------------------------------------------
r6733 | grayman | 2016-12-22 07:42:03 -0500 (Thu, 22 Dec 2016) | 1 line

Apply "reduce build time" linux patch from NightStalker.
------------------------------------------------------------------------

When I first built TDM (under Linux) back then, it was utterly intolerable that the entire system was getting rebuilt every time, no matter how small the alterations, so I dug in and found the fix.  Actually, I'm surprised to see this ancient (5+ years!) thread and to realize only now just how long that problem went unfixed, especially given how annoying it was and that the fix was (essentially) a 1-line change! :)

As for using SCons, I realize that others may disagree, but even though I'd never used it prior to my involvement with TDM, I don't mind it at all.  In fact, I rather like it.  So, FWIW, I would discourage any change to the TDM build system that would replace SCons.  Of course, a good, healthy discussion about potential improvements of any sort is always welcomed, IMO.



#16 demilich666

demilich666

    Member

  • Member
  • PipPip
  • 16 posts

Posted 27 December 2017 - 03:49 PM

OK thanks for the reply.  I have no preferences for the 'build system' (i.e. scons), but looking at the Wiki and the COMPILE.txt file in the source it looks like (and correct me if I'm wrong) a fairly manual build process, involving all the usual headaches regarding OS dependencies, configuration drift, 3rd party libraries that have to be compiled, library paths, etc, etc.

 

What is used for a consistent build environment?  How do you know if the build is broken?  What if it works on one user's PC and not the others?

 

Would using a free CI service (e.g. Travis or Circle) be desirable?  I also think I saw that using Github isn't doable because of the size of the repository.  Would splitting up the repo between code and assets help?

 

I'm wary of making suggestions without having the full picture - all I have to go by is the Wiki and the readmes in the source code, but if any of this sounds like it's worth looking into I'd be happy to do it.



#17 NightStalker

NightStalker

    Linux hero

  • Development Role
  • PipPip
  • 162 posts

Posted 27 December 2017 - 06:09 PM

I think many of your questions are best answered by someone else, so I hope others will comment here.  I can only speak for the Linux build.  I've never compiled TDM under Windows (or Mac OSX).  And I have no experience with Continuous Integration (CI) systems.  (In fact, your use of the acronym sent me searching to understand it! :blush:)  So, heeding the words of Mark Twain, I will keep my mouth shut and let everyone think I'm a fool rather than open my mouth and remove all doubt. ;)

I will say that the various disorganized build instructions are undoubtedly somewhat obsolete, misleading, and incomplete.  There are files in the source code, Wiki pages, and $DEITY-knows-what lurking elsewhere.  Yikes, what a mess! :wacko: Given the wide variety of Linux distributions, it's very difficult to thoroughly document the process of TDM building.  Improving this has been discussed before.

The TDM content is already split into source code and "assets".  The source code tree is public (as seen in my earlier 'svn' command).  The much larger "assets" tree is currently private.  I suspect that the suggestion earlier in this thread that free hosting sites are too limited for TDM repositories might only apply to the "assets" tree.  Site hosting is done by 'taaaki', so that's his area.

Oh, and just for the record, my "build process" on TDM is simply:

scons -Q -j2 BUILD_GAMEPAK=0 NO_GCH=1 BUILD=release

There's certainly room for improvement in many areas of TDM, so I'd encourage any/all interest and suggestions.



#18 demilich666

demilich666

    Member

  • Member
  • PipPip
  • 16 posts

Posted 27 December 2017 - 07:10 PM

OK.  That's the first time I've seen that build command, so either I didn't look hard enough or the docs are out of date.  I'll give that a try.

 

For the record, I'm only looking at Linux at the moment as well, as that's where all my experience with this stuff is.  But obviously getting something working for Windows as well would be just as valuable.

 

What I was thinking of doing was just creating a Docker image for the build environment, and seeing if we could use it in a free hosted CI service (or one that is free for open-source projects at least).  Haven't really looked at any of them yet - just wanted to see if there was interest first.  Also for it to be truly useful I would also probably want all the steps involved in packaging it up for distribution as well.



#19 demilich666

demilich666

    Member

  • Member
  • PipPip
  • 16 posts

Posted 27 December 2017 - 08:08 PM

OK got it to build in my Docker image (using the linuxBuild.sh script), but might be missing a couple of files...?

scons: done building targets.

real    3m12.136s
user    31m43.112s
sys     3m35.660s
mv: cannot stat 'gamex86-base.so': No such file or directory
cp: cannot stat 'tdm_game02.pk4': No such file or directory

EDIT: I also notice there are a lot of binaries included in the source, which helps contribute to the large repository size.  This is another area where Docker would help - all this stuff would be in the Docker image instead of the source repository and you would have clear separation of code and dependencies.  You could also have a different build image for each target platform (Debian, Ubuntu, MacOS, etc). 

 

So what this would provide is:

 

- isolated, immutable build environments for all target platforms

- reduce repository size by moving binaries and other dependencies to Docker build images (and possibly pave the way to moving to Github - which may help attract more contributors, and make it easier to use free online services.  I'm not sure how many support SVN for example...)

- provide an easy way for anyone to build the project without having to worry about missing libraries, dependencies, etc

 

 

Happy to share what I've got so far, and provide more info for anyone who's interested...


Edited by demilich666, 27 December 2017 - 08:47 PM.

  • Anderson likes this

#20 NightStalker

NightStalker

    Linux hero

  • Development Role
  • PipPip
  • 162 posts

Posted 27 December 2017 - 09:25 PM

Although that file that you referenced which comes with the source code ('COMPILING.txt' is the actual name) is pretty useless (IMHO), it does give the link to the Wiki "Compilation Guide" for TDM.  Although that Wiki page certainly could use some attention, it does show one variant of a "scons" command.  But, frankly, it's buried in the noise and easy to miss.  I'm not a huge fan of wikis, especially for something as important as publishing a project's building instructions.

Good that you've found the 'linuxBuild.sh' script.  After my first few compiles of TDM, I've never used that script.  It might be useful to others, though.  I think some of the commands in it were superseded by developments since 2.05 was released.  A lot has changed since the last time I was heavily into building TDM and I've also forgotten a lot of details.

The official Linux builds are currently done by 'grayman', but (IIRC) he's said that he's offered to let someone else take that task over and nobody ever has.  (I suspect he'd still be happy to offload that.)  So anything new would undoubtedly need grayman's blessing.  And with TDM 2.06 being beta-tested now in the push for its release, it's probably not the best time to ask him to devote time considering any future build changes, even if they might improve his situation in the long run.  Taaaki is also involved in the official releases, so you'd probably need to understand what he does.  AFAIK, none of that is published as a set of instructions and getting a reply from taaaki can sometimes take a while.

I hope I don't sound too discouraging.  I like what you're doing so far but I don't call any of the shots.  I'm just trying to give a realistic picture of the situation so that you don't waste time and/or get frustrated.

And, yes, the SVN trees are a mess, IMHO.  This is not how it should be set up!



#21 grayman

grayman

    Master Builder, Coder

  • Active Developer
  • PipPipPipPipPip
  • 12159 posts

Posted 27 December 2017 - 10:01 PM

stgatilov and duzenko worked on the build changes for 2.06.

They are now stable and the Linux builds are easier to do than in 2.05.

I'm not in favor of replacing what we currently have. It works well when all the necessary parts are in place.

Any future changes from 2.07 on must have buy-in from the coders.

There were at least 3 efforts mounted in the past to move off of scons for the Linux build, and they all died. I see no reason to start another such effort, since the current system works fine.

If you want to do something for your personal use or a learning exercise, that's fine. But none of it goes into SVN unless it's been vetted by the team.

And I seriously doubt if anyone will accept moving off of MSVS; we have too much invested in that tool.


  • Anderson likes this

#22 demilich666

demilich666

    Member

  • Member
  • PipPip
  • 16 posts

Posted 28 December 2017 - 08:29 AM

Grayman: believe me I understand where you are coming from....

 

This wouldn't require replacing scons or MSVS (I'm assuming this means Visual Studio).  Rather it still uses them and complements them.  Think of it as abstracting the build process up into another layer - one that is far easier to manage (both for newbies and experiences devs), as well as paving the way for other possible improvements like mentioned earlier.

 

Also I'm not interested in suggesting anything goes into SVN - this is just a demonstration on how things can be improved.

 

For those interested, I've put what I have up on Github, with a readme: https://github.com/t...9/darkmod-build


  • Anderson likes this

#23 duzenko

duzenko

    Advanced Member

  • Active Developer
  • PipPipPip
  • 951 posts

Posted 28 December 2017 - 08:41 AM

I know nothing about docker except a read a few months ago where it was ridiculed.

I don't see how installing a few more tools (cmake, docker, etc) makes life easier for a newbie like me, not that I have any experience or interest in linux builds. Things are definitely easier on Windows, where I can just download VS and open a solution in it.



#24 demilich666

demilich666

    Member

  • Member
  • PipPip
  • 16 posts

Posted 28 December 2017 - 09:36 AM

I'm not suggesting CMake.  I'm not suggesting replacing scons.  All I'm suggesting is a way to create and use a build environment that is instantly available to everyone and will work every single time. 

 

It would be useful for those interested in Linux builds, or those who work on Mac or Linux.  You can even run Docker on Windows, and build for Linux from your Windows environment. Or run in it Mac and also build for Linux. That's kind of the point.

 

You can think of Docker has an extremely lightweight Virtual Machine that you can spin up instantly (literally milliseconds) with any OS in order to run a particular task (in this case, to build Darkmod), and then it disappears.  The OS (or image) you can build yourself and put anything you want in it (see the Dockerfile in the Github link I provided).  Again, this is just one use case for Docker - the others aren't really applicable here.

 

If you don't have any interest or experience with Linux builds, then it won't be of interest - totally understandable.  But for someone like me, who wants to build the project in the quickest and easiest way possible, and have it work every single time, it is very useful.

 

Regarding Windows - it's a different story and not really part of this Docker thing.  But I was also looking at this whole thing from an automation angle like Tels mentioned in the first post in this thread.  This is also doable with VS but like I said it would require a different approach.

 

You would get a better idea of how it's helpful if you worked through the Github example I provided - but if it's not your area of interest then by all means give it a miss.

 

Also - I would be interested in reading whatever article you said ridiculed Docker.  The only people that I have heard ridicule Docker are those who have never used it and don't really understand the benefits of it.  I've worked in software development for 18 years, and container technology is the single most useful thing to emerge in the mainstream in the past few years (along with cloud computing).


  • Anderson likes this

#25 demilich666

demilich666

    Member

  • Member
  • PipPip
  • 16 posts

Posted 28 December 2017 - 10:05 AM

To better illustrate, let's contrast the different ways of doing this:

 

The Old Way

 

1. Obtain a build machine (PC, VM, or server)

2. Install required OS (Ubuntu, Debian, Windows, Mac, etc)

3. Update OS

4. Download and install required OS dependencies (e.g. apt install mesa-common-dev, scons, subversion, etc).

5. Download and install required project dependencies (e.g. Boost).  May require building from source, etc

6.  Download project source

7. Run the build script, and hope (or pray) that it 'just works'.  Which it won't.

8. Spend the next 3 hours figuring out what's wrong with your setup.

9.  finally get it working

10.  Update some OS dependency, and find that your project build no longer works.

11.  Someone updates the build script or adds a new dependency, and the build no longer works because they haven't updated the Wiki

12.  etc, etc

 

Note that instead of step 5. people may opt to include libraries in the source tree.  This leads to repository bloat.

 

The New Way (with Docker installed)

 

1.  Download project source

2.  Run single docker command.  This always works because the 'build image' will have been updated with any required changes as well - if it hadn't the code wouldn't build for the person who made changes to the source code.

 

Note the 'build image' doesn't have to be built by the user - it can be shared via dockerhub and 'pulled' locally.  This is done implicitly if it doesn't exist locally when you do a 'docker run'.  The user doesn't have to know anything about it at all, in fact.


  • Anderson likes this




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users