Jump to content


Photo

My experience with The Dark Mod under Linux

linux

  • Please log in to reply
10 replies to this topic

#1 Hamlet

Hamlet

    Member

  • Member
  • PipPip
  • 24 posts

Posted 21 December 2016 - 10:54 PM

I am reporting here my experience with compiling TheDarkMod from sources on Linux.
 
I am going to attach some patches, but beware that especially the ones related to build may be specific to the system.
I have used a Gentoo Linux distribution, with multilib enabled (that means that I have a lot of libraries compiled also in 32 bit version).
Relevant packages:
GNU compiler: 5.4.0
Boost libraries: 1.62.0
The Dark Mode: SVN trunk (2.06+, last commit: 6731, Dec 15, 2016).
 
Finally, I am new to this program and I have gone mostly from scratch, likely ignoring a big deal of lore that may demonstrate some of this as obsolete, non-optimal or plain wrong and dangerous. I'll try to catch up with the threads developing on the subject.
 
I downloaded the sources with:
git svn clone --stdlayout https://svn.thedarkmod.com/publicsvn/darkmod_src thedarkmod
git tag FromSVN # to keep track of my own changes

(because I am the GIT type), and it took long. Nothing I am writing here depends on using GIT instead of SVN anyway.

 
First, I should have applied immediately NightStalker's patch. I didn't, and I regret. It's from this forum, at http://forums.thedar...nux-compilation.
 
Also, I did some tuning on the build scripts, that are good for general purpose but not for distribution (the distribution plays safe and assumes Pentium3). I enabled as many CPUs as available and the "native" instruction set (Attached File  build_tuning.patch.txt   1.46KB   5 downloads).
 
IDLib tries to forward-declare std::basic_string, which is a standard C++ class; unfortunately its declaration is an implementation detail, and the one in PUGIXML turns out to be incompatible with the one shipped by GCC:
error: reference to ‘basic_string’ is ambiguous
std::basic_string<char, std::char_traits<char>, std::allocator<char> > PUGIXML_FUNCTION as_utf8(const std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >& str);

And the system string class from GCC 5 is different from the one that was used to build the Boost libraries that are shipped. Sadly, the only thing I could manage was to hard-code the system Boost libraries in the build paths. In my case they are at /usr/lib32, so the magic took the form of:

grep -r -l -e 'lib/libboost' * | xargs sed -i -e 's@/usr/local/lib/libboost\(.*\).a@/usr/lib32/libboost\1.so@g' -e 's@#.*linux/boost/lib/libboost\(.*\).a@/usr/lib32/libboost\1.so@g'

Attached File  boost_link.sh.txt   227bytes   5 downloads (note the "/usr/lib32/" path in two places).

This, and changing PUGI header (Attached File  pugi_strings.patch.txt   693bytes   11 downloads) did the job.
 
Now, there are linking problems with duplicate definitions. I really do not understand how that thing was supposed to work, but anyway I changed the definition of INLINE for GCC so that it avoids emitting multiple definitions for the same functions. Also I had to manually fix a couple of places for the same reason (Attached File  inline.patch.txt   3.35KB   7 downloads).
 
Next, a recent commit added a source file (idlib/geometry/RenderMatrix.cpp) but it did not add it to the source list for Linux build. There is also some piece that is Windows specific and that needs to be disabled (Attached File  RenderMatrix_in_Linux.patch.txt   1.1KB   7 downloads).
 
After this, running ./linuxBuild.sh produces gamesx86.so and thedarkmod.x86, which I copied manually into the directory where I had a downloaded installation (2.04), and that makes it a 2.06. The updater has still Boost link problems, so I use the shipped one.
 
Just that. Now compilation works to me all the way from commit 6600 to 6731, with some exceptions here and there. The first commit works well enough that I could actually start playing Saint Lucia. The latter, though, is buggy in that my character is always considered in deep darkness: convenient for him, but not for my gameplay. More about this in another post, I guess.
 
If you want to give these patches a try, you can run
patch -p1 < patchname.patch

in the main source directory (the attached patches should be renamed, with .txt removed; the shell script operates the substitution above).

 

Edited by Hamlet, 22 December 2016 - 11:31 PM.

  • Crinkelite, duzenko, nbohr1more and 1 other like this

#2 NightStalker

NightStalker

    Linux hero

  • Development Role
  • PipPip
  • 158 posts

Posted 22 December 2016 - 01:31 AM

Nice work, Hamlet!

It looks like we've been trodding some of the same ground, unfortunately. I probably should have posted the patches I was using to make latest SVN build, but even though it built, it would not run. And I wasn't confident that some of the patches I'd applied were robust enough for all OSes, so I never published them. My apologies if that caused you any wasted time. :(

I think your 'inline.patch' accomplishes the same thing that my 2 DevIL imaging library patches (used against 2.04 source code in this case) did from this post. But it looks like we took different approaches. I haven't had a chance yet to see what the effect of the different approaches might be, but I'll take a closer look tomorrow.

I think Crinkelite and duzenko (both now, at least part-time, running 32-bit Ubuntu 16.10 with GCC 6.2.0, IIRC) will be able to use your 'pugi_strings.patch' to great advantage. I'm running GCC 5.3.0 compared to your 5.4.0, and I've never seen those errors. But I'll still apply your patch to make sure that it has no negative effect, as an extra datapoint for those who decide to apply it.

One comment on your 'build_tuning.patch'.... With my SCons patch, there should be no need to do the 'touch scons.signatures.dblite' step in the 'linuxBuild.sh' file at all now, which is why that patch eliminated it entirely. Did you find that it's still needed on your setup? Just curious. I'm pretty sure that if you deleted the signature file, SCons will still work, but as my post mentioned, that was not the case in the past, hence someone's addition of the 'touch' in the script as a less-than-optimal work-around.

The other 2 things in that patch (CPU architecture and maximizing CPU core use) were things I had not even considered, so I really like those bits and will apply those parts of your patch to my local SVN branch ASAP.

As for the Boost libraries, I've taken a different tack, which I had not yet discussed on the forum before now. Just a few days ago, I'd compiled and run TDM using the Boost headers and library that come with Slackware 14.2 (and probably a lot of other Linux distros these days), thereby dropping 124 MB (!!!) of header files and about 7 MB of static library files from the TDM source distribution. But I was waiting to bring that up until after the 2.05-beta hubbub died down. I just wanted to prove that I could compile and run TDM without the TDM-supplied Boost stuff and it worked perfectly. A lot of that other build stuff could stand some attention too, IMHO. It's all on my list, but I just don't have enough time ATM.

So, I think we're on a very similar track, with a similar mindset. So I'm very glad to see your posts here. I'll try to post my patches a bit earlier than I have, in case it saves you any time in the future. I wasn't even planning on posting that SCons "time saver" patch when I did, but I was seeing people compiling under Linux, so I pushed it out earlier than planned. I'll consider this all a "lesson learned" for me. :)

Thanks again for the patches! I hope that they'll be put to good use reasonably soon.



#3 nbohr1more

nbohr1more

    Darkmod PR, Wordsmith

  • Development Role
  • PipPipPipPipPip
  • 8043 posts

Posted 22 December 2016 - 09:18 AM

Are you able to run "William Steel 3: Cleighmoor" without crashing?

 

(The lightgem darkness issue is covered here: http://forums.thedar...-invisible-man/ )


Please visit TDM's IndieDB site and help promote the mod:

http://www.indiedb.c...ds/the-dark-mod

(Yeah, shameless promotion... but traffic is traffic folks...)

#4 Hamlet

Hamlet

    Member

  • Member
  • PipPip
  • 24 posts

Posted 23 December 2016 - 12:25 AM

Hello Nightstalker,

 

since I am a newby here, I consider myself excused for behaving like one, and posting lengthy answers line by line. Here is one.

 

Nice work, Hamlet!

It looks like we've been trodding some of the same ground, unfortunately. I probably should have posted the patches I was using to make latest SVN build, but even though it built, it would not run. And I wasn't confident that some of the patches I'd applied were robust enough for all OSes, so I never published them. My apologies if that caused you any wasted time. :(

Let me stress one point: my "work" was with a totally amateur approach, aimed to "make it work".

I did "waste" some time, mostly due to my inability to find and/or understand the documentation (I went through compiling and installing DarkRadiant believing it was a newer version of The Dark Mod, I let you imagine my surprise when I started it... at least it explained the different style and level of the code). I am ashamed to admit, I actually quite enjoyed the process.

 

I think your 'inline.patch' accomplishes the same thing that my 2 DevIL imaging library patches (used against 2.04 source code in this case) did from this post. But it looks like we took different approaches. I haven't had a chance yet to see what the effect of the different approaches might be, but I'll take a closer look tomorrow.

 

Yes. I did not use yours because when I started that work, one week ago, I had no account on this forum and therefore I would be denied access to your attached files <_< .

You decided to make the extern inline static inline, I just made them inline. It's a subtle difference, and I am used to a different pattern completely where those functions are either defined only in their library and only declared in the header, or defined in the header as inline. Anyway, I suggest you not to waste your time with this patch of mine.

 

I think Crinkelite and duzenko (both now, at least part-time, running 32-bit Ubuntu 16.10 with GCC 6.2.0, IIRC) will be able to use your 'pugi_strings.patch' to great advantage. I'm running GCC 5.3.0 compared to your 5.4.0, and I've never seen those errors. But I'll still apply your patch to make sure that it has no negative effect, as an extra datapoint for those who decide to apply it.

 

 

Note that that patch alone will not be enough. I believe the lack of a forwarding header for string as there is iosfwd for iostream is a known "problem" (cf. http://www.gotw.ca/gotw/034.htm), but that would not help neither.

My understanding of the problem is: the precompiled Boost libraries want the type of std::string that was where they were compiled (I grasped Ubuntu 10.10, maybe?), while the program uses the system ones. The code was declaring std::basic_string as in the distributed Boost, my patch as the system one; but the discrepancy is still there and will come out at link time.

Boost and C++ STL are very tightly bound, and the best option is rely on the system one for both. It comes with a price, since Boost interface is not exactly stable. Distributing both Boost and STL would be madness though. Which brings us to your comment that...

As for the Boost libraries, I've taken a different tack, which I had not yet discussed on the forum before now. Just a few days ago, I'd compiled and run TDM using the Boost headers and library that come with Slackware 14.2 (and probably a lot of other Linux distros these days), thereby dropping 124 MB (!!!) of header files and about 7 MB of static library files from the TDM source distribution. But I was waiting to bring that up until after the 2.05-beta hubbub died down. I just wanted to prove that I could compile and run TDM without the TDM-supplied Boost stuff and it worked perfectly. A lot of that other build stuff could stand some attention too, IMHO. It's all on my list, but I just don't have enough time ATM.

 

 

... if I read your text right, it's not a different tack at all, it's exactly the same. I have used Boost 1.62.0, and it worked amazingly well. But my "shell patch" hard-codes the path of the libraries in my system, so it's no good.

My best bet would be to tell the linker nothing about where they are, and expect them in the standard library directories which are always checked.

One comment on your 'build_tuning.patch'.... With my SCons patch, there should be no need to do the 'touch scons.signatures.dblite' step in the 'linuxBuild.sh' file at all now, which is why that patch eliminated it entirely. Did you find that it's still needed on your setup? Just curious. I'm pretty sure that if you deleted the signature file, SCons will still work, but as my post mentioned, that was not the case in the past, hence someone's addition of the 'touch' in the script as a less-than-optimal work-around.

 

 

The reason I added that was I saw a complaint "that file does not exist", and I though "why not to touch it in that case". But I admit I can't reproduce that message with the head version.

The number of jobs patch is good to go and I endorse its merging. The other is good for me and you, but not for a distributed binary. If the librarian builds the executable for the processor that has become available tomorrow, and my PC is five years old, I won't be able to run it. Ok, pentium3 is maybe a bit too conservative...

 

Thanks again for the patches! I hope that they'll be put to good use reasonably soon.

 

I stress one last time the "caveat emptor" message and the fact that they are not great stuff. But once a direction is chosen some of that stuff can be made well. I think especially to the decision about Boost.



#5 NightStalker

NightStalker

    Linux hero

  • Development Role
  • PipPip
  • 158 posts

Posted 23 December 2016 - 02:33 PM

Aside: I'm splitting this reply into 2 parts because the forum software warns, "You have posted more than the allowed number of quoted blocks of text".

since I am a newby here, I consider myself excused for behaving like one, and posting lengthy answers line by line.

I do that all the time on various forums. I never knew that it made me look like a newbie. :blush: I just thought I was being thorough. :) But I guess maybe I should take the hint of the forum software which made me split this post into 2 threads posts, eh? ;)

Let me stress one point: my "work" was with a totally amateur approach, aimed to "make it work".

Trying to get "latest SVN" to compile, my approach was the same. Even with our similar aims, I still liked at least one of your patches (the one to 'idlib/geometry/RenderMatrix.cpp') better than my own!

As for Dark Radiant, I ran into some problem with compiling the FTGL dependency but never looked very deeply into it. It's on my "to do" list, just way down there.

Yes. I did not use yours because when I started that work, one week ago, I had no account on this forum and therefore I would be denied access to your attached files <_< .

Sorry about that. Those patches were short enough that I could have just "inlined" them rather than using an attachment, but my thinking was more for ease of the person potentially applying them. I had not even considered that the forum software made them inaccessible to non-account folks, but I'll keep that in mind from now on.

You decided to make the extern inline static inline, I just made them inline. It's a subtle difference, and I am used to a different pattern completely where those functions are either defined only in their library and only declared in the header, or defined in the header as inline. Anyway, I suggest you not to waste your time with this patch of mine.

Well, one way or another, using yours or mine (I still haven't had a chance to examine and consider if there are any hidden implications of either of our techniques), my plan is to test some more and eventually push a fix into SVN. But I want to be very careful not to break the Linux building process for grayman (or, ideally, anyone else).

Note that that patch alone will not be enough.

You are clearly much more familiar and comfortable with both C++ and Boost than I am, so I very much appreciate your comments. I've pointed Crinkelite and duzenko to this thread from my other thread, in case it helps them even a little.

... if I read your text right, it's not a different tack at all, it's exactly the same.

You're right, of course. Somehow I didn't look carefully enough and I got a bit confused, in part because I only addressed the main TDM app, not the 'tdm_update' app. Regardless, I have some ideas about improving the way things are compiled under Linux and will look into them, test them, and (hopefully) get others to agree that they're useful changes so that I can eventually commit them. I just need to find the time.

The reason I added that was I saw a complaint "that file does not exist", and I though "why not to touch it in that case". But I admit I can't reproduce that message with the head version.

Grayman has already checked my patch into SVN. So I'll leave things as they are unless you find a reason why it's causing problems (i.e. with no 'touch'), in which case I will look into it immediately.

 

(reply continued in next post -->)


Edited by NightStalker, 23 December 2016 - 02:38 PM.


#6 NightStalker

NightStalker

    Linux hero

  • Development Role
  • PipPip
  • 158 posts

Posted 23 December 2016 - 02:36 PM

(<-- Reply continued from previous post)

The number of jobs patch is good to go and I endorse its merging.

I agree. I plan to test it soon and (hopefully) commit it to SVN, barring any disapproval beforehand.

The other is good for me and you, but not for a distributed binary.

Very true. I would not commit it to SVN "as-is" because it would potentially mess up grayman's builds for the Linux masses. But it gives me ideas (for some later date) that might allow us the "best of both worlds" -- i.e. where the person building Linux for the official TDM releases (ostensibly, grayman) still uses the "Lowest Common (supported) Denominator" for architecture/CPU, but us "build-it-ourself" types (at least for Linux) can still benefit from better-than-LCD CPUs. Another item added to my "TDM to do" list.

I stress one last time the "caveat emptor" message and the fact that they are not great stuff.

Understood, but please don't sell your own efforts short. I'm just a newbie here myself, but I feel comfortable saying that I suspect that everyone here appreciates your efforts. I know I sure do!

But once a direction is chosen some of that stuff can be made well. I think especially to the decision about Boost.

Completely agree. Over time, I hope some of these oddities in the Linux build setup can be improved in various ways. I will look forward to any/all input you have on any of these matters, whether now or any time in the future!



#7 jackiebean

jackiebean

    Newbie

  • Member
  • Pip
  • 6 posts

Posted 22 January 2017 - 08:47 PM

I just recently built TDM on a lubuntu install, then had a serious issue where i had to make a new partition and install linux mint instead, but i was able to (so far) simply copy and paste the whole directory to linux mint and even run TDM, however i had problems getting TDM updater to run. I copied a newer version over to the directory but have not tried it yet since i have all the missions downloaded and updated anyway. Will probably be testing it out sooner or later when i have time.

 

I think the reason i was able to just copy it over is because the dependency libraries are already installed in mint but not supported any more in lubuntu, which to me is weird, but i guess since linux mint is the most stable distro of linux (or one of the most stable distros), it makes sense they would cover more bases to support all these native games.



#8 NightStalker

NightStalker

    Linux hero

  • Development Role
  • PipPip
  • 158 posts

Posted 22 January 2017 - 10:28 PM

Hi, jackiebean... Welcome to the forums!

I've never run Linux Mint (I'm a long-time Slackware guy), so I cannot comment on the dependencies.

But if you continue having problems running the "updater" ('tdm_update.linux') app, please let us know, with as many details as you can provide (especially including the output of the log file) and we'll be sure to assist.  Also, be aware that you can compile the updater app itself, quickly and independently of the game, in case that helps.  Simply drop into the 'tdm_update' directory and issue a 'scons' command.

And try 'tdm_update.linux --help' to see a list of useful options, including (but not limited to) '--noselfupdate'.  But don't try using the '--dry-run' option -- it's advertised but not supported, something I intend to fix in the near future by eliminating it from the '--help' output.

Oh, another thing that may be "biting" you: The updater is intolerant of read-only PK4 files in your installation (run-time) directory, so if you have any of those, make them write-able before running the updater.

Finally, be aware that TDM 2.05 is about to be released and will include a fix to the updater app for a serious bug in the time-stamping of "resource" files, so you may want to wait until then to do any serious experimenting and updating.  But don't let that stop you from playing around with a 2.04 installation if you're motivated to do so.

If you encounter any issues you need help with, don't hesitate to let us know.  Good luck!


  • jackiebean likes this

#9 jackiebean

jackiebean

    Newbie

  • Member
  • Pip
  • 6 posts

Posted 22 January 2017 - 10:41 PM

Yes I am fully aware of the time-stamp bug, but I was not aware of read only issues with the pk4 files, so that may be stopping the updater from running? If so then it may be a leftover issue from copying the directories over to the new install, since much of the time for some reason mint locks files copied from one install to another (even if I use the same Linux user name and password from one distro to the other) and I have to open the whole folder as root so I can take ownership of the files and change them to read/write. Not a huge issue and completely fixable.

 

I noticed the time-stamp issue from one of the other posts i read and that probably explains why it keeps attempting to update the mission files because of the mismatch in some of them. I'm not sure if the update fixes the time-stamp or causes the problem to begin with, but i do not intend to update the mission files for a while and look forward to that bug and others being fixed in the next build.


Edited by jackiebean, 22 January 2017 - 10:42 PM.


#10 NightStalker

NightStalker

    Linux hero

  • Development Role
  • PipPip
  • 158 posts

Posted 22 January 2017 - 11:10 PM

[...] but I was not aware of read only issues with the pk4 files, so that may be stopping the updater from running?

Yes. Going back to my old notes on this, I was seeing errors like this when running 'tdm_update.linux' with read-only PK4 files:

Applying differential update package...
[=                        ] 6.2% Preparing PK4: tdm_env01.pk4 tdm_update.linux: /home/rand/Desktop/darkmod_src_2.04/include/boost/smart_ptr/shared_ptr.hpp:648: typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = tdm::ZipFileWrite]: Assertion `px != 0' failed.

Not a very intuitive message, is it? ;) Looking into that is another thing on my list, assuming I get the time someday.

I'm not sure if the update fixes the time-stamp or causes the problem to begin with, [...]

The updater (even in 2.04) is not causing the problem, it's just not fixing (until the 2.05 release) the bad timestamps the way it should. (The whole story is long and convoluted and it's already covered in that link I gave, so I won't repeat things here.)  But the problem should start to go away beginning with 2.05. :)



#11 jackiebean

jackiebean

    Newbie

  • Member
  • Pip
  • 6 posts

Posted 23 January 2017 - 07:21 PM

no need to explain

 

so i finally got the updater to work right and then switched to the nvidia driver and the game works great. i may have to tweak settings simply because it is running the GPU hard and the fan is nuts, but that is not due to the game itself, but the display setup is what i suspect.

 

it is kind of annoying but the problem lies with the fact that it is a laptop, and i have a secondary monitor since the main one went out. if i could figure out how to completely bypass the built in monitor then it would probably behave much better, but i guess i may have to start saving for a refurb desktop from new egg or tiger direct.

 

EDIT: ok, something new to add, for those who want the game to go to their external monitor in linux machines. If they use their Nvidia control panel or equivalent, and save to a new xorg config file, then they can force the laptop monitor off, and make the secondary the primary monitor in the xorg settings, which will allow the game to play in full screen on the external monitor, and the GPU will run a lot cooler. I was having issues with the graphics due to the fact i could not run it in full screen because it was forcing the laptop monitor back on (albeit without the back-lights behind the LCD, so basically black with faint details) and until i adjusted things i had to try playing in windowed mode, which is not very GPU friendly. Still have yet to figure out the controls and what the objective really is in the first mission, but i know it runs ok. I got killed by the first two guards.

 

Now i have to find out if i can balance the graphics settings just right for better eye candy.


Edited by jackiebean, 24 January 2017 - 08:09 PM.






Also tagged with one or more of these keywords: linux

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users